Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

Variational Model Merging for Pareto Front Estimation in Multitask Finetuning

T0 review · 4 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Bayesian model merging with flexible posteriors makes merged models track joint multitask training more faithfully across task weightings.

desk verdict Useful Bayesian unification of model merging with a practical preview tool, but the headline 'necessarily better' claim is unproven and likely false without assumptions; the empirical trend is consistent, if under-reported. read the letter →

arxiv 2412.08147 v2 pith:FLEBBMWN submitted 2024-12-11 cs.LG cs.AIstat.ML

Hugo Monzón Maldonado , Nico Daheim , Thomas Möllenhoff , Iryna Gurevych , Mohammad Emtiyaz Khan This is my paper · ORCID
classification cs.LGcs.AIstat.ML
keywords multitaskfinetuningmodelmergingParetofrontestimationvariationalinferenceposteriorexponentialfamilymixtureofGaussianstaskweighting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper wants to make the search over task weights in multitask finetuning cheap by estimating the whole performance tradeoff surface without retraining. Its proposal, Variational Model Merging, treats each separately finetuned model's posterior as a surrogate for that task's loss, then forms merged parameters for any weighting $\alpha$ by taking the mode of the weighted product of posteriors. The central claim is that the more flexible the posterior family used per task, the better the merged preview tracks the performance of models actually finetuned jointly, across the full range of $\alpha$ values. Simple averaging and Task Arithmetic fall out as isotropic-Gaussian special cases, and the paper validates the ordering on vision and language transformers. If the claim is right, practitioners can screen task weightings in seconds to minutes instead of running many joint-training experiments.

What carries the argument

The engine is posterior merging: the identity $p_\alpha \propto p_0^{\gamma}\prod_t p_t^{\alpha_t}$ means weighted-multitask inference can be obtained by combining per-task posteriors instead of retraining. Per-task posteriors are fitted variationally as exponential-family or mixture-of-Gaussian distributions, and the merged parameter is the mode of the weighted product (Eq. 8), which is closed form for exponential families. Algorithm 1 iterates an EM-style fixed point for mixture-of-Gaussians that generalizes a classical mode-finding procedure. Replacing each task loss by $-\log q_t$ is what makes the surrogates accurate over a region rather than at a single point, and the flexibility of the posterior family controls how much of the relevant region is covered.

What would settle it

Take a two-task problem with known losses, compute the true Pareto front by joint training on a fine $\alpha$ grid, then compute isotropic, full-Gaussian and mixture previews from single-task models. If the full-Gaussian or mixture preview has higher mean squared error to the true front, or finds a best $\alpha$ whose true performance is worse than the isotropic version's, the claimed monotone improvement is falsified for that setting.

Watch

Extended reading notes

Core claim

The paper's claim is that Pareto-front estimation by model merging is a Bayesian posterior-merging problem, and that moving to more expressive posterior families strictly helps. Given task losses $\ell_t$ and a prior $p_0$, each single-task posterior $p_t$ is approximated variationally by $q_t$ in an exponential family; merging with weights $\alpha$ gives $\hat{\theta}_\alpha$ as the mode of the product $p_0^{\gamma}\prod_t q_t^{\alpha_t}$. For Gaussian $q_t$ this reduces to Hessian-weighted merging, and for mixtures the paper derives an EM fixed-point algorithm. The theoretical result stated is that more flexible posteriors necessarily yield better estimates of the Pareto front: a full-Gaussian merge is expected to beat an isotropic one, and a mixture to beat a single Gaussian. Experiments on logistic regression, ResNet-20, ViT-B/32, RoBERTa and LoRA-finetuned GEMMA-2B show previews improving in this order while staying far cheaper than joint training.

Load-bearing premise

The argument rests on the variationally fitted per-task posterior $q_t$ being an accurate stand-in for the true task loss everywhere the merged solutions travel; if a more flexible $q_t$ lowers the KL divergence but moves its mode away from the regions that matter for some $\alpha$, the preview for those weights can get worse instead of better.

Editorial extensions

If this is right

  • Task Arithmetic and simple parameter averaging become the isotropic-Gaussian special case of one framework, so any improvement in posterior approximation translates directly into a new merging rule.
  • Previews over a grid of $\alpha$ values cost seconds to minutes after single-task finetuning, versus hours for each joint finetuning run, making broader weight searches practical for large models.
  • Hessian-based merging methods become full-Gaussian special cases, giving a unified explanation of why they outperform naive averaging.
  • For mixture posteriors, increasing the number of components tightens the preview: on the CIFAR-10 ResNet experiment the best previewed region moves toward the joint-training optimum as the number of components grows.
  • Because the merged mode is closed form for exponential families, the same recipe extends to non-Gaussian posteriors, not just Gaussians, as the paper illustrates with a Beta-Bernoulli example.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A concrete test of the 'necessarily better' claim would be to measure whether lower variational KL for each task always translates into lower preview error on the $\alpha$ grid; the paper reports MSE but does not tie the two numbers together per $\alpha$, so a systematic correlation study would separate the theorem from the approximation.
  • The monotonicity may fail for utility functions that care about one task's worst-case behavior, because a more flexible posterior can concentrate mass in a region that is globally accurate but misses the exact minimizer for some $\alpha$; worst-case Pareto error is the sharper quantity.
  • If the posterior family is made too flexible, each task's surrogate could overfit its own training loss and produce previews that look good in-sample but mislead on validation, suggesting the method should be paired with held-out preview evaluation.
  • The same posterior-merging identity suggests a route to task-weight search during continual pretraining: previews could warm-start the last fraction of pretraining where the task mix is rebalanced.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. This paper proposes a Bayesian model-merging framework for generating cheap “previews” of multitask finetuning behavior across task-weight vectors α. Single-task finetuned models are converted into exponential-family or mixture-of-Gaussian posterior surrogates q_t, and the weighted multitask solution is approximated by the mode of the product ∏_t q_t(θ)^{α_t}. The paper recovers Task Arithmetic and Hessian-weighted merging as Gaussian special cases, adds an EM-based mixture-of-Gaussians merging algorithm, and evaluates preview quality on logistic regression, ResNet-20, ViT-B/32, RoBERTa, and GEMMA-2B experiments. The advertised main theoretical result is that more flexible posteriors necessarily yield better estimates of Pareto fronts.

Significance. The practical contribution is real and useful: the paper demonstrates with diverse vision and language models that merging can produce fast previews of reweighting behavior, and the hyperparameter details in the appendices make the empirical setup unusually transparent. The Bayesian interpretation also provides a clean unification of several existing merging methods and a principled route to new mixture-based merges. However, the headline theoretical claim is not established; the manuscript's actual support is empirical, and the claim as stated is both underdefined and not derived. If the authors rework the central claim into a proven statement under explicit assumptions or an explicitly empirical statement, the paper would be a solid contribution to the model-merging and multitask-finetuning literature.

major comments (4)
  1. [Abstract; Sec. 3.3, Eqs. (7)–(8)] The abstract's statement that “more flexible posteriors necessarily yield better estimates of Pareto fronts” is not a proven theorem in the manuscript, and the discussion in Sec. 3.3 does not establish it. Equation (7) minimizes D_KL[q_t || p_t], which controls each surrogate marginally, while the preview θ̂_α in Eq. (8) and Alg. 1 is the mode of the weighted product ∏_t q_t(θ)^{α_t}. A reduction in KL can shift probability mass or change the tails of q_t in ways that move the product's mode away from the true weighted minimizer θ_α for some α. The toy validation in App. B.1 uses log-sum-exp losses, which are convex and give unimodal posteriors, so it does not test the multimodal failure mode. The claim should be replaced either by a theorem proved under explicit assumptions (for example, log-concavity or exact membership of p_t in the chosen exponential family) or by an empirical/expected statement.
  2. [Sec. 2–4; Tables 1 and 2] The paper never defines the error of a “Pareto-front estimate” formally. The abstract's theoretical claim is therefore not attached to a precise quantity: Tables 1 and 2 report the MSE of predicted accuracies and the best-preview accuracy, not a Pareto-front metric under a stated definition. Without such a definition, the claimed comparison between posterior families is unfalsifiable, and the empirical “consistently” wording should be tied to the metric actually reported.
  3. [Table 2, Fig. 3 row; Sec. 4.1] The empirical results do not support “necessarily better” even informally. In the ResNet-20 row of Table 2, the Hessian-weighted preview finds a best α whose true multitask accuracy (64.6%) is below the value found by simple merging (67.4%), and the preview accuracy itself decreases from 62.4% to 62.3%. The paper's statement in Sec. 4 that “we see a consistent trend that the performance improves as we use better posteriors” is too strong for this row; it should be reported as mixed or approximately equal and reconciled with the theoretical claim.
  4. [Sec. 3.4; App. A.3; Alg. 1] For mixture posteriors, the closed-form argument in App. A.2 does not apply, because App. A.3 explicitly states that the EM iterates converge only to a local maximum. The mode returned by Alg. 1 can therefore depend on initialization, number of components, and convergence tolerance. The claim that more flexible mixtures necessarily improve previews is especially unsupported without additional analysis of these local-mode effects, and the paper should state clearly that the mixture previews are heuristic rather than guaranteed.
minor comments (5)
  1. [Title page] The manuscript contains two different titles—“Variational Model Merging for Pareto Front Estimation in Multitask Finetuning” in the arXiv header and “How to Weight Multitask Finetuning? Fast Previews via Bayesian Model-Merging” in the running text—and they should be aligned.
  2. [Fig. 3; App. C.1] There are small typographical errors: “Hessian-Weigthed” and “Mixture-Weigthed” appear in Fig. 3, and App. C.1 contains “capture more the the exact solution.”
  3. [Table 2] The pairing between preview scores and the gray true scores is difficult to read in monochrome; an explicit column grouping or legend should indicate which values are previews and which are the corresponding multitask-finetuning results, and the table should state which α produced each value.
  4. [Sec. 3.3] The sentence “We can just use such optimizers to compute the posterior q_t” is vague about which optimizer (Adam, Adam-like, or IVON) solves Eq. (7) and under what approximation; please specify the exact relationship.
  5. [App. B.4] The phrase “The pretrained and finetuned checkpoints of ViT-B-32 a model based on CLIP” contains a stray article and should be rewritten.

Circularity Check

0 steps flagged · score 0.0 of 10

No construction-level circularity: the merged-preview targets are exact multitask finetuning runs, and the key merging formulas are derived in the paper rather than imported as self-citations.

full rationale

The paper's derivation chain is self-contained in the relevant sense. Per-task posteriors q_t are fitted only through the variational objective Eq. 7 using each task's own data, and the merged parameter btheta_alpha is then computed from those q_t via Eq. 8 or Algorithm 1 for a grid of alpha values. The preview quality is measured against separately computed exact multitask finetuning (Eq. 1 / Eq. 6), so the predicted quantity is not used to fit any parameter or posterior in the method. Existing methods such as Task Arithmetic and Hessian-weighted merging are recovered as special cases, and the Hessian-weighted formula of Daheim et al. (2024) is re-derived in Appendix A.1 rather than merely cited. The closed-form exponential-family MAP argument in Appendix A.2 is also derived explicitly, with the Khan & Rue (2023) citation used only as a pointer to a standard variational result. The same-group citations (Daheim et al. 2024; Khan & Rue 2023; Lin et al. 2019; Shen et al. 2024) support algorithms and background, but the load-bearing claim about better previews is validated against external benchmarks (CIFAR-10, ViT-B/32, RoBERTa, GEMMA-2B) and against exact joint finetuning, so no prediction reduces to its inputs by construction. The abstract's stronger assertion that more flexible posteriors 'necessarily' yield better Pareto-front estimates is not backed by a theorem in the text and is a correctness/overclaim concern rather than a circularity: the variational objective does not include preview quality, and the paper provides no proof that KL improvement of each q_t translates monotonically to the merged mode's behaviour over all alpha. That is an unsupported inference, not a self-referential reduction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

No new physical or mathematical entities are introduced; 'preview' is a methodological concept. The central claim rests on the variational-surrogate assumption and on standard Bayesian/exponential-family machinery, plus several hand-chosen hyperparameters that affect the empirical comparison.

free parameters (4)
  • K (number of mixture components / IVON runs) = 10-30 per task
    Chosen by hand; the empirical claim that more components improve previews depends on this number.
  • EM iterations for mixture merging = 5-10
    Fixed in Alg. 1; convergence to a good local mode is assumed but not verified.
  • IVON temperature / sample-size weighting = 5e4 to 2e7 depending on experiment
    Balances likelihood and prior in the variational objective; set per experiment, not tuned to preview quality.
  • Evaluation grid spacing for alpha = 0.02 to 0.1
    Affects reported MSE and contour comparisons; not a method parameter but a hand-chosen evaluation detail.
assumptions (5)
  • domain assumption Generalized Bayesian construction p(D_t|theta) proportional to exp(-l_t(theta)) and p0 proportional to exp(-R0(theta)) is valid.
    Used in Eq. 5-6 so that posterior merging recovers the weighted multitask objective; standard in generalized Bayes but still an assumption about losses and priors.
  • domain assumption Each single-task optimum theta_t has zero gradient and the Hessian captures local curvature (Laplace approximation).
    Invoked in Sec. 3.1 Eq. 4 and in building Gaussian posteriors from theta_t and H_t.
  • ad hoc to paper A variational q_t that minimizes KL to the exact posterior p_t is a globally accurate surrogate for l_t over the regions visited by all alpha values.
    This is the load-bearing premise in Sec. 3.3; lower KL is assumed to translate into better merged-posterior modes, which is not formally established.
  • standard math The MAP of an exponential-family merged posterior has a closed form via theta = (nabla A)^{-1}(t(lambda_alpha)).
    App. A.2 relies on standard exponential-family / conjugate-prior results.
  • domain assumption The EM procedure in Eq. 12 converges to a useful local mode of the weighted mixture objective.
    Sec. 3.4 and Alg. 1 assume a few EM iterations give a representative mode; non-concavity means local optima are possible.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Variational Model Merging for Pareto Front Estimation in Multitask Finetuning." pith.science (2026). https://pith.science/paper/FLEBBMWN

@misc{pith2026241208147,
  author       = {Pith},
  title        = {Pith review of: Variational Model Merging for Pareto Front Estimation in Multitask Finetuning},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FLEBBMWN}},
  note         = {Machine review of arXiv:2412.08147}
}
read the original abstract

Pareto fronts are useful to find good task-mixing strategies for multitask finetuning, but they are also costly to compute. To reduce costs, recent works have used existing model merging methods to help train cheap surrogate models to estimate the Pareto fronts. However, no work has yet considered designing new model-merging methods to directly, and provably, improve the quality of Pareto fronts. Here, we fill this gap by proposing a new Bayesian approach called Variational Model Merging. In this approach, existing model-merging methods are obtained as special cases of "posterior-merging" when Gaussian posteriors are used and new model-merging strategies can be derived by using non-Gaussian posteriors. Our main theoretical result is to show that more flexible posteriors necessarily yield better estimates of Pareto fronts. For instance, a Pareto front estimate obtained by merging full-Gaussian posteriors is expected to be better than that obtained by using isotropic Gaussian posteriors. We validate the theory through extensive empirical results on vision and language transformers where better Gaussian families consistently yields better or comparable Pareto fronts. Our work is a rare instance where Bayesian ideas are used to improve Pareto analysis.

Figures

Figures reproduced from arXiv: 2412.08147 by the authors.

Figure 1
Figure 1. Our goal is to aid the search for good weights in weighted multitask finetuning. We show a [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. An illustration of our Bayesian approach to improve preview quality for a toy multitask [PITH_FULL_IMAGE:figures/full_fig_p002_2.png] view at source ↗
Figure 3
Figure 3. Results on image classification using ResNet-20 on CIFAR-10 with three tasks constructed [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: Results using ViT-B/32 on GTSRB, RESISC45, SVHN (top) and EuroSAT, Cars, Sun397 [PITH_FULL_IMAGE:figures/full_fig_p009_4.png]
Figure 5
Figure 5. Figure 5: Merging of multitask finetuned RoBERTa models on pairs of sentiment analysis tasks. [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Preview of reweightings for RoBERTA multitask finetuned on three sentiment analysis tasks. [PITH_FULL_IMAGE:figures/full_fig_p010_6.png]
Figure 7
Figure 7. Figure 7: Here, we merge LoRA-finetuned GEMMA-2B models trained on IWSLT2017de-en and [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]
Figure 8
Figure 8. Figure 8: Multitask Learning on MNIST. As the posterior approximation gets more expressive the [PITH_FULL_IMAGE:figures/full_fig_p020_8.png]
Figure 9
Figure 9. Figure 9: Similar to the previous setting, on balanced tasks improving the posterior approximation [PITH_FULL_IMAGE:figures/full_fig_p020_9.png]
Figure 10
Figure 10. Figure 10: Here, we merge LoRA-finetuned GEMMA-2B models trained on IWSLT2017de-en [PITH_FULL_IMAGE:figures/full_fig_p020_10.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Modeling Multi-Task Model Merging as Adaptive Projective Gradient Descent

    cs.LG 2025-01 conditional novelty 6.0 of 10

    DOGE merges fine-tuned models by optimizing a data-free loss-gap proxy with gradient steps projected orthogonal to a shared task subspace, improving average accuracy over previous methods.

Reference graph

Works this paper leans on

65 extracted references · 46 canonical work pages · cited by 1 Pith paper

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Muppet: Massive multi-task representations with pre-finetuning

    Armen Aghajanyan, Anchit Gupta, Akshat Shrivastava, Xilun Chen, Luke Zettlemoyer, and Sonal Gupta. Muppet: Massive multi-task representations with pre-finetuning. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2021. URL https://aclanthology.org/2021.emnlp-main.468

    work page 2021

  3. [3]

    Safety-tuned LL a MA s: Lessons from improving the safety of large language models that follow instructions

    Federico Bianchi, Mirac Suzgun, Giuseppe Attanasio, Paul Rottger, Dan Jurafsky, Tatsunori Hashimoto, and James Zou. Safety-tuned LL a MA s: Lessons from improving the safety of large language models that follow instructions. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=gT5hALch9z

    work page 2024

  4. [4]

    Neural networks for pattern recognition

    Christopher M Bishop. Neural networks for pattern recognition. Oxford university press, 1995. URL https://global.oup.com/academic/product/neural-networks-for-pattern-recognition-9780198538646

    work page 1995

  5. [5]

    P. G. Bissiri, C. C. Holmes, and S. G. Walker. A general framework for updating belief distributions. J. R. Stat. Soc. Ser. B Methodol., 2016. URL https://rss.onlinelibrary.wiley.com/doi/abs/10.1111/rssb.12158

    doi:10.1111/rssb.12158 2016

  6. [6]

    Mode-finding for mixtures of G aussian distributions

    Miguel Carreira-Perpi \ n \' a n. Mode-finding for mixtures of G aussian distributions. IEEE Trans. Pattern Anal. Mach. Intell. (PAMI), 2000. URL https://ieeexplore.ieee.org/document/888716

    work page 2000

  7. [7]

    Multitask learning

    Rich Caruana. Multitask learning. Mach. Learn., 28 0 (1): 0 41--75, 1997. URL https://doi.org/10.1023/A:1007379606734

    doi:10.1023/a:1007379606734 1997

  8. [8]

    PAC-B ayesian supervised classification: The thermodynamics of statistical learning

    Olivier Catoni. PAC-B ayesian supervised classification: The thermodynamics of statistical learning. Institute of Mathematical Statistics Lecture Notes, 2007. URL https://www.jstor.org/stable/i20461497

    work page 2007

Show all 65 references
  1. [9]

    Overview of the IWSLT 2017 evaluation campaign

    Mauro Cettolo, Marcello Federico, Luisa Bentivogli, Jan Niehues, Sebastian St \"u ker, Katsuhito Sudoh, Koichiro Yoshino, and Christian Federmann. Overview of the IWSLT 2017 evaluation campaign. In Proceedings of the 14th International Conference on Spoken Language Translation...

    2017

  2. [10]

    GradNorm : Gradient normalization for adaptive loss balancing in deep multitask networks

    Zhao Chen, Vijay Badrinarayanan, Chen-Yu Lee, and Andrew Rabinovich. GradNorm : Gradient normalization for adaptive loss balancing in deep multitask networks. In International Conference on Machine Learning (ICML), 2018. URL https://proceedings.mlr.press/v80/chen18a.html

    2018

  3. [11]

    Remote sensing image scene classification: Benchmark and state of the art

    Gong Cheng, Junwei Han, and Xiaoqiang Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105 0 (10): 0 1865--1883, 2017. URL http://dx.doi.org/10.1109/JPROC.2017.2675998

    2017

  4. [12]

    Zhao, Yanping Huang, Andrew M

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro - Ros, Marie Pellat, Kevin Robin...

    2024

  5. [13]

    Model merging by uncertainty-based gradient matching

    Nico Daheim, Thomas M \"o llenhoff, Edoardo Ponti, Iryna Gurevych, and Mohammad Emtiyaz Khan. Model merging by uncertainty-based gradient matching. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=D7KJmfEDQP

    2024

  6. [14]

    ColD fusion: Collaborative descent for distributed multitask finetuning

    Shachar Don - Yehiya, Elad Venezian, Colin Raffel, Noam Slonim, and Leshem Choshen. ColD fusion: Collaborative descent for distributed multitask finetuning. In Annual Meeting of the Association for Computational Linguistics (ACL), 2023. URL https://aclanthology.org/2023.acl-long.46

    2023

  7. [15]

    An image is worth 16x16 words: Transformers for image recognition at scale

    Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, Jakob Uszkoreit, and Neil Houlsby. An image is worth 16x16 words: Transformers for image recognition at...

    2021

  8. [16]

    GLaM: efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. GLaM: efficient scaling of language models with mixture-of-experts. In International Conference on Machine Learning (ICML), 2022. URL htt...

    2022

  9. [17]

    Durrant-Whyte and Mike Stevens

    Hugh F. Durrant-Whyte and Mike Stevens. Data fusion in decentralised sensing networks. In International Conference on Information Fusion, 2001. URL https://api.semanticscholar.org/CorpusID:43837722

    2001

  10. [18]

    Knowledge card: Filling LLM s' knowledge gaps with plug-in specialized language models

    Shangbin Feng, Weijia Shi, Yuyang Bai, Vidhisha Balachandran, Tianxing He, and Yulia Tsvetkov. Knowledge card: Filling LLM s' knowledge gaps with plug-in specialized language models. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.n...

    2024

  11. [19]

    Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabilities

    Kazuki Fujii, Taishi Nakamura, Mengsay Loem, Hiroki Iida, Masanari Ohi, Kakeru Hattori, Hirai Shota, Sakae Mizuki, Rio Yokota, and Naoaki Okazaki. Continual pre-training for cross-lingual LLM adaptation: Enhancing japanese language capabilities. In First Conference on Language...

    2024

  12. [20]

    Gemma 2: Improving open language models at a practical size, 2024 a

    Gemma Team . Gemma 2: Improving open language models at a practical size, 2024 a . URL https://arxiv.org/abs/2408.00118

    2024 arXiv

  13. [21]

    Gemma: Open models based on G emini research and technology, 2024 b

    Gemma Team . Gemma: Open models based on G emini research and technology, 2024 b . URL https://arxiv.org/abs/2403.08295

    2024 arXiv

  14. [22]

    Multi-loss weighting with coefficient of variations

    Rick Groenendijk, Sezer Karaoglu, Theo Gevers, and Thomas Mensink. Multi-loss weighting with coefficient of variations. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021. URL https://openaccess.thecvf.com/content/WACV2021/html/Groenendijk_Multi-Loss_We...

    2021

  15. [23]

    Deep residual learning for image recognition

    Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2016. URL https://ieeexplore.ieee.org/document/7780459

    2016

  16. [24]

    Euro SAT : A novel dataset and deep learning benchmark for land use and land cover classification

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Euro SAT : A novel dataset and deep learning benchmark for land use and land cover classification. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2019. URL https://ieeexplore...

    2019

  17. [25]

    Detection of traffic signs in real-world images: The G erman T raffic S ign D etection B enchmark

    Sebastian Houben, Johannes Stallkamp, Jan Salmen, Marc Schlipsing, and Christian Igel. Detection of traffic signs in real-world images: The G erman T raffic S ign D etection B enchmark. In International Joint Conference on Neural Networks (IJCNN), 2013. URL https://ieeexplore....

    2013

  18. [26]

    Lo RA : Low-rank adaptation of large language models

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lo RA : Low-rank adaptation of large language models. In International Conference on Learning Representations (ICLR), 2022. URL https://openreview.net/forum?id=nZeVKeeFYf9

    2022

  19. [27]

    Editing models with task arithmetic

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=6t0Kwf8-jrj

    2023

  20. [28]

    Multitask mayhem: Unveiling and mitigating safety gaps in LLM s fine-tuning

    Essa Jan, Nouar AlDahoul, Moiz Ali, Faizan Ahmad, Fareed Zaffar, and Yasir Zaki. Multitask mayhem: Unveiling and mitigating safety gaps in LLM s fine-tuning. arXiv:2409.15361, 2024. URL https://arxiv.org/abs/2409.15361

    2024 arXiv

  21. [29]

    ForkMerge : Mitigating negative transfer in auxiliary-task learning

    Junguang Jiang, Baixu Chen, Junwei Pan, Ximei Wang, Dapeng Liu, Jie Jiang, and Mingsheng Long. ForkMerge : Mitigating negative transfer in auxiliary-task learning. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://openreview.net/forum?id=vZHk1QlBQW

    2023

  22. [30]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc - Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. In International Conference on Learning Representations (ICLR), 2023. URL https://openreview.net/forum?id=FCnohuR6AnM

    2023

  23. [31]

    The B ayesian learning rule

    Mohammad Emtiyaz Khan and H vard Rue. The B ayesian learning rule. J. Mach. Learn. Res. (JMLR), 2023. URL https://jmlr.org/papers/v24/22-0291.html

    2023

  24. [32]

    Fast and scalable bayesian deep learning by weight-perturbation in Adam

    Mohammad Emtiyaz Khan, Didrik Nielsen, Voot Tangkaratt, Wu Lin, Yarin Gal, and Akash Srivastava. Fast and scalable bayesian deep learning by weight-perturbation in Adam . In International Conference on Machine Learning (ICML), 2018. URL https://proceedings.mlr.press/v80/khan18a.html

    2018

  25. [33]

    3D object representations for fine-grained categorization

    Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3D object representations for fine-grained categorization. In International Conference on Computer Vision Workshops (CVPRW), 2013. URL https://ieeexplore.ieee.org/document/6755945

    2013

  26. [34]

    Fast and simple natural-gradient variational inference with mixture of exponential-family approximations

    Wu Lin, Mohammad Emtiyaz Khan, and Mark Schmidt. Fast and simple natural-gradient variational inference with mixture of exponential-family approximations. In International Conference on Machine Learning (ICML), 2019. URL https://proceedings.mlr.press/v97/lin19b.html

    2019

  27. [35]

    MFTCoder : Boosting code LLM s with multitask fine-tuning

    Bingchang Liu, Chaoyu Chen, Cong Liao, Zi Gong, Huan Wang, Zhichao Lei, Ming Liang, Dajun Chen, Min Shen, Hailian Zhou, Hang Yu, and Jianguo Li. MFTCoder : Boosting code LLM s with multitask fine-tuning. arXiv:2311.02303, 2023. URL https://arxiv.org/abs/2311.02303

    2023 arXiv

  28. [36]

    RoBERTa : A robustly optimized BERT pretraining approach, 2019

    Yinhan Liu, Myle Ott, Naman Goyal, Jingfei Du, Mandar Joshi, Danqi Chen, Omer Levy, Mike Lewis, Luke Zettlemoyer, and Veselin Stoyanov. RoBERTa : A robustly optimized BERT pretraining approach, 2019. URL http://arxiv.org/abs/1907.11692. arXiv:1907.11692

    2019 arXiv

  29. [37]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In International Conference on Learning Representations (ICLR), 2019. URL https://openreview.net/forum?id=Bkg6RiCqY7

    2019

  30. [38]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. Learning word vectors for sentiment analysis. In Annual Meeting of the Association for Computational Linguistics (ACL), 2011. URL http://www.aclweb.org/anthology/P11-1015

    2011

  31. [39]

    Guerreiro, Ricardo Rei, Duarte M

    Pedro Henrique Martins, Patrick Fernandes, João Alves, Nuno M. Guerreiro, Ricardo Rei, Duarte M. Alves, José Pombal, Amin Farajian, Manuel Faysse, Mateusz Klimaszewski, Pierre Colombo, Barry Haddow, José G. C. de Souza, Alexandra Birch, and André F. T. Martins. Euro LLM : Mult...

    2024 arXiv

  32. [40]

    Merging models with F isher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with F isher-weighted averaging. In Advances in Neural Information Processing Systems (NeurIPS), 2022. URL https://openreview.net/forum?id=LSKlp_aceOC

    2022

  33. [41]

    Niklas Muennighoff, Thomas Wang, Lintang Sutawika, Adam Roberts, Stella Biderman, Teven Le Scao, M. Saiful Bari, Sheng Shen, Zheng Xin Yong, Hailey Schoelkopf, Xiangru Tang, Dragomir Radev, Alham Fikri Aji, Khalid Almubarak, Samuel Albanie, Zaid Alyafeai, Albert Webson, Edward...

    2023 doi

  34. [42]

    Arthur G. O. Mutambara. Decentralized estimation and control for multisensor systems. Routledge, 1998. URL https://www.routledge.com/Decentralized-Estimation-and-Control-for-Multisensor-Systems/Mutambara/p/book/9780849318658

    1998

  35. [43]

    Reading digits in natural images with unsupervised feature learning

    Yuval Netzer, Tao Wang, Adam Coates, Alessandro Bissacco, Baolin Wu, and Andrew Y Ng. Reading digits in natural images with unsupervised feature learning. In NIPS Workshop on Deep Learning and Unsupervised Feature Learning, 2011. URL http://ufldl.stanford.edu/housenumbers/nips...

    2011

  36. [44]

    The variational gaussian approximation revisited

    Manfred Opper and C \'e dric Archambeau. The variational gaussian approximation revisited. Neural computation, 21 0 (3): 0 786--792, 2009

    2009

  37. [45]

    Task arithmetic in the tangent space: Improved editing of pre-trained models

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. Task arithmetic in the tangent space: Improved editing of pre-trained models. In Thirty-seventh Conference on Neural Information Processing Systems, 2023. URL https://openreview.net/forum?id=0A9f2jZDGW

    2023

  38. [46]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, John Schulman, Jacob Hilton, Fraser Kelton, Luke Miller, Maddie Simens, Amanda Askell, Peter Welinder, Paul F Christiano, Jan Leike, a...

    2022

  39. [47]

    Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales

    Bo Pang and Lillian Lee. Seeing stars: Exploiting class relationships for sentiment categorization with respect to rating scales. In Annual Meeting of the Association for Computational Linguistics (ACL), 2005. URL https://aclanthology.org/P05-1015/

    2005

  40. [48]

    Fine-tuning aligned language models compromises safety, even when users do not intend to! In International Conference on Learning Representations (ICLR), 2024

    Xiangyu Qi, Yi Zeng, Tinghao Xie, Pin-Yu Chen, Ruoxi Jia, Prateek Mittal, and Peter Henderson. Fine-tuning aligned language models compromises safety, even when users do not intend to! In International Conference on Learning Representations (ICLR), 2024. URL https://openreview...

    2024

  41. [49]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, Gretchen Krueger, and Ilya Sutskever. Learning transferable visual models from natural language supervision. In International Con...

    2021

  42. [50]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J. Liu. Exploring the limits of transfer learning with a unified text-to-text transformer. J. Mach. Learn. Res. (JMLR), 21 0 (140): 0 1--67, 2020. URL http://j...

    2020

  43. [51]

    Learning to reweight examples for robust deep learning

    Mengye Ren, Wenyuan Zeng, Bin Yang, and Raquel Urtasun. Learning to reweight examples for robust deep learning. In International Conference on Machine Learning (ICML), 2018

    2018

  44. [52]

    An overview of multi-task learning in deep neural networks

    Sebastian Ruder. An overview of multi-task learning in deep neural networks. arXiv:1706.05098, 2017. URL http://arxiv.org/abs/1706.05098

    2017 arXiv

  45. [53]

    Variational learning is effective for large deep networks

    Yuesong Shen, Nico Daheim, Bai Cong, Peter Nickl, Gian Maria Marconi, Bazan Clement Emile Marcel Raoul, Rio Yokota, Iryna Gurevych, Daniel Cremers, Mohammad Emtiyaz Khan, and Thomas M \"o llenhoff. Variational learning is effective for large deep networks. In International Con...

    2024

  46. [54]

    Manning, Andrew Ng, and Christopher Potts

    Richard Socher, Alex Perelygin, Jean Wu, Jason Chuang, Christopher D. Manning, Andrew Ng, and Christopher Potts. Recursive deep models for semantic compositionality over a sentiment treebank. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2013. URL ...

    2013

  47. [55]

    ZipIt! M erging models from different tasks without training

    George Stoica, Daniel Bolya, Jakob Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. ZipIt! M erging models from different tasks without training. In International Conference on Learning Representations (ICLR). OpenReview.net, 2024. URL https://openreview.net/forum?id=LEYUkvdUhq

    2024

  48. [56]

    Self-influence guided data reweighting for language model pre-training

    Megh Thakkar, Tolga Bolukbasi, Sriram Ganapathy, Shikhar Vashishth, Sarath Chandar, and Partha Talukdar. Self-influence guided data reweighting for language model pre-training. In Conference on Empirical Methods in Natural Language Processing (EMNLP), 2023. URL https://aclanth...

    2023

  49. [57]

    A B ayesian committee machine

    Volker Tresp. A B ayesian committee machine. Neural computation, 2000. URL https://direct.mit.edu/neco/article-abstract/12/11/2719/6426/A-Bayesian-Committee-Machine

    2000

  50. [58]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. Model soups: averaging weights of multiple fine-tuned models improves accuracy without in...

    2022

  51. [59]

    Ehinger, Aude Oliva, and Antonio Torralba

    Jianxiong Xiao, James Hays, Krista A. Ehinger, Aude Oliva, and Antonio Torralba. SUN database: Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2010. URL https://ieeexplore.ieee.org/document/5539970

    2010

  52. [60]

    Data selection for language models via importance resampling

    Sang Michael Xie, Shibani Santurkar, Tengyu Ma, and Percy S Liang. Data selection for language models via importance resampling. In Advances in Neural Information Processing Systems (NeurIPS), 2023. URL https://openreview.net/forum?id=uPSQv0leAu&noteId=3EMr1ZhaRY

    2023

  53. [61]

    Towards few-shot adaptation of foundation models via multitask finetuning

    Zhuoyan Xu, Zhenmei Shi, Junyi Wei, Fangzhou Mu, Yin Li, and Yingyu Liang. Towards few-shot adaptation of foundation models via multitask finetuning. In International Conference on Learning Representations (ICLR), 2024. URL https://openreview.net/forum?id=1jbh2e0b2K

    2024

  54. [62]

    FORML : Learning to reweight data for fairness

    Bobby Yan, Skyler Seto, and Nicholas Apostoloff. FORML : Learning to reweight data for fairness. In ICML DataPerf Workshop, 2022. URL https://arxiv.org/abs/2202.01719

    2022 arXiv

  55. [63]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. In International Conference on Learning Representations (ICLR), 2024

    2024

  56. [64]

    T. Zhang. Theoretical analysis of a class of randomized regularization methods. In Conference on Learning Theory (COLT), 1999. URL https://dl.acm.org/doi/abs/10.1145/307400.307433

    1999

  57. [65]

    Character-level Convolutional Networks for Text Classification

    Xiang Zhang, Junbo Zhao, and Yann LeCun. Character-level Convolutional Networks for Text Classification . In Advances in Neural Information Processing Systems (NeurIPS), 2015. URL https://papers.nips.cc/paper_files/paper/2015/hash/250cf8b51c773f3f8dc8b4be867a9a02-Abstract.html

    2015

Pith tools

Reviewed August 11, 2026 · model on record in the stance chip above.