Pith. sign in

REVIEW 4 major objections 5 minor 60 references

DivMerge sets task-arithmetic weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model on unlabeled task inputs, and claims this keeps performance high as the number of tasks grows.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

DivMerge learns task-arithmetic merging weights by minimizing Jensen-Shannon divergence between each specialist model and the merged model, improving multi-task performance and scalability.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection DivMerge's divergence objective is genuinely new and the small-model results are clean, but the theory overreaches and the implemented loss is a greedy surrogate of the claimed JS centroid. the 4 major comments →

arxiv 2509.02108 v3 pith:27CJUL25 submitted 2025-09-02 cs.LG

DivMerge: A divergence-based model merging method for multi-tasking

Brahim Touayouch , Lo\"ic Fosse , G\'eraldine Damnati , Gw\'enol\'e Lecorv\'e This is my paper
classification cs.LG
keywords model mergingtask arithmeticJensen-Shannon divergencemulti-task learningweight disentanglementunlabeled datalanguage modelstask interference
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes DivMerge, a model-merging method that sets task-arithmetic coefficients by minimizing the Jensen-Shannon (or KL) divergence between each fine-tuned specialist model and the merged model, evaluated on unlabeled inputs from each task. The authors claim this makes merging robust to task interference: in pairwise GLUE merges their layer-level JS variant reaches 99.18% average normalized performance, and when all seven GLUE tasks are merged it stays near 93%. The same pattern holds on four T5 generation tasks, where it outperforms model averaging, SLERP, TIES, and AdaMerging. Because no labels are needed—only input examples—the method works even when specialists were fine-tuned by someone else. The central claim is that divergence minimization is itself a reliable way to balance task importance and limit interference, not just another heuristic.

Core claim

On the paper's own terms, the discovery is that the coefficients of task arithmetic can be obtained by solving Γ* = argmin_Γ Σ_t D_{X_t}(θ_t ∥ Φ^Γ_n), i.e. by choosing the merged model that minimizes the Jensen-Shannon divergence to each specialist on that specialist's own input distribution. This objective has two formal properties: it is zero exactly when the merge satisfies weight disentanglement (Proposition 1), and for KL divergence it is an M-projection approximation of classical multi-task learning (Proposition 2). The implemented algorithm estimates the divergence by decoding greedily from each specialist and comparing token-level logits to the merged model. Experiments report that t

What carries the argument

The central object is the task vector τ_t = θ_t − θ_0 (the parameter shift from a pretrained model to a fine-tuned specialist) and the task arithmetic merge Φ^Γ_n = θ_0 + Σ_t Γ_t τ_t. The optimization objective is the summed divergence D_{X_t}(θ_t ∥ Φ^Γ_n) over each task's unlabeled input distribution, whose minimizer is a probabilistic centroid of the specialists' output distributions when D is JS. Divergences are estimated at the token level on greedily decoded sequences from each specialist, and coefficients Γ are learned by gradient descent, per task or per layer.

Load-bearing premise

The theory measures divergence over all possible output sequences, but the implementation compares logits on only one greedily decoded (most-likely) sequence per input; if that one trajectory does not represent the model's typical outputs, the learned weights may not be the divergence-minimizing ones the theory assumes.

What would settle it

Run DivMerge on the same GLUE pairs twice: once with the current greedy decoding and once with several stochastically sampled sequences per input, or with a Monte Carlo estimate of the expectation. If the layer-level ANP changes by more than a small margin, or if the sampled-sequence version produces materially different coefficients, then the implemented method is not actually minimizing the JS divergence defined in Eq. 4 and the theoretical link to weight disentanglement breaks. A simpler check is to merge 8 or 10 tasks and see whether the near-93% seven-task ANP continues to hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If DivMerge's central claim holds, multi-task model merging needs no labeled data or task annotations beyond unlabeled inputs from each task, so pre-existing specialists can be combined without access to training labels.
  • Divergence-based coefficient search should degrade more gracefully than prior methods as task count grows; the paper reports average ANP near 93% at seven GLUE tasks versus sharp drops for AdaMerging, TIES, SLERP, and model averaging.
  • Layer-level coefficient granularity consistently improves over task-level coefficients in the paper's experiments.
  • The JS centroid interpretation gives an information-theoretic handle on task importance: the optimizer automatically weights tasks via their input distributions rather than manual balancing.
  • Because any differentiable merging function could replace task arithmetic, the same divergence objective may extend to other merging schemes beyond task arithmetic.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • A natural stress test the paper does not run is LoRA-based merging; if DivMerge's robustness depends on full fine-tuning producing sparse task vectors, its advantage may shrink when specialists are LoRA adapters.
  • The greedy-decoding estimator replaces an expectation over sampled sequences with one trajectory; if greedy trajectories are biased, the learned coefficients Γ may not minimize the true JS divergence, and the theoretical guarantees in Propositions 1–2 would apply to a different objective than the one implemented.
  • The paper's Proposition 3 bound suggests DivMerge should tolerate approximate input distributions; a practical consequence is that synthetic or retrieved proxy inputs for each task could serve as the unlabeled data, widening applicability.
  • The coefficient-independence plots hint that some tasks have stable, task-intrinsic coefficients while others interact; this could be used to predict which task pairs are safe to merge before merging.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes DivMerge, a task-arithmetic model-merging method that sets the merging coefficients Γ by minimizing a sum of divergence terms, sum_t D_Xt(θ_t || Φ_Γ_n), where D is either KL or JS divergence between the output distributions of the specialist and the merged model. The authors claim formal links to weight disentanglement (Proposition 1) and to classical multi-task learning via an M-projection objective (Proposition 2). They evaluate the method on seven GLUE classification tasks with Qwen2.5-0.5B and on four generative tasks with T5-Base, comparing against model averaging, Multi-SLERP, TIES, AdaMerging, and variants. In exhaustive pairwise and multi-task merging experiments they report that the layer-level JS variant reaches 99.18% average ANP on GLUE pairs and remains near 93% when merging all seven tasks, outperforming all baselines. They also report correlation between divergence and cross-task performance, convergence dynamics, and low data requirements. The central claim is that divergence minimization on unlabeled task inputs is a reliable, scalable way to set task-arithmetic coefficients.

Significance. If the empirical results hold, this is a practically useful contribution: the method is simple, label-free, and consistently outperforms strong baselines across all task combinations, with smaller degradation as the number of tasks grows. The evaluation protocol is a strength: all task combinations are tested, confidence intervals are reported, and the generative-task setup uses publicly available checkpoints. The divergence-performance correlation analysis in Section 5.1 is a nice sanity check. However, the theoretical framing currently overclaims. Proposition 1 is not established as stated because it ignores off-support behavior, Proposition 2 is largely definitional, and the implemented loss uses a greedy single-sequence proxy rather than the expectation over sampled sequences required by the stated objective. These issues do not necessarily invalidate the empirical findings, but they weaken the paper's central explanation of why the method works.

major comments (4)
  1. [§3.2 / Appendix A.1, Definition 1 and Eq. (4)] Proposition 1's 'if and only if' is not proven. The argument in A.1 only shows that sum_t D_Xt = 0 iff, for each t, M(x;θ_t)=M(x;Φ_Γ_n) for x in the support S_Xt (up to almost-everywhere equivalence). Definition 1 also requires M(x;Φ_Γ_n)=M(x;θ0) for x outside the union of the task supports. Since Eq. (4) has no term penalizing off-support behavior, a Γ that matches the specialists on S_Xt but behaves arbitrarily off-support yields zero objective while violating weight disentanglement. The proposition should be weakened to 'on the merged tasks' or an explicit off-support penalty should be added.
  2. [§3.2 / Appendix A.2, Proposition 2] Proposition 2 is close to definitional. The proof introduces the 'M-projection multi-task objective' L^M_MT as (1/|T|) Σ H(M(X_t;θ_t), M(X_t;θ)) and then observes that its minimizer is also the minimizer of Σ KL(M(X_t;θ_t)||M(X_t;θ)), which is exactly Eq. (4) with D=KL. This does not establish a quantitative link to the classical multi-task loss L_MT unless one assumes M(X_t;θ_t) equals P_{Y_t|X_t} on the support, and no approximation bound is provided. The claim that the method 'is an approximation of the classical multi-task learning objective' therefore overstates what is shown. Please provide an explicit bound in terms of per-task fine-tuning error, or reframe Proposition 2 as a motivation rather than a theorem.
  3. [Algorithm 1 / Appendix B.1] The implemented optimization uses a greedy single-rollout proxy, not the expectation over sampled sequences used in Eq. (4) and in the formal definition in B.1. For each input x, Algorithm 1 generates one greedy continuation y^x_t from M(x;θ_t) and compares per-token logits on that single path. The paper acknowledges 'for sake of simplicity we stick to some greedy procedure.' Greedy decoding concentrates on one mode of a multi-modal autoregressive distribution, so the optimized Γ is not the JS centroid of the specialists' output distributions, and Propositions 1 and 2 do not transfer to the implemented loss. Since the central claim is that divergence minimization reliably sets task-arithmetic coefficients, the authors should either optimize over sampled rollouts, or present the greedy variant as a heuristic and provide evidence (e.g., correlation with the exact divergence on a subset) th
  4. [Appendix D.2 / Table 9] The text states that hyperparameters 'allow us to maximize multi-task performance on evaluation sets.' If 'evaluation sets' are the same test sets on which ANP is reported, then the headline results in Tables 3, 6, and 7 are selected on the test set, which is not a clean evaluation. At minimum, clarify whether this refers to the validation split, and if test-based selection was used, describe it explicitly or use nested validation for all optimized methods. The reported margins are often only a few points, so this issue is material to the empirical claims.
minor comments (5)
  1. [Table 7] The header of Table 7 appears misformatted: the column grouping 'Adamerging Forward (ours) JS (ours) Adamerging Forward (ours) JS (ours)' is ambiguous and likely omits the intended method names for the Task Level and Layer Level variants. Please fix the table layout.
  2. [Abstract / Conclusion] The claim of robustness 'as the number of tasks grows' is based on up to seven classification tasks and four generation tasks. Consider qualifying this as 'in the tested range' to avoid overgeneralization.
  3. [Appendix E] The statement that 'a wide range of merging methods are based on task arithmetic' and the implicit claim that task arithmetic encompasses most methods are too strong. Proposition 4 only covers the Euclidean centroid; many merging methods (e.g., TIES, DARE, SLERP variants) involve nonlinear operations that are not exactly expressible as task arithmetic with scalar coefficients. Please qualify the claim.
  4. [Section 3, Eq. (4)] The notation in Eq. (4) overloads θ_t and Φ_Γ_n as both parameter vectors and output distributions. Since the divergence is between conditional distributions, consider writing M(·|·;θ_t) and M(·|·;Φ_Γ_n) explicitly in the objective or adding a remark that the notation is shorthand.
  5. [Limitations] Minor wording: 'we tempt to address' should be 'we attempt to address'.

Circularity Check

2 steps flagged

The theoretical grounding is largely definitional: Prop. 1 restates weight disentanglement as the zero of the Eq. 4 objective, and Prop. 2 defines the 'M-projection multi-task objective' to match Eq. 4; the empirical ANP comparisons are independent.

specific steps
  1. self definitional [Sec. 3.2, Proposition 1; proof in App. A.1 (Eq. 4, Definition 1)]
    "Moreover, by the properties of the KL and JS divergences, we have: DXt(θt∥ΦΓn)=0⇔∀x∈ SXt, M(x;θt)=M(x;ΦΓn)"

    Weight disentanglement (Definition 1) is defined as M(x; f(θ0,{τt},Γ)) = M(x; θ0+τt) for x∈S_Xt. The proof shows Eq. 4 is zero exactly when, for every x∈S_Xt, M(x;θt)=M(x;ΦΓn). Since KL and JS are zero only when the two conditional distributions are equal, the 'if and only if' is a restatement of the on-support part of Definition 1 in divergence notation. It does not independently show that the objective forces disentanglement; it names the zero-objective condition. The outside-support clause of Definition 1 is not enforced by Eq. 4, so even the stated equivalence holds only for the part that is definitionally identical to the objective.

  2. self definitional [Sec. 3.2, Proposition 2; proof in App. A.2 (Definition 2 and Eq. 4)]
    "Based on this, we define the M-projection multi-task objective as follows: LM MT(θ)≜1 |T | X t∈T H(M(X t;θ t), M(Xt;θ)). Again, by Lemma 1, we have arg min θ LM MT(θ) = arg min θ KL (M(Xt;θ t)∥M(X t;θ)). This concludes the proof."

    The classical multi-task objective (Definition 2) is L_MT(θ)=1/|T| Σ_t H(P_{Y_t|X_t}, M(X_t;θ)). The proof first notes each specialist θ_t is an M-projection of P_{Y_t|X_t}, then defines a new objective L^M_MT whose minimizer is exactly the KL form of Eq. 4. This is a definition, not a derivation: the true conditional P_{Y_t|X_t} is replaced by M(X_t;θ_t), and no bound or argument shows that argmin KL(M(X_t;θ_t)||M(X_t;θ)) approximates argmin KL(P_{Y_t|X_t}||M(X_t;θ)). The claimed approximation of multi-task learning is therefore true by construction; the name 'M-projection multi-task objective' is introduced to make Eq. 4 its minimizer.

full rationale

The paper's empirical contribution is self-contained: task-arithmetic coefficients are optimized on validation inputs and evaluated by ANP on held-out GLUE and T5 test sets against public baselines (Tables 3, 6, 7, Fig. 2). No fitted parameter is relabeled as a prediction, and no load-bearing conclusion depends on a self-citation (the Fosse et al. reference is used only for task formalism). However, the two theoretical propositions that claim to ground the method in multi-task learning and weight disentanglement are definitional. Proposition 1 equates zero objective value with the on-support output equality that already defines weight disentanglement. Proposition 2 defines a new 'M-projection multi-task objective' whose minimizer is the KL form of Eq. 4 by construction, rather than proving an approximation to the actual multi-task objective. These are genuine circular-derivation steps in the theoretical framing, but they do not invalidate the independent empirical ANP results, so the overall circularity is partial. The greedy single-rollout proxy acknowledged in Sec. B.1 is a theory-implementation gap, not a circularity.

Axiom & Free-Parameter Ledger

5 free parameters · 6 axioms · 0 invented entities

The method itself is a fitting procedure: the merging coefficients Gamma are learned from unlabeled validation inputs. The hand-chosen hyperparameters in Table 9 and the greedy-decoding proxy are the main extraneous inputs. No new physical or conceptual entities are introduced. The key domain assumptions are that validation inputs and greedy trajectories faithfully represent the task distributions, and that fine-tuned models are good proxies for the true task conditionals.

free parameters (5)
  • Merging coefficient initialization = 0.5
    All Gamma_t initialized to 0.5 (Table 9); chosen by hand, affects optimization trajectory and final ANP.
  • Learning rate = 1e-2
    Used for all DivMerge and AdaMerging runs (Table 9); selected by hand to maximize evaluation performance.
  • Number of epochs = 4
    Training budget for coefficient optimization (Table 9); chosen by hand.
  • Batch size = 4 x n
    Batch size scales with number of tasks n (Table 9).
  • Validation dataset size = 200 (task level) / 400 (layer level)
    Number of unlabeled inputs per task used for divergence estimation (Table 9); chosen by hand.
axioms (6)
  • domain assumption All tasks share a common input-output space with X=Y, and models are treated as conditional text distributions (Sec. 3).
    Restricts the method to generative text-to-text tasks; classification is reformulated as label generation.
  • domain assumption Fine-tuned models theta_t are close enough to the true task conditionals P_Yt|Xt for the M-projection objective to approximate the true multi-task objective (Prop. 2).
    The paper states effectiveness depends on the individual fine-tuned models' performance but provides no bound.
  • domain assumption Available validation-set inputs approximate the true task input distribution P_Xt (Sec. 4, Limitations).
    Coefficients are fitted on validation inputs; Proposition 3 bounds the error in terms of total variation but no TV estimate is given.
  • standard math The JS/KL divergence equals zero if and only if the distributions are equal (used in Prop. 1).
    Standard information-theoretic property; valid for KL and JS.
  • domain assumption Task supports are non-overlapping as in Definition 1; merging through task arithmetic preserves off-support outputs.
    Definition 1 inherited from Ortiz-Jimenez et al.; the paper only measures task-vector cosine similarity, not support overlap.
  • ad hoc to paper Greedy single-sequence decoding is a sufficient proxy for the expectation over model-generated sequences in divergence computation (App. B.1).
    The theoretical equations require averaging over y<t ~ M1, but experiments use greedy decoding; no error analysis is given.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of DivMerge: A divergence-based model merging method for multi-tasking." pith.science (2026). https://pith.science/paper/27CJUL25

@misc{pith2026250902108,
  author       = {Pith},
  title        = {Pith review of: DivMerge: A divergence-based model merging method for multi-tasking},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/27CJUL25}},
  note         = {Machine review of arXiv:2509.02108}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Multi-task learning (MTL) is often achieved by merging datasets before fine-tuning, but the growing availability of fine-tuned models has led to new approaches such as model merging via task arithmetic. A major challenge in this setting is task interference, which worsens as the number of tasks increases. We propose a method that merges models trained on different tasks into a single model, maintaining strong performance across all tasks. Our approach leverages Jensen-Shannon divergence to guide the merging process without requiring additional labelled data, and automatically balances task importance. Unlike existing methods, our approach remains robust as the number of tasks grows and consistently outperforms prior work.

Figures

Figures reproduced from arXiv: 2509.02108 by Brahim Touayouch, G\'eraldine Damnati, Gw\'enol\'e Lecorv\'e, Lo\"ic Fosse.

Figure 1
Figure 1. Figure 1: Illustration of the divergence-based model merging method. This figure shows the merging loss associated [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Evolution of the average ANP metric as a function of the number of merged tasks. For each num￾ber of tasks k on the x-axis, several merging experi￾ments were conducted (n k  in total), and we report the 95% confidence interval. tion in the average ANP, which is consistent with the fact that task interference becomes more appar￾ent. We can also observe that, regardless of the number of merged tasks, our me… view at source ↗
Figure 3
Figure 3. Figure 3: ANP metric of merged task pairs with Task Level and Layer Level JS Divergence as a function of training iterations. 0 50 100 150 200 250 300 350 400 Dataset Size 75 76 77 78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 A v g. A N P Model Averaging Multi-SLERP TIES JS Task Level JS Task Level 95% CI JS Layer Level JS Layer Level 95% CI [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Impact of dataset size on the performance [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: DXi (θi∥θj ) values for different divergences. (i corresponds to the row index, while j corresponds to the column index.) the extended sequence x + y. Next, we forward propagate this extended sequence through the second model M2 once, and obtain the probability distributions of the tokens generated by M1. This allows us to compare the token-level distributions of M1 and M2 on the same generated sequence. U… view at source ↗
Figure 6
Figure 6. Figure 6: Cosine similarity matrices of task vectors between different tasks. Left: Similarity between GLUE [PITH_FULL_IMAGE:figures/full_fig_p015_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Evolution of the task coefficients across training iterations. The first graph shows the coefficient assigned [PITH_FULL_IMAGE:figures/full_fig_p017_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Visualization of coefficient values for a fixed reference task versus coefficient values for the remaining [PITH_FULL_IMAGE:figures/full_fig_p018_8.png] view at source ↗
Figure 9
Figure 9. Figure 9: Coefficient values for each task at different T5 checkpoints. Each plot fixes a reference task and compares [PITH_FULL_IMAGE:figures/full_fig_p019_9.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

60 extracted references · 24 canonical work pages · 2 internal anchors

  1. [1]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    Shun-ichi Amari and Hiroshi Nagaoka. 2000. Methods of information geometry, volume 191. American Mathematical Soc

    work page 2000

  4. [4]

    Dana H Ballard. 1987. Modular learning in neural networks. In Proceedings of the sixth National conference on Artificial intelligence-Volume 1, pages 279--284

    work page 1987

  5. [5]

    J. Baxter. 2000. https://doi.org/10.1613/jair.731 A Model of Inductive Bias Learning . Journal of Artificial Intelligence Research, 12:149--198

    doi:10.1613/jair.731 2000

  6. [6]

    Shai Ben-David and Reba Schuller. 2003. https://doi.org/10.1007/978-3-540-45167-9_41 Exploiting Task Relatedness for Multiple Task Learning . In Gerhard Goos, Juris Hartmanis, Jan Van Leeuwen, Bernhard Sch \"o lkopf, and Manfred K. Warmuth, editors, Learning Theory and Kernel Machines , volume 2777, pages 567--580. Springer Berlin Heidelberg, Berlin, Heidelberg

    work page doi:10.1007/978-3-540-45167-9_41 2003

  7. [7]

    Yochai Blau and Tomer Michaeli. 2018. The perception-distortion tradeoff. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 6228--6237

    work page 2018

  8. [8]

    Yochai Blau and Tomer Michaeli. 2019. Rethinking lossy compression: The rate-distortion-perception tradeoff. In International Conference on Machine Learning, pages 675--685. PMLR

    work page 2019

  9. [9]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, and 1 others. 2020. Language models are few-shot learners. Advances in neural information processing systems, 33:1877--1901

    2020

  10. [10]

    Rich Caruana. 1997. https://doi.org/10.1023/A:1007379606734 Multitask Learning . Machine Learning, 28(1):41--75

    doi:10.1023/a:1007379606734 1997

  11. [11]

    Alexandra Chronopoulou, Jonas Pfeiffer, Joshua Maynez, Xinyi Wang, Sebastian Ruder, and Priyanka Agrawal. 2023. https://arxiv.org/abs/2311.09344 Language and Task Arithmetic with Parameter-Efficient Layers for Zero-Shot Summarization . Preprint, arXiv:2311.09344

    Pith/arXiv arXiv 2023

  12. [12]

    T. M. Cover and J. A. Thomas. 1991. Elements of Information Theory. John Wiley & Sons, Inc

    work page 1991

  13. [13]

    Imre Csisz \'a r. 1975. I-divergence geometry of probability distributions and minimization problems. The annals of probability, pages 146--158

    work page 1975

  14. [14]

    Dietterich

    Thomas G. Dietterich. 2000. Ensemble methods in machine learning. In Multiple Classifier Systems, pages 1--15, Berlin, Heidelberg. Springer Berlin Heidelberg

    work page 2000

  15. [15]

    Olivier Ferret. 2025. Projeter pour mieux fusionner: une histoire de bandit et de lit

    work page 2025

  16. [16]

    Chris Fifty, Ehsan Amid, Zhe Zhao, Tianhe Yu, Rohan Anil, and Chelsea Finn. 2021. Efficiently Identifying Task Groupings for Multi-Task Learning . In Advances in Neural Information Processing Systems , volume 34, pages 27503--27516. Curran Associates, Inc

    work page 2021

  17. [17]

    Lo \" c Fosse, Fr \'e d \'e ric B \'e chet, Beno \^ t Favre, G \'e raldine Damnati, Gw \'e nol \'e Lecorv \'e , Maxime Darrin, Philippe Formont, and Pablo Piantanida. 2025. Statistical deficiency for task inclusion estimation. arXiv preprint arXiv:2503.05491

    work page arXiv 2025

  18. [18]

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. 2024. Arcee's mergekit: A toolkit for merging large language models. arXiv preprint arXiv:2403.13257

    Pith/arXiv arXiv 2024

  19. [19]

    Karsten Grove and Hermann Karcher. 1973. How to conjugate c 1-close group actions. Mathematische Zeitschrift, 132(1):11--20

    work page 1973

  20. [20]

    Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, Weizhu Chen, and 1 others. 2022. Lora: Low-rank adaptation of large language models. ICLR, 1(2):3

    2022

  21. [21]

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. 2022. Editing models with task arithmetic. arXiv preprint arXiv:2212.04089

    Pith/arXiv arXiv 2022

  22. [22]

    Pavel Izmailov, Dmitrii Podoprikhin, Timur Garipov, Dmitry Vetrov, and Andrew Gordon Wilson. 2018. Averaging weights leads to wider optima and better generalization. arXiv preprint arXiv:1803.05407

    Pith/arXiv arXiv 2018

  23. [23]

    Young Kyun Jang, Dat Huynh, Ashish Shah, Wen-Kai Chen, and Ser-Nam Lim. 2024. https://arxiv.org/abs/2405.00571 Spherical linear interpolation and text-anchoring for zero-shot composed image retrieval . Preprint, arXiv:2405.00571

    Pith/arXiv arXiv 2024

  24. [24]

    Wooseong Jeong and Kuk-Jin Yoon. 2025. Selective task group updates for multi-task optimization. arXiv preprint arXiv:2502.11986

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  25. [25]

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. 2022. Dataless knowledge fusion by merging weights of language models. arXiv preprint arXiv:2212.09849

    Pith/arXiv arXiv 2022

  26. [26]

    Tushar Khot, Peter Clark, Michal Guerquin, Peter Jansen, and Ashish Sabharwal. 2020. Qasc: A dataset for question answering via sentence composition. arXiv:1910.11473v2

    Pith/arXiv arXiv 2020

  27. [27]

    Hyoseo Kim, Dongyoon Han, and Junsuk Choe. 2024. Negmerge: Consensual weight negation for strong machine unlearning. arXiv preprint arXiv:2410.05583

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  28. [28]

    Diederik P Kingma and Jimmy Ba. 2014. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980

    Pith/arXiv arXiv 2014

  29. [29]

    Solomon Kullback and Richard A Leibler. 1951. On information and sufficiency. The annals of mathematical statistics, 22(1):79--86

    1951

  30. [30]

    Kevin Kuo, Amrith Setlur, Kartik Srinivas, Aditi Raghunathan, and Virginia Smith. 2025. Exact unlearning of finetuning data via model merging at scale. arXiv preprint arXiv:2504.04626

    Pith/arXiv arXiv 2025

  31. [31]

    Bill Yuchen Lin, Wangchunshu Zhou, Ming Shen, Pei Zhou, Chandra Bhagavatula, Yejin Choi, and Xiang Ren. 2020. https://doi.org/10.18653/v1/2020.findings-emnlp.165 C ommon G en: A constrained text generation challenge for generative commonsense reasoning . In Findings of the Association for Computational Linguistics: EMNLP 2020, pages 1823--1840, Online. As...

    doi:10.18653/v1/2020.findings-emnlp.165 2020

  32. [32]

    Chin-Yew Lin. 2004. https://aclanthology.org/W04-1013/ ROUGE : A package for automatic evaluation of summaries . In Text Summarization Branches Out, pages 74--81, Barcelona, Spain. Association for Computational Linguistics

    2004

  33. [33]

    Maas, Raymond E

    Andrew L. Maas, Raymond E. Daly, Peter T. Pham, Dan Huang, Andrew Y. Ng, and Christopher Potts. 2011. http://www.aclweb.org/anthology/P11-1015 Learning word vectors for sentiment analysis . In Proceedings of the 49th Annual Meeting of the Association for Computational Linguistics: Human Language Technologies, pages 142--150, Portland, Oregon, USA. Associa...

    2011

  34. [34]

    Michael S Matena and Colin A Raffel. 2022. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703--17716

    2022

  35. [35]

    Andreas Maurer, Massimiliano Pontil, and Bernardino Romera-Paredes. 2016. The benefit of multitask representation learning. Journal of Machine Learning Research, 17(81):1--32

    work page 2016

  36. [36]

    Frank Nielsen. 2020. On a generalization of the jensen--shannon divergence and the jensen--shannon centroid. Entropy, 22(2):221

    work page 2020

  37. [37]

    Guillermo Ortiz-Jimenez, Alessandro Favero, and Pascal Frossard. 2023. Task arithmetic in the tangent space: Improved editing of pre-trained models. Advances in Neural Information Processing Systems, 36:66727--66754

    work page 2023

  38. [38]

    Jonas Pfeiffer, Aishwarya Kamath, Andreas R \"u ckl \'e , Kyunghyun Cho, and Iryna Gurevych. 2021. https://doi.org/10.48550/arXiv.2005.00247 AdapterFusion : Non-Destructive Task Composition for Transfer Learning . Preprint, arXiv:2005.00247

    Pith/arXiv arXiv doi:10.48550/arxiv.2005.00247 2021

  39. [39]

    Jonas Pfeiffer, Sebastian Ruder, Ivan Vuli \'c , and Edoardo Maria Ponti. 2023. https://arxiv.org/abs/2302.11529 Modular Deep Learning . Preprint, arXiv:2302.11529

    Pith/arXiv arXiv 2023

  40. [40]

    Colin Raffel, Noam Shazeer, Adam Roberts, Katherine Lee, Sharan Narang, Michael Matena, Yanqi Zhou, Wei Li, and Peter J Liu. 2020. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of machine learning research, 21(140):1--67

    2020

  41. [41]

    Pranav Rajpurkar, Jian Zhang, Konstantin Lopyrev, and Percy Liang. 2016. https://doi.org/10.18653/v1/D16-1264 SQ u AD : 100,000+ questions for machine comprehension of text . In Proceedings of the 2016 Conference on Empirical Methods in Natural Language Processing, pages 2383--2392, Austin, Texas. Association for Computational Linguistics

    doi:10.18653/v1/d16-1264 2016

  42. [42]

    Henry Scheff \'e . 1947. https://arxiv.org/abs/2235739 A Useful Convergence Theorem for Probability Distributions . The Annals of Mathematical Statistics, 18(3):434--438

    work page arXiv 1947

  43. [43]

    Zhang Shengyu, Dong Linfeng, Li Xiaoya, Zhang Sen, Sun Xiaofei, Wang Shuhe, Li Jiwei, Runyi Hu, Zhang Tianwei, Fei Wu, and 1 others. 2023. Instruction tuning for large language models: A survey. arXiv preprint arXiv:2308.10792

    arXiv 2023

  44. [44]

    Trevor Standley, Amir Zamir, Dawn Chen, Leonidas Guibas, Jitendra Malik, and Silvio Savarese. 2020. Which tasks should be learned together in multi-task learning? In International conference on machine learning, pages 9120--9132. PMLR

    work page 2020

  45. [45]

    Charles Stein. 1956. Inadmissibility of the usual estimator for the mean of a multivariate normal distribution. In Proceedings of the third Berkeley symposium on mathematical statistics and probability, volume 1: Contributions to the theory of statistics, volume 3, pages 197--207. University of California Press

    work page 1956

  46. [46]

    Gilbert W Stewart. 1993. On the early history of the singular value decomposition. SIAM review, 35(4):551--566

    work page 1993

  47. [47]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, ukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. Advances in neural information processing systems, 30

    2017

  48. [48]

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel R. Bowman. 2019. https://arxiv.org/abs/1804.07461 Glue: A multi-task benchmark and analysis platform for natural language understanding . Preprint, arXiv:1804.07461

    Pith/arXiv arXiv 2019

  49. [49]

    Ke Wang, Nikolaos Dimitriadis, Guillermo Ortiz-Jimenez, Fran c ois Fleuret, and Pascal Frossard. 2024. Localizing task information for improved model merging and compression. arXiv preprint arXiv:2405.07813

    Pith/arXiv arXiv 2024

  50. [50]

    Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Clement Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, and 1 others. 2019. Huggingface's transformers: State-of-the-art natural language processing. arXiv preprint arXiv:1910.03771

    Pith/arXiv arXiv 2019

  51. [51]

    Andrew KC Wong and Manlai You. 1985. Entropy and distance of random graphs with application to structural pattern recognition. IEEE transactions on pattern analysis and machine intelligence, (5):599--609

    work page 1985

  52. [52]

    Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt

    Mitchell Wortsman, Gabriel Ilharco, Samir Yitzhak Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S. Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, and Ludwig Schmidt. 2022. https://arxiv.org/abs/2203.05482 Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time . Preprint,...

    Pith/arXiv arXiv 2022

  53. [53]

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin Raffel, and Mohit Bansal. 2023. https://arxiv.org/abs/2306.01708 Ties-merging: Resolving interference when merging models . Preprint, arXiv:2306.01708

    Pith/arXiv arXiv 2023

  54. [54]

    An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, and 22 others. 2024 a . Qwen2.5 technical report. arXiv preprint arXiv:2412.15115

    Pith/arXiv arXiv 2024

  55. [56]

    Enneng Yang, Li Shen, Guibing Guo, Xingwei Wang, Xiaochun Cao, Jie Zhang, and Dacheng Tao. 2024 c . Model merging in llms, mllms, and beyond: Methods, theories, applications and opportunities. arXiv preprint arXiv:2408.07666

    Pith/arXiv arXiv 2024

  56. [57]

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. 2024 d . https://arxiv.org/abs/2310.02575 Adamerging: Adaptive model merging for multi-task learning . Preprint, arXiv:2310.02575

    Pith/arXiv arXiv 2024

  57. [58]

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. 2024. https://arxiv.org/abs/2311.03099 Language models are super mario: Absorbing abilities from homologous models as a free lunch . Preprint, arXiv:2311.03099

    Pith/arXiv arXiv 2024

  58. [59]

    Tianhe Yu, Saurabh Kumar, Abhishek Gupta, Sergey Levine, Karol Hausman, and Chelsea Finn. 2020. Gradient surgery for multi-task learning. Advances in neural information processing systems, 33:5824--5836

    work page 2020

  59. [60]

    Yuefeng Zhang. 2023. A rate-distortion-classification approach for lossy image compression. Digital Signal Processing, 141:104163

    work page 2023

  60. [61]

    Yuyan Zhou, Liang Song, Bingning Wang, and Weipeng Chen. 2024. Metagpt: Merging large language models using model exclusive task arithmetic. arXiv preprint arXiv:2406.11385

    Pith/arXiv arXiv 2024

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.