Pith. sign in

REVIEW 3 major objections 5 minor 29 references

Model merging can be recast as particle swarm search: starting from the experts, their sparsified copies, and the base model, five data-scored steps produce a merged model that beats fixed-recipe baselines on every tested LLM family.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

PSO-Merging applies particle swarm optimization over model weight space, seeded with original and sparsified experts, to build multitask models that outperform existing merging baselines on several language benchmarks.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A practical, reproducible PSO merging recipe whose headline efficiency claim over Evo is undermined by a budget mismatch (PSO gets (2n+1) fitness evaluations per step vs Evo's n). the 3 major comments →

arxiv 2508.19839 v1 pith:AZCWH5T4 submitted 2025-08-27 cs.LG cs.AI

PSO-Merging: Merging Models Based on Particle Swarm Optimization

Kehao Zhang , Shaolei Zhang , Yang Feng This is my paper
classification cs.LG cs.AI
keywords model mergingparticle swarm optimizationmultitask language modelssparsification (DARE)gradient-free optimizationtask vectorslarge language models
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims that merging several task-specialized language models into one multitask model is best done as a guided search rather than with a fixed averaging recipe. It adapts Particle Swarm Optimization to the parameter space: the swarm starts at the expert models, at randomly sparsified copies of them, and at the pre-trained base model, and each candidate is scored by its average performance on a small sample of each task's data. Over a handful of update steps, particles are pulled toward the best positions seen by the swarm and by themselves, and the final best particle becomes the merged model. Across Flan-T5-Base on GLUE and across Llama-2-13B, Llama-3-8B, and Mistral-7B on instruction-following, math, and code tasks, the paper reports the highest average multitask score among the compared methods, with convergence in about five steps and memory requirements of a single inference run. The case being made: data-driven merging need not cost gradients, and very small data samples suffice to steer it.

Core claim

The central claim is that model merging — combining task-specialized experts into one model that handles all tasks — works best when posed as a particle swarm optimization in parameter space. The load-bearing discovery is the initialization: the swarm starts at the experts, at DARE-sparsified copies of them, and at the pre-trained base model. Fitness is the average score on a small optimization set — 50 samples per GLUE task, or a 1:10 training split for the larger models — and each update pulls a particle toward the global and personal bests, a data-guided linear combination plus momentum. After five steps for LLMs (50 for Flan-T5), the global best particle is the merged model; reported ave

What carries the argument

The central object is the particle swarm: 2n+1 candidate merged models in parameter space, each starting from an expert, a sparsified expert, or the pre-trained base model. The PSO velocity update v = w·v_prev + c1·r1·(θ_gbest − θ) + c2·r2·(θ_pbest − θ) pulls each particle toward the swarm's best position and its own best position; expanding the position update shows each step is a linear combination of current position, global best, and personal best plus momentum — the convex recombination Task Arithmetic showed works, now data-guided. The DARE sparsification (random drop-and-rescale at rate 0.8) does double duty: it damps parameter conflict and doubles the particle count, giving the swarm

Load-bearing premise

The small optimization set — 50 labeled samples per GLUE task, or a 1:10 training split for the larger models — scores candidates accurately enough that the particle picked after a few steps is also the best on the held-out test set; if those few hundred examples mislead the scores, the selected merge overfits the proxy and the reported test advantages could shrink or disappear.

What would settle it

Redraw the optimization set many times (different random 50-sample subsets per GLUE task; different 1:10 splits for GSM8K, MBPP, and SciQ) and rerun PSO-Merging from the same experts, then look at the spread of held-out test scores of the selected merged model. If the chosen particle's test performance varies widely across draws or its average falls below the best data-independent baseline, the data-guidance claim fails. A sharper variant: score candidate particles on one sample but select on a disjoint in-domain sample (or an out-of-distribution proxy), and check whether the merged model stil

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Multitask merging becomes practical at LLM scale without backpropagation: the whole search costs forward passes on a few hundred examples, with memory of a single inference run (about 14 GB for three 7B models).
  • The same loop handles more experts with no change: merging four Llama-3-8B experts (adding SciQ) reaches the best average score in the paper, 64.47.
  • Convergence is fast enough to matter operationally: most particles converge within five steps for LLMs, so the claimed advantage is available at a small evaluation budget.
  • The reported edge depends on the initialization composition and the momentum setting: the full seven-particle pool (experts plus sparsified experts plus base model) beats expert-only and sparsified-only pools, and w=0.2 is the only tested momentum value at which all particles converge.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial: because every particle is a linear combination of expert (or sparsified-expert) and base parameters, the method doubles as a probe of parameter-space geometry — if a few data-scored steps beat hand-set interpolation rules, fine-tuned experts likely sit in a connected region where small guided moves stay beneficial; this could be tested directly by tracing the fitness landscape along lin
  • Editorial: the fitness proxy is the point of fragility — the method inherits whatever bias the 50-sample or 1:10 split carries, so a stress test redrawing the optimization set many times (or drawing it from a different distribution) would show how much of the reported edge is genuine transfer versus selection on the proxy.
  • Editorial: an untested efficiency win is to evaluate only the global-best trajectory each step rather than all particles, roughly halving forward passes or permitting twice the steps at the same budget — a direct way to sharpen the paper's efficiency claim.
  • Editorial: the stated limitation — merging experts from different base models or architectures is unexplored — also bounds the method's mechanism, since the velocity arithmetic adds parameter vectors in a shared space; extending PSO-Merging there would first require an aligned parameter space.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes PSO-Merging, a gradient-free, data-driven model merging method. The swarm is initialized with the pre-trained model, the fine-tuned expert models, and DARE-sparsified versions of the experts. PSO is then run for a small number of steps with fitness defined as the average multitask score on a small optimization set, and the global-best particle is returned as the merged model. Experiments are reported on Flan-T5-Base over eight GLUE tasks, on Llama-2-13B, Llama-3-8B, and Mistral-7B-v0.3 over three-task settings (AlpacaEval, MBPP, GSM8K), and on a four-expert Llama-3-8B setting. The paper reports average-score improvements over data-independent baselines, the CMA-ES-based Evo baseline, and, where feasible, gradient-based baselines. Additional analyses study the momentum coefficient, the number of particles, convergence, and memory usage.

Significance. If the experimental evidence is sound, PSO-Merging is a useful contribution: it avoids gradient computation, uses only small optimization sets, and appears to converge quickly. The paper ships code, evaluates on multiple architectures, and includes an analysis of the sparsification mechanism. The main strength is the simple, practical recipe of using the pre-trained and sparsified experts to seed a small PSO search. However, the reported evidence currently contains a load-bearing arithmetic error in the evaluation budget comparison with the Evo baseline, and the stopping rule / hyperparameters appear to be selected with test-set information. These issues must be resolved before the efficiency and general outperformance claims are accepted. The limitations section is honest about the same-base-model scope, which is acceptable for the paper's stated setting.

major comments (3)
  1. [Section 3.2, Eq. (4)] The paper states that the Evo baseline is run with n*S evaluation iterations to 'align the evaluation count with that of PSO-Merging'. But PSO-Merging has m = 1 + 2n particles (Eq. 4) and evaluates every particle at every step, so its budget is m*S = (2n+1)*S function evaluations. For the Llama/Mistral experiments (n=3, S=5), PSO-Merging uses 35 evaluations while Evo is allotted 15; for Flan-T5-Base (n=8, S=50), PSO uses 850 and Evo 400. The reported average-score advantages over Evo in Tables 1-3 (e.g., 56.82 vs 50.18 on Llama-2-13B, 55.35 vs 51.47 on Mistral-7B) could therefore reflect roughly twice as much search effort rather than the PSO update rule. The Evo comparison needs to be rerun with an equal number of function evaluations, or PSO-Merging should be reported with a matched budget, before the efficiency/outperformance claim is supportable.
  2. [Section 4.3 and Section 3.2] The choice S=5 for the LLM experiments appears to be justified using Figure 3, which plots the test-set score of the global-best particle as a function of optimization steps and concludes that 'PSO-Merging achieves satisfactory performance on the test set within just 5 steps'. Selecting the number of steps after observing test performance is a form of test-set leakage for the final reported numbers. The momentum coefficient w=0.2 is also selected from the optimization curves of the same Llama-3-8B setting (Section 4.1). Please select S and w using only optimization/validation splits, and/or report results for a range of S and w with the test set used strictly once.
  3. [Section 3.3, Tables 1-3] All reported numbers are single runs, despite the procedure being stochastic: the sparsification masks in Eq. (3) are random Bernoulli draws, and PSO uses random coefficients r1, r2. With optimization sets as small as 50 samples per GLUE task and 1:10 splits for the LLM tasks, point estimates may be noisy. Several of the headline average improvements are only a few points (e.g., 81.24 vs 79.54 for DELLA-Merging on Flan-T5; 56.82 vs 52.23 for TIES on Llama-2-13B), and without variance estimates or multiple seeds it is unclear whether these differences are meaningful. Please report mean and standard deviation over multiple independent runs, or otherwise quantify the variability of the optimization procedure.
minor comments (5)
  1. [Abstract / Section 3.3] The abstract says 'generally outperforms baseline merging methods', but Table 1 shows PSO-Merging is not best on several individual GLUE tasks (e.g., CoLA 68.17 vs Task Arithmetic 69.13; SST-2 91.06 vs Task Arithmetic 91.74; STSB 71.94 vs DELLA 75.96). The claim is accurate only for the average multitask score. Consider wording this more precisely.
  2. [Section 2.3 / Conclusion] The conclusion says sparsification enables 'a larger number of linearly independent particles'. Random Bernoulli sparsification with different masks makes the task vectors linearly independent with high probability, but this is not shown or guaranteed in the text; please qualify the claim.
  3. [Section 4.5] The memory comparison reports '14GB' for PSO-Merging vs 42GB for Adamerging on three 7B models. Please clarify precision, batch size, and whether this is peak memory for loading all experts sequentially. This would make the efficiency claim reproducible.
  4. [Tables 2 and 3] The column header 'A VG' appears to be a typo for 'AVG'. Also, Figure 3's legend and axis labels should make clear that only the 'Test_score' line is on the test set; the other lines are on the optimization set.
  5. [Related Work] Model Swarms [28] is mentioned as another iterative merging method but is not compared experimentally. A brief sentence explaining why it is excluded (e.g., different problem formulation or scale) would help the reader place PSO-Merging relative to the closest swarm-based prior work.

Circularity Check

0 steps flagged

No significant circularity: PSO-Merging is an empirical search evaluated on held-out test splits; no equation reduces to its inputs.

full rationale

The claimed result is not derived from a first-principles model; it is an empirical search. The merged model is θ_merged = θ^(S)_gbest selected by maximizing f(θ)=1/n Σ score_i(θ) on an optimization split (Sections 2.2 and 3.3), while the reported multitask scores are measured on held-out test splits (Table 5: GSM8K 131 train vs 1,319 test; MBPP 50 vs 500; AlpacaEval 73 vs 732; 50 GLUE training samples per task). The test numbers are therefore not equal to the fitness function by construction. Equations (5)-(7) are the PSO update definition, not a hidden reuse of the output; the sparsification step (Eq. 3) and PSO itself are imported from external prior work with explicit citations, not from the authors' own prior results, and no uniqueness theorem or ansatz is smuggled via self-citation. The only issues I found are experimental-design concerns rather than circularity: Section 4.1 chooses w=0.2 from optimization-set curves on Llama-3-8B, and Section 4.3 justifies S=5 using the test-set convergence curve; and Section 3.2's claim that Evo's n*S evaluations 'aligns the evaluation count with that of PSO-Merging' is arithmetically inconsistent because PSO evaluates m=1+2n particles for S steps (e.g., 7*5=35 evaluations vs 3*5=15 for n=3, S=5). Those points bear on overfitting and budget-matched comparison, but they do not make any reported result equivalent to its input by construction, so the circularity score is 0.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 0 invented entities

The central method relies on standard PSO dynamics, DARE sparsification from prior work, and a domain-specific assumption that small task-data samples guide the search. There are no new physical entities, forces, or conserved quantities. The free parameters are hand-set or chosen from benchmark-specific analysis rather than derived from first principles.

free parameters (5)
  • w (momentum coefficient) = 0.2
    Chosen after sensitivity analysis in Fig. 2 on Llama-3-8B; affects convergence and whether particles optimize.
  • c1 (global best weight) = 2.0
    Standard PSO setting, not tuned.
  • c2 (personal best weight) = 2.0
    Standard PSO setting, not tuned.
  • p (sparsification drop rate) = 0.8
    Fixed DARE sparsification rate from prior work, applied to all methods that use sparsification including ours.
  • S (optimization steps) = 50 for Flan-T5, 5 for Llama/Mistral
    Selected based on convergence analysis (Fig. 3) and compute budget; fewer steps used for larger models.
axioms (4)
  • domain assumption Fine-tuned expert models share the same base model and architecture, so their parameter vectors are directly addable and linearly combinable.
    Sec 2.1 assumes experts theta_t are fine-tuned from the same pre-trained theta_0; the Limitations section acknowledges this is not explored for different bases.
  • domain assumption A small optimization set from each task's training data is a faithful proxy for test performance and does not lead to overfitting during PSO search.
    Sec 3.3 defines fitness using 50 samples per GLUE task and 1:10 splits for LLM tasks; the whole method selects the particle with maximum optimization score.
  • domain assumption DARE sparsification with drop rate p=0.8 preserves each expert's capabilities while reducing parameter conflicts.
    Borrowed from prior work [12] and applied here; the paper relies on this for the sparsified particles to be useful starting points.
  • domain assumption PSO's velocity update with fixed hyperparameters converges to a good merged model within S steps.
    Standard PSO convergence is assumed; only w is analyzed empirically in Sec 4.1, and no convergence proof is given for the parameter-space search.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of PSO-Merging: Merging Models Based on Particle Swarm Optimization." pith.science (2026). https://pith.science/paper/AZCWH5T4

@misc{pith2026250819839,
  author       = {Pith},
  title        = {Pith review of: PSO-Merging: Merging Models Based on Particle Swarm Optimization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AZCWH5T4}},
  note         = {Machine review of arXiv:2508.19839}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Model merging has emerged as an efficient strategy for constructing multitask models by integrating the strengths of multiple available expert models, thereby reducing the need to fine-tune a pre-trained model for all the tasks from scratch. Existing data-independent methods struggle with performance limitations due to the lack of data-driven guidance. Data-driven approaches also face key challenges: gradient-based methods are computationally expensive, limiting their practicality for merging large expert models, whereas existing gradient-free methods often fail to achieve satisfactory results within a limited number of optimization steps. To address these limitations, this paper introduces PSO-Merging, a novel data-driven merging method based on the Particle Swarm Optimization (PSO). In this approach, we initialize the particle swarm with a pre-trained model, expert models, and sparsified expert models. We then perform multiple iterations, with the final global best particle serving as the merged model. Experimental results on different language models show that PSO-Merging generally outperforms baseline merging methods, offering a more efficient and scalable solution for model merging.

Figures

Figures reproduced from arXiv: 2508.19839 by Kehao Zhang, Shaolei Zhang, Yang Feng.

Figure 1
Figure 1. Figure 1: An overview of PSO-Merging. We begin by sparsifying all fine-tuned LLM experts. The [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The score variations on the optimization set for all particles with different [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The score variations on both the opti￾mization set and the test set over the course of 40 optimization steps. Test_score denotes the score of the global best particle on the test set. All other lines represent the scores of different particles on the optimization set. In this section, we investigate the convergence behavior of PSO-Merging. We present in Fig￾ure 3 the variation in the fitness scores of the … view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 4 canonical work pages

  1. [1]

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-V oss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel M. Ziegler, Jeffrey Wu, Clemens Winter, Christopher Hesse, Mark Chen, Eric Sigler, Mateusz Litwi...

    Pith/arXiv arXiv 2020

  2. [2]

    Llama 2: Open foundation and fine-tuned chat models

    Hugo Touvron, Louis Martin, Kevin Stone, Peter Albert, Amjad Almahairi, Yasmine Babaei, Nikolay Bashlykov, Soumya Batra, Prajjwal Bhargava, Shruti Bhosale, Dan Bikel, Lukas Blecher, Cristian Canton-Ferrer, Moya Chen, Guillem Cucurull, David Esiobu, Jude Fernandes, Jeremy Fu, Wenyin Fu, Brian Fuller, Cynthia Gao, Vedanuj Goswami, Naman Goyal, Anthony Harts...

    Pith/arXiv arXiv doi:10.48550/arxiv.2307.09288 2023

  3. [3]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas Wang, Timothée Lacroix, and William El Sayed. Mistral 7b, 2023. URL https://arxi...

    Pith/arXiv arXiv 2023

  4. [4]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ah- mad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Arthur Hinsvark, Arun Rao, Aston Zhang, Aurelien Rodriguez, Austen Gregerson, Ava...

    Pith/arXiv arXiv 2024

  5. [5]

    Zhao, Yanping Huang, Andrew M

    Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Tay, William Fedus, Yunxuan Li, Xuezhi Wang, Mostafa Dehghani, Siddhartha Brahma, Albert Webson, Shixiang Shane Gu, Zhuyun Dai, Mirac Suzgun, Xinyun Chen, Aakanksha Chowdhery, Alex Castro-Ros, Marie Pellat, Kevin Robinson, Dasha Valter, Sharan Narang, Gaurav Mishra, Adams Yu, Vincent Y . Zhao, Yanpi...

    work page 2024

  6. [6]

    Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huajian Xin, Huazuo Gao, Hui Li, Hui Qu, J

    DeepSeek-AI, Aixin Liu, Bei Feng, Bing Xue, Bingxuan Wang, Bochao Wu, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai Dai, Daya Guo, Dejian Yang, Deli Chen, Dongjie Ji, Erhang Li, Fangyun Lin, Fucong Dai, Fuli Luo, Guangbo Hao, Guanting Chen, Guowei Li, H. Zhang, Han Bao, Hanwei Xu, Haocheng Wang, Haowei Zhang, Honghui Ding, Huaj...

  7. [7]

    Direct preference optimization: Your language model is secretly a reward model

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36, 2024

    2024

  8. [8]

    Training language models to follow instructions with human feedback

    Long Ouyang, Jeffrey Wu, Xu Jiang, Diogo Almeida, Carroll Wainwright, Pamela Mishkin, Chong Zhang, Sandhini Agarwal, Katarina Slama, Alex Ray, et al. Training language models to follow instructions with human feedback. Advances in neural information processing systems, 35:27730–27744, 2022

    2022

  9. [9]

    DeepSeek-AI, Daya Guo, Dejian Yang, Haowei Zhang, Junxiao Song, Ruoyu Zhang, Runxin Xu, Qihao Zhu, Shirong Ma, Peiyi Wang, Xiao Bi, Xiaokang Zhang, Xingkai Yu, Yu Wu, Z. F. Wu, Zhibin Gou, Zhihong Shao, Zhuoshu Li, Ziyi Gao, Aixin Liu, Bing Xue, Bingxuan Wang, Bochao Wu, Bei Feng, Chengda Lu, Chenggang Zhao, Chengqi Deng, Chenyu Zhang, Chong Ruan, Damai D...

    Pith/arXiv arXiv 2025

  10. [10]

    Deep model fusion: A survey

    Weishi Li, Yong Peng, Miao Zhang, Liang Ding, Han Hu, and Li Shen. Deep model fusion: A survey. arXiv preprint arXiv:2309.15698, 2023

    Pith/arXiv arXiv 2023

  11. [11]

    Ties-merging: Resolving interference when merging models

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models. In A. Oh, T. Nau- mann, A. Globerson, K. Saenko, M. Hardt, and S. Levine, editors, Advances in Neu- ral Information Processing Systems , volume 36, pages 7093–7115. Curran Associates, Inc., 2023. URL https://proceedings.neu...

    work page 2023

  12. [12]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. In Forty-first Interna- tional Conference on Machine Learning , 2024. URL https://openreview.net/forum? id=fq0NaiU8Ex

    work page 2024

  13. [13]

    Editing models with task arithmetic, 2023

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic, 2023. URL https://arxiv.org/abs/2212.04089

    Pith/arXiv arXiv 2023

  14. [14]

    Merging models with fisher-weighted averaging

    Michael S Matena and Colin A Raffel. Merging models with fisher-weighted averaging. Advances in Neural Information Processing Systems, 35:17703–17716, 2022

    2022

  15. [15]

    Adamerging: Adaptive model merging for multi-task learning

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning. arXiv preprint arXiv:2310.02575, 2023

    Pith/arXiv arXiv 2023

  16. [16]

    Evolutionary optimization of model merging recipes

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes. arXiv preprint arXiv:2403.13187, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    Kennedy and R

    J. Kennedy and R. Eberhart. Particle swarm optimization. In Proceedings of ICNN’95 - International Conference on Neural Networks, volume 4, pages 1942–1948 vol.4, 1995. doi: 10.1109/ICNN.1995.488968

    arXiv 1942

  18. [18]

    Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models

    Jinliang Lu, Ziliang Pang, Min Xiao, Yaochen Zhu, Rui Xia, and Jiajun Zhang. Merge, ensemble, and cooperate! a survey on collaborative strategies in the era of large language models. arXiv preprint arXiv:2407.06089, 2024. 13

    Pith/arXiv arXiv 2024

  19. [19]

    Della-merging: Reducing interference in model merging through magnitude-based sampling

    Pala Tej Deep, Rishabh Bhardwaj, and Soujanya Poria. Della-merging: Reducing interference in model merging through magnitude-based sampling. arXiv preprint arXiv:2406.11617, 2024

    Pith/arXiv arXiv 2024

  20. [20]

    Rankmean: Module-level importance score for merging fine-tuned llm models

    Gabriel Perin, Xuxi Chen, Shusen Liu, Bhavya Kailkhura, Zhangyang Wang, and Brian Gal- lagher. Rankmean: Module-level importance score for merging fine-tuned llm models. In Findings of the Association for Computational Linguistics: ACL 2024, pages 1776–1782, 2024

    work page 2024

  21. [21]

    Fusionbench: A compre- hensive benchmark of deep model fusion

    Anke Tang, Li Shen, Yong Luo, Han Hu, Bo Du, and Dacheng Tao. Fusionbench: A compre- hensive benchmark of deep model fusion. arXiv preprint arXiv:2406.03280, 2024

    arXiv 2024

  22. [22]

    GLUE: A multi-task benchmark and analysis platform for natural language understanding

    Alex Wang, Amanpreet Singh, Julian Michael, Felix Hill, Omer Levy, and Samuel Bowman. GLUE: A multi-task benchmark and analysis platform for natural language understanding. In Tal Linzen, Grzegorz Chrupała, and Afra Alishahi, editors, Proceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP , pages 353– 355, B...

    doi:10.18653/v1/w18-5446 2018

  23. [23]

    Hashimoto

    Xuechen Li, Tianyi Zhang, Yann Dubois, Rohan Taori, Ishaan Gulrajani, Carlos Guestrin, Percy Liang, and Tatsunori B. Hashimoto. Alpacaeval: An automatic evaluator of instruction-following models. https://github.com/tatsu-lab/alpaca_eval, 5 2023

    2023

  24. [24]

    Training verifiers to solve math word problems

    Karl Cobbe, Vineet Kosaraju, Mohammad Bavarian, Mark Chen, Heewoo Jun, Lukasz Kaiser, Matthias Plappert, Jerry Tworek, Jacob Hilton, Reiichiro Nakano, et al. Training verifiers to solve math word problems. arXiv preprint arXiv:2110.14168, 2021

    Pith/arXiv arXiv 2021

  25. [25]

    Program synthesis with large language models

    Jacob Austin, Augustus Odena, Maxwell Nye, Maarten Bosma, Henryk Michalewski, David Dohan, Ellen Jiang, Carrie Cai, Michael Terry, Quoc Le, et al. Program synthesis with large language models. arXiv preprint arXiv:2108.07732, 2021

    Pith/arXiv arXiv 2021

  26. [26]

    xfinder: Robust and pinpoint answer extraction for large language models

    Qingchen Yu, Zifan Zheng, Shichao Song, Zhiyu Li, Feiyu Xiong, Bo Tang, and Ding Chen. xfinder: Robust and pinpoint answer extraction for large language models. arXiv preprint arXiv:2405.11874, 2024

    Pith/arXiv arXiv 2024

  27. [27]

    Crowdsourcing multiple choice science questions

    Johannes Welbl, Nelson F Liu, and Matt Gardner. Crowdsourcing multiple choice science questions. arXiv preprint arXiv:1707.06209, 2017

    Pith/arXiv arXiv 2017

  28. [28]

    Model swarms: Col- laborative search to adapt llm experts via swarm intelligence

    Shangbin Feng, Zifeng Wang, Yike Wang, Sayna Ebrahimi, Hamid Palangi, Lesly Miculicich, Achin Kulshrestha, Nathalie Rauschmayr, Yejin Choi, Yulia Tsvetkov, et al. Model swarms: Col- laborative search to adapt llm experts via swarm intelligence. arXiv preprint arXiv:2410.11163, 2024. A Training Details for Experts Based on Llama-3-8B and Mistral-7B-v0.3 In...

    Pith/arXiv arXiv 2024

  29. [2024]

    URL https://arxiv.org/abs/2412.19437

    Pith/arXiv arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.