Pith. sign in

REVIEW 4 major objections 6 minor 70 references

Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A learned selector decides which source LLMs to fuse, and the paper reports it halves the number of degrading tasks.

desk verdict A useful extension of FuseLLM-style fusion with adaptive selection; the method appears to work, but the headline '50% interference reduction' is not causally pinned down by the experiments. read the letter →

arxiv 2505.23844 v1 pith:L7WFPKSX submitted 2025-05-28 cs.CL

Zhenglun Kong , Zheng Zhan , Shiyue Hou , Yifan Gong , Xin Meng , Pengwei Sui , Peiyan Dong , Xuan Shen
show 5 more authors
Zifeng Wang Pu Zhao Hao Tang Stratis Ioannidis Yanzhi Wang
This is my paper
classification cs.CL
keywords knowledgefusionlargelanguagemodelsadaptiveselectioninterferencedynamicweightingmodelintegrationLLMmerging
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Building one stronger language model from several weaker or specialized ones usually means blending all of their next-token probability distributions into the target model. The paper argues that this all-in fusion is the source of the trouble: candidates that are irrelevant to a given task dilute what the target knows, and the paper shows that with FuseLLM the fraction of Big-Bench Hard and MMLU tasks that degrade grows as more models are added. Its proposed framework, Fusion-X, instead learns to score every source model from its token-level probability matrices, selects a subset on each training batch, and fuses only the selected candidates with normalized weights. The authors report that this reduces knowledge interference by up to 50% compared with FuseLLM, with average gains roughly doubled on BBH (2.7% to 5.3% over Llama-2-7B), while keeping the target model's parameter count and memory footprint unchanged.

What carries the argument

The load-bearing mechanism is the Adaptive Selection Network (ASN), a three-layer linear network with GELU activations that maps the concatenated, flattened, layer-normalized probability matrices of all candidate LLMs to one logit per candidate. Softmax turns those logits into selection probabilities, and a threshold $\tau=0.15$ converts them into a per-batch subset of candidates, with the highest-probability candidate used as a fallback so at least one source is always selected. The dynamic weighted fusion step then forms a fused distribution $P_f$ as the weighted sum of the selected candidates' probability matrices, and the feedback loss $L_{\mathrm{feed}} = \mathrm{CV}^2(\{\hat p_j\})$ penalizes concentration of the normalized selection weights. The design work it does is to make both which models fuse and how strongly they fuse depend on the input and the training step, turning fusion into a learned, data-dependent operation.

What would settle it

A reader could test the central claim by replacing the Adaptive Selection Network's scores with random scores drawn from the same distribution and keeping the same threshold and fusion weights: if random selection reproduces the 50% reduction in degrading BBH tasks, the learned signal is not load-bearing. A second check is to shuffle the token-probability entries within each source matrix before feeding them to the ASN; if benchmark gains persist, the network is not actually reading the distributions.

Watch

Extended reading notes

Core claim

The central discovery is that knowledge fusion of LLMs is better framed as a per-sample routing problem than as a fixed blending problem. Fusing all available source models treats every candidate as equally useful, and the paper documents the failure mode: on BBH, FuseLLM finishes below the target model on 10 of 27 tasks, and the degradation ratio grows with more sources. Fusion-X's Adaptive Selection Network takes the flattened and layer-normalized probability matrices of all candidates, produces a softmax score per candidate, and keeps only those above a threshold; the selected distributions are then combined with weights proportional to the normalized scores. A feedback-driven loss, defined as the squared coefficient of variation of the candidate weights, prevents the selector from collapsing onto one small subset. In the paper's experiments this yields an average +5.3% on the 27 BBH tasks over the target model, with only five tasks degrading, which the authors count as a 50% reduction in knowledge interference relative to FuseLLM; similar gains hold at 100M, 3B, 7B, and 8B scales and on MMLU, commonsense, and code benchmarks.

Load-bearing premise

The load-bearing premise is that a three-layer network can learn genuinely useful selection scores from the flattened, token-aligned probability distribution matrices of the source LLMs; if those matrices carry no recoverable signal about which models would harm the target, the adaptive selector cannot outperform fixed or random fusion.

Editorial extensions

If this is right

  • Scaling the number of fused source models can improve the target rather than degrade it: Fusion-X reports rising accuracy when moving from three to five candidates at both 100M and 7B scales, while FuseLLM's degradation ratio rises.
  • Heterogeneous models with different architectures, tokenizers, and vocabularies can be integrated without ensemble memory overhead or weight-merging's uniform-architecture constraint.
  • Training becomes cheaper: the paper reports reaching the same perplexity in about half the training steps and matching FuseLLM-level performance with about three times fewer tokens.
  • Interference can be measured and controlled: the fraction of tasks falling below the target model on BBH drops from ten (FuseLLM) to five (Fusion-X), and the paper interprets this as up to 50% less knowledge interference.
  • The learned selector has interpretable behavior: in the 100M-scale run it consistently favors Tiny-Starcoder over Pythia-160M, indicating the scores track something like per-task utility rather than raw model size.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An unstated consequence of the paper's design is that once the selector is trained, it could be reused at inference time to route each new input to a small subset of source models, turning the framework into a cheap conditional ensemble without further training.
  • The same interference mechanism should appear in knowledge distillation and multi-task continual learning, so a parallel testable extension is to downweight or skip examples from sources whose distributions disagree strongly with the target, rather than always blending them.
  • The paper does not test whether the selection scores match an oracle ranking of source models by per-task validation performance; a reader could check this by correlating ASN probabilities with per-source task accuracies.
  • Because the input to the selector is only probability matrices, the approach may transfer to other sequence models with token-level distributions, such as speech or biological sequence models, provided the token-alignment step is adapted.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes Fusion-X, a framework for aggregating knowledge from multiple source LLMs into a single target LLM. The main components are an adaptive selection network (ASN) that scores source models from their token-level probability matrices, a dynamic weighted fusion step that combines the selected models' distributions, and a feedback loss intended to prevent the selector from collapsing onto a single source. The method is evaluated on commonsense, BBH, MMLU, and code benchmarks across model scales from 160M to 8B, and the authors report consistent gains over continued training and FuseLLM, including a headline claim of a 50% reduction in knowledge interference measured by the number of BBH tasks that degrade relative to the target model. The paper also includes ablations over selection count, selection metric, fusion method, threshold, and the feedback loss, plus a comparison with several ensemble, MoE, and weight-merging baselines.

Significance. If the central claim is validated, the paper would make a useful step toward scalable and flexible integration of heterogeneous LLMs: it targets a real limitation of FuseLLM-style knowledge fusion, and the experimental scope is broad, spanning multiple model families, scales, and benchmarks. The authors provide a code repository and explicit ablations of the main design choices, which are strengths. However, the causal role of the adaptive selector is not yet established: the reported gains could come from the weighted fusion, the continued-training objective, or simple dropping of low-quality sources, and the headline '50% reduction' rests on a binary per-task count without uncertainty quantification. The significance of the contribution is therefore conditional on additional controlled experiments and reproducibility details.

major comments (4)
  1. [§4.1, Eq. (4); Algorithm 1] The ASN architecture as written is not implementable for variable-length sequences. Eq. (4) applies linear layers f1, f2, f3 to the flattened concatenation Pcat of matrices Pi in R^{N×V}; since N, the sequence length, varies across training examples, the input dimension to f1 is not fixed. Algorithm 1 additionally reports logit tensors of shape (L, D, N), which is inconsistent with Eq. (4). Please specify whether the selector is applied per token, per sequence, or with a fixed-length window, and give the exact input/output dimensions and how variable lengths are handled. This detail is necessary to understand what the ASN actually computes and to reproduce the method.
  2. [§5.2, Table 3; §6] The headline '50% reduction in knowledge interference' is a binary count of tasks below the target-model baseline (10 for FuseLLM vs. 5 for Fusion-X) from a single run, with no error bars, multiple seeds, or significance test. At 27 tasks, one task flip changes the reduction from 50% to 40%, and several per-task exact-match scores in Table 3 differ by less than one point. The same fragility applies to the degradation percentages in Fig. 1 and to Table 6. Please report variance over seeds and a statistical test (or at least a confidence interval) for the degradation count.
  3. [§5.2, Table 3; §4.1] The causal role of the ASN is not isolated. Within the knowledge-fusion category, FuseLLM is the only comparison, and the 'Selection count' rows of Table 1 change the selection policy together with the fusion weights and the feedback loss; there is no control in which the learned scores are replaced by random selection or by a fixed rule (e.g., always select the source with lowest validation perplexity) while the remaining components are held fixed. As a result, the gains over FuseLLM could be explained by dropping low-quality sources, by the dynamic weighted fusion, or by the continued-training objective rather than by the learned per-input selection that is the paper's main contribution. Please add such controls.
  4. [Appendix C, Fig. 6] The reported selection distribution is stable over training and concentrated on one or two sources; this is consistent with the ASN having learned a fixed preference rather than a context-dependent relevance signal. The text interprets stability as dynamic adjustment, but no analysis shows how selection varies across inputs or whether the learned scores contain information beyond average source quality. The claim of per-input adaptivity in §4.1 ('based on input data and current learning context') therefore needs supporting evidence, such as per-input selection statistics or a comparison of learned scores with a fixed ranking of source models.
minor comments (6)
  1. [Eq. (10)] The loss D(Tt, Pf) is used without defining D; please specify that it is a cross-entropy or KL divergence between the target and fused distributions.
  2. [Abstract and §5.1] The abstract and conclusion refer to 'three benchmarks', but §5.1 and Appendix G describe four (CS, BBH, MMLU, MultiPL-E); please align the count and the descriptions.
  3. [Table 3] Percentage changes for tasks with near-zero baselines, such as Multistep Arithmetic Two (+1100%), are uninformative and should be accompanied by raw score differences.
  4. [§5.2] The statement that results have a standard deviation of −0.02 to +0.02 lacks seed information; please explain how this was computed and over how many runs.
  5. [Appendix B] The hyperparameters λ_fuse and λ_feed are tuned on 10% of the validation set and only the best configuration is shown; please report sensitivity of the main results to these values.
  6. [Abstract; References] There is a typo ('avaliable') in the abstract/code line, and reference [8] has incomplete author formatting; please fix these.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the adaptive selection network is trained end-to-end on general language and fusion objectives, and the reported knowledge-interference reduction is an empirical held-out result, not a fitted identity.

full rationale

The paper's central mechanism, the Adaptive Selection Network, produces logits z_phi from flattened source-model probability matrices via Eq. (4), and the whole system is trained by minimizing Eq. (10): the LM loss, the fusion cross-entropy loss, and a feedback loss that only encourages balance among selected candidates. There is no term in this objective that encodes the BBH per-task degradation count or the '50% reduction' number reported later. The threshold tau = 0.15 is a scalar hyperparameter chosen by ablations, not a parameter fitted to reproduce the target outcome. Evaluation is conducted on held-out benchmarks (CommonSense, BBH, MMLU, MultiPL-E) against FuseLLM and several other baselines, and the gains are empirical measurements rather than consequences of a definitional identity. The paper's self-citations (e.g., pruning, quantization, and token-reduction works by the same group) appear in related-work enumerations and are not load-bearing for the fusion claim; no uniqueness theorem or prior result by the same authors is invoked to forbid alternatives. The 50% interference reduction is a derived comparison of degraded-task counts, not a fitted quantity, and the paper does not construct any equation in which the prediction equals an input by definition. Therefore no circular step is present; the main weaknesses are evidentiary (e.g., lack of a random-selection control), which concern correctness and support, not circularity.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The method does not introduce new physical or conceptual entities, but depends on several domain assumptions about token alignment, training data representativeness, and the informativeness of output distributions.

free parameters (3)
  • selection threshold τ = 0.15
    Chosen via grid search over {0.2, 0.15, 0.12} (Table 1) to maximize performance.
  • fusion loss weight λ_fuse = 0.1
    Selected through grid search (Appendix B) to balance loss components.
  • feedback loss weight λ_feed = 0.5
    Selected through grid search (Appendix B) to balance loss components.
assumptions (3)
  • domain assumption Token alignment via MinED from FuseLLM preserves enough information across heterogeneous tokenizers.
    The method relies on the MinED alignment algorithm (Appendix F) to compare probability distributions between models with different vocabularies; if alignments are noisy, the selection network's inputs may be misleading.
  • domain assumption The MiniPile training corpus is representative of the tasks evaluated.
    Continual training on MiniPile is used for all methods; if this corpus is not representative, the fused model may not generalize to benchmarks like BBH and MMLU.
  • domain assumption The probability distribution matrices of source LLMs provide sufficient signal for the selection network.
    The ASN operates on these matrices (Section 4.1); if the distributions do not encode which model is useful for a given context, selection cannot improve over uniform fusion.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation." pith.science (2026). https://pith.science/paper/L7WFPKSX

@misc{pith2026250523844,
  author       = {Pith},
  title        = {Pith review of: Enabling Flexible Multi-LLM Integration for Scalable Knowledge Aggregation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/L7WFPKSX}},
  note         = {Machine review of arXiv:2505.23844}
}
read the original abstract

Large language models (LLMs) have shown remarkable promise but remain challenging to continually improve through traditional finetuning, particularly when integrating capabilities from other specialized LLMs. Popular methods like ensemble and weight merging require substantial memory and struggle to adapt to changing data environments. Recent efforts have transferred knowledge from multiple LLMs into a single target model; however, they suffer from interference and degraded performance among tasks, largely due to limited flexibility in candidate selection and training pipelines. To address these issues, we propose a framework that adaptively selects and aggregates knowledge from diverse LLMs to build a single, stronger model, avoiding the high memory overhead of ensemble and inflexible weight merging. Specifically, we design an adaptive selection network that identifies the most relevant source LLMs based on their scores, thereby reducing knowledge interference. We further propose a dynamic weighted fusion strategy that accounts for the inherent strengths of candidate LLMs, along with a feedback-driven loss function that prevents the selector from converging on a single subset of sources. Experimental results demonstrate that our method can enable a more stable and scalable knowledge aggregation process while reducing knowledge interference by up to 50% compared to existing approaches. Code is avaliable at https://github.com/ZLKong/LLM_Integration

Figures

Figures reproduced from arXiv: 2505.23844 by the authors.

Figure 1
Figure 1. Scaling number of fusion candidates. We show accuracy (histogram, higher the better) and the percentage of tasks degrading the baseline (Line chart, lower the better) when integrating three, four, and five LLMs on the BBH and MMLU benchmarks. Dotted lines represent the baseline. Existing solutions, such as ensemble methods [22, 35, 67], enhance predic￾tion performance by aggregating out￾puts from multiple models but… view at source ↗
Figure 2
Figure 2. Overall framework: Multiple LLMs are evaluated and selected based on performance by an adaptive selection network. Top candidates then proceed through a dynamic weighted fusion process guided by a feedback loss to enhance the ability of the target LLM. The lower right shows results on CommonSense, MMLU, and Big-Bench Hard benchmark. process includes inferring all candidate models and then ranking them, which can be … view at source ↗
Figure 3
Figure 3. Results on ME benchmark (3&4 LLM). Big-Bench Hard Evaluation. The results of the Fusion-X model com￾pared to baseline methods on the BBH benchmark few-shot CoT prompting with exact match (EM) are presented in Tab. 3. We report results across all 27 tasks and compare the perfor￾mance against continued training of the target model and FuseLLM, eval￾uating the integration of four LLMs (Llama-2-7B, OpenLLaMA-7B, MPT￾7B,… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Left: Training perplexity. During training, our method exhibits greater consistency than existing methods and requires fewer training steps to achieve comparable perplexity. Right: Scaling number of tokens. Comparison between varying scales of training data on BBH. Num…
Figure 5
Figure 5. Figure 5: Loss grid search. Smaller and darker circle means lower perplexity. Training Procedure. During training, the model processes batches of candidate outputs and rewards. The rewards are first flattened and normalized. The Adaptive Selection Network computes selection prob…
Figure 6
Figure 6. Figure 6: Candidate selection distribution. The Left shows the selection for each training step, and the right shows the proportion of each selection for the training [PITH_FULL_IMAGE:figures/full_fig_p017_6.png]
Figure 7
Figure 7. Figure 7: Comparison of Q&A examples between Llama-2, FuseLLM, and Fusion- [PITH_FULL_IMAGE:figures/full_fig_p020_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

70 extracted references · 32 canonical work pages

  1. [1]

    Evolutionary optimization of model merging recipes.Nature Machine Intelligence, pages 1–10, 2025

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes.Nature Machine Intelligence, pages 1–10, 2025

    work page 2025

  2. [2]

    Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization.Advances in Neural Information Processing Systems, 35:8265–8277, 2022

    Devansh Arpit, Huan Wang, Yingbo Zhou, and Caiming Xiong. Ensemble of averages: Improv- ing model selection and boosting performance in domain generalization.Advances in Neural Information Processing Systems, 35:8265–8277, 2022

    work page 2022

  3. [3]

    Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023

    BIG bench authors. Beyond the imitation game: Quantifying and extrapolating the capabilities of language models.Transactions on Machine Learning Research, 2023

    2023

  4. [4]

    Pythia: A suite for analyzing large language models across training and scaling

    Stella Biderman, Hailey Schoelkopf, Quentin Gregory Anthony, Herbie Bradley, Kyle O’Brien, Eric Hallahan, Mohammad Aflah Khan, Shivanshu Purohit, USVSN Sai Prashanth, Edward Raff, et al. Pythia: A suite for analyzing large language models across training and scaling. In International Conference on Machine Learning, pages 2397–2430. PMLR, 2023

    2023

  5. [5]

    GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021

    Sid Black, Gao Leo, Phil Wang, Connor Leahy, and Stella Biderman. GPT-Neo: Large Scale Autoregressive Language Modeling with Mesh-Tensorflow, March 2021. If you use this software, please cite it using these metadata

    work page 2021

  6. [6]

    Multipl- e: a scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on Software Engineering, 2023

    Federico Cassano, John Gouwar, Daniel Nguyen, Sydney Nguyen, Luna Phipps-Costin, Donald Pinckney, Ming-Ho Yee, Yangtian Zi, Carolyn Jane Anderson, Molly Q Feldman, et al. Multipl- e: a scalable and polyglot approach to benchmarking neural code generation.IEEE Transactions on Software Engineering, 2023

    work page 2023

  7. [7]

    Meditron- 70b: Scaling medical pretraining for large language models, 2023

    Zeming Chen, Alejandro Hernández-Cano, Angelika Romanou, Antoine Bonnet, Kyle Matoba, Francesco Salvi, Matteo Pagliardini, Simin Fan, Andreas Köpf, Amirkeivan Mohtashami, Alexandre Sallinen, Alireza Sakhaeirad, Vinitra Swamy, Igor Krawczuk, Deniz Bayazit, Axel Marmet, Syrielle Montariol, Mary-Anne Hartley, Martin Jaggi, and Antoine Bosselut. Meditron- 70b...

    work page 2023

  8. [8]

    Chinese-vicuna: A chinese instruction-following llama-based model

    Zhenyi Lu Chenghao Fan and Jie Tian. Chinese-vicuna: A chinese instruction-following llama-based model. 2023

    work page 2023

Show all 70 references
  1. [9]

    Med42 – evaluating fine-tuning strategies for medical llms: Full-parameter vs

    Clément Christophe, Praveen K Kanithi, Prateek Munjal, Tathagata Raha, Nasir Hayat, Ronnie Rajan, Ahmed Al-Mahrooqi, Avani Gupta, Muhammad Umar Salman, Gurpreet Gosal, Bhargav Kanakiya, Charles Chen, Natalia Vassilieva, Boulbaba Ben Amor, Marco AF Pimentel, and Shadab Khan. Me...

    2024

  2. [10]

    Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    Peter Clark, Isaac Cowhey, Oren Etzioni, Tushar Khot, Ashish Sabharwal, Carissa Schoenick, and Oyvind Tafjord. Think you have solved question answering? try arc, the ai2 reasoning challenge.arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  3. [11]

    Glam: Efficient scaling of language models with mixture-of-experts

    Nan Du, Yanping Huang, Andrew M Dai, Simon Tong, Dmitry Lepikhin, Yuanzhong Xu, Maxim Krikun, Yanqi Zhou, Adams Wei Yu, Orhan Firat, et al. Glam: Efficient scaling of language models with mixture-of-experts. InInternational Conference on Machine Learning, pages 5547–5569. PMLR, 2022

    2022

  4. [12]

    Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    William Fedus, Barret Zoph, and Noam Shazeer. Switch transformers: Scaling to trillion parameter models with simple and efficient sparsity.Journal of Machine Learning Research, 23(120):1–39, 2022

    2022

  5. [13]

    A framework for few-shot language model evaluation, 12 2023

    Leo Gao, Jonathan Tow, Baber Abbasi, Stella Biderman, Sid Black, Anthony DiPofi, Charles Foster, Laurence Golding, Jeffrey Hsu, Alain Le Noac’h, Haonan Li, Kyle McDonell, Niklas Muennighoff, Chris Ociepa, Jason Phang, Laria Reynolds, Hailey Schoelkopf, Aviya Skowron, Lintang S...

    2023

  6. [14]

    Koala: A dialogue model for academic research

    Xinyang Geng, Arnav Gudibande, Hao Liu, Eric Wallace, Pieter Abbeel, Sergey Levine, and Dawn Song. Koala: A dialogue model for academic research. Blog post, April 2023

    2023

  7. [15]

    Openllama: An open reproduction of llama, May 2023

    Xinyang Geng and Hao Liu. Openllama: An open reproduction of llama, May 2023. 10

    2023

  8. [16]

    Arcee’s mergekit: A toolkit for merging large language models.arXiv preprint arXiv:2403.13257, 2024

    Charles Goddard, Shamane Siriwardhana, Malikeh Ehghaghi, Luke Meyers, Vlad Karpukhin, Brian Benedict, Mark McQuade, and Jacob Solawetz. Arcee’s mergekit: A toolkit for merging large language models.arXiv preprint arXiv:2403.13257, 2024

    2024 arXiv

  9. [17]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  10. [18]

    Measuring massive multitask language understanding, 2021

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding, 2021

    2021

  11. [19]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. stat, 1050:9, 2015

    2015

  12. [20]

    Categorical reparameterization with gumbel-softmax

    Eric Jang, Shixiang Gu, and Ben Poole. Categorical reparameterization with gumbel-softmax. arXiv preprint arXiv:1611.01144, 2016

    2016 arXiv

  13. [21]

    Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    Albert Q Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts.arXiv preprint arXiv:2401.04088, 2024

    2024 arXiv

  14. [22]

    Llm-blender: Ensembling large language models with pairwise ranking and generative fusion

    Dongfu Jiang, Xiang Ren, and Bill Yuchen Lin. Llm-blender: Ensembling large language models with pairwise ranking and generative fusion. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 14165–14178, 2023

    2023

  15. [23]

    Dataless knowledge fusion by merging weights of language models

    Xisen Jin, Xiang Ren, Daniel Preotiuc-Pietro, and Pengxiang Cheng. Dataless knowledge fusion by merging weights of language models. InThe Eleventh International Conference on Learning Representations, 2022

    2022

  16. [24]

    The minipile challenge for data-efficient language models.arXiv preprint arXiv:2304.08442, 2023

    Jean Kaddour. The minipile challenge for data-efficient language models.arXiv preprint arXiv:2304.08442, 2023

    2023 arXiv

  17. [25]

    Token reduction should go beyond efficiency in generative models – from vision, language to multimodality, 2025

    Zhenglun Kong, Yize Li, Fanhu Zeng, Lei Xin, Shvat Messica, Xue Lin, Pu Zhao, Manolis Kellis, Hao Tang, and Marinka Zitnik. Token reduction should go beyond efficiency in generative models – from vision, language to multimodality, 2025

    2025

  18. [26]

    Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    Dmitry Lepikhin, HyoukJoong Lee, Yuanzhong Xu, Dehao Chen, Orhan Firat, Yanping Huang, Maxim Krikun, Noam Shazeer, and Zhifeng Chen. Gshard: Scaling giant models with condi- tional computation and automatic sharding.arXiv preprint arXiv:2006.16668, 2020

    2006 arXiv

  19. [27]

    Distinct but correct: generating diversified and entity-revised medical response

    Bin Li, Bin Sun, Shutao Li, Encheng Chen, Hongru Liu, Yixuan Weng, Yongping Bai, and Meiling Hu. Distinct but correct: generating diversified and entity-revised medical response. Science China Information Sciences, 67(3):132106, 2024

    2024

  20. [28]

    Towards better chinese-centric neural machine translation for low-resource languages.Computer Speech & Language, 84:101566, 2024

    Bin Li, Yixuan Weng, Fei Xia, and Hanjun Deng. Towards better chinese-centric neural machine translation for low-resource languages.Computer Speech & Language, 84:101566, 2024

    2024

  21. [29]

    Efficient transformer-based large scale language representations using hardware-friendly block structured pruning

    Bingbing Li, Zhenglun Kong, Tianyun Zhang, Ji Li, Zhengang Li, Hang Liu, and Caiwen Ding. Efficient transformer-based large scale language representations using hardware-friendly block structured pruning. In Trevor Cohn, Yulan He, and Yang Liu, editors,Findings of the Associat...

    2020

  22. [30]

    Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    Raymond Li, Loubna Ben Allal, Yangtian Zi, Niklas Muennighoff, Denis Kocetkov, Chenghao Mou, Marc Marone, Christopher Akiki, Jia Li, Jenny Chim, et al. Starcoder: may the source be with you!arXiv preprint arXiv:2305.06161, 2023

    2023 arXiv

  23. [31]

    A comprehensive review of multi-agent reinforcement learning in video games.Authorea Preprints, 2025

    Zhengyang Li, Qijin Ji, Xinghong Ling, and Quan Liu. A comprehensive review of multi-agent reinforcement learning in video games.Authorea Preprints, 2025

    2025

  24. [32]

    Rora: Efficient fine-tuning of llm with reliability optimization for rank adaptation.arXiv preprint arXiv:2501.04315, 2025

    Jun Liu, Zhenglun Kong, Peiyan Dong, Xuan Shen, Pu Zhao, Hao Tang, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, et al. Rora: Efficient fine-tuning of llm with reliability optimization for rank adaptation.arXiv preprint arXiv:2501.04315, 2025. 11

    2025 arXiv

  25. [33]

    Toward adaptive large language models structured pruning via hybrid-grained weight importance assessment.arXiv preprint arXiv:2403.10799, 2024

    Jun Liu, Zhenglun Kong, Pu Zhao, Changdi Yang, Hao Tang, Xuan Shen, Geng Yuan, Wei Niu, Wenbin Zhang, Xue Lin, et al. Toward adaptive large language models structured pruning via hybrid-grained weight importance assessment.arXiv preprint arXiv:2403.10799, 2024

    2024

  26. [34]

    Zhengzhong Liu, Aurick Qiao, Willie Neiswanger, Hongyi Wang, Bowen Tan, Tianhua Tao, Junbo Li, Yuqi Wang, Suqi Sun, Omkar Pangarkar, Richard Fan, Yi Gu, Victor Miller, Yonghao Zhuang, Guowei He, Haonan Li, Fajri Koto, Liping Tang, Nikhil Ranjan, Zhiqiang Shen, Xuguang Ren, Rob...

    2023

  27. [35]

    Routing to the expert: Efficient reward-guided ensemble of large language models.arXiv preprint arXiv:2311.08692, 2023

    Keming Lu, Hongyi Yuan, Runji Lin, Junyang Lin, Zheng Yuan, Chang Zhou, and Jingren Zhou. Routing to the expert: Efficient reward-guided ensemble of large language models.arXiv preprint arXiv:2311.08692, 2023

    2023 arXiv

  28. [36]

    Pack of llms: Model fusion at test-time via perplexity optimization.arXiv preprint arXiv:2404.11531, 2024

    Costas Mavromatis, Petros Karypis, and George Karypis. Pack of llms: Model fusion at test-time via perplexity optimization.arXiv preprint arXiv:2404.11531, 2024

    2024 arXiv

  29. [37]

    Specinfer: Accelerating generative llm serving with speculative inference and token tree verification, 2023

    Xupeng Miao, Gabriele Oliaro, Zhihao Zhang, Xinhao Cheng, Zeyu Wang, Rae Ying Yee Wong, Zhuoming Chen, Daiyaan Arfeen, Reyna Abhyankar, and Zhihao Jia. Specinfer: Accelerating generative llm serving with speculative inference and token tree verification, 2023

    2023

  30. [38]

    Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    Todor Mihaylov, Peter Clark, Tushar Khot, and Ashish Sabharwal. Can a suit of armor conduct electricity? a new dataset for open book question answering.arXiv preprint arXiv:1809.02789, 2018

    2018 arXiv

  31. [39]

    Diverse weight averaging for out-of-distribution generalization

    Alexandre Rame, Matthieu Kirchmeyer, Thibaud Rahier, Alain Rakotomamonjy, Patrick Gal- linari, and Matthieu Cord. Diverse weight averaging for out-of-distribution generalization. Advances in Neural Information Processing Systems, 35:10821–10836, 2022

    2022

  32. [40]

    Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    Noam Shazeer, Azalia Mirhoseini, Krzysztof Maziarz, Andy Davis, Quoc Le, Geoffrey Hinton, and Jeff Dean. Outrageously large neural networks: The sparsely-gated mixture-of-experts layer.arXiv preprint arXiv:1701.06538, 2017

    2017 arXiv

  33. [41]

    Agile-quant: Activation-guided quantization for faster inference of llms on the edge

    Xuan Shen, Peiyan Dong, Lei Lu, Zhenglun Kong, Zhengang Li, Ming Lin, Chao Wu, and Yanzhi Wang. Agile-quant: Activation-guided quantization for faster inference of llms on the edge. InProceedings of the AAAI Conference on Artificial Intelligence, volume 38, pages 18944–18951, 2024

    2024

  34. [42]

    Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge.arXiv preprint arXiv:2402.10787, 2024

    Xuan Shen, Zhenglun Kong, Changdi Yang, Zhaoyang Han, Lei Lu, Peiyan Dong, Cheng Lyu, Chih-hsiang Li, Xuehang Guo, Zhihao Shu, et al. Edgeqat: Entropy and distribution guided quantization-aware training for the acceleration of lightweight llms on the edge.arXiv preprint arXiv:...

    2024 arXiv

  35. [43]

    Knowledge unlearning for llms: Tasks, methods, and challenges.arXiv preprint arXiv:2311.15766, 2023

    Nianwen Si, Hao Zhang, Heyu Chang, Wenlin Zhang, Dan Qu, and Weiqiang Zhang. Knowledge unlearning for llms: Tasks, methods, and challenges.arXiv preprint arXiv:2311.15766, 2023

    2023 arXiv

  36. [44]

    Zipit! merging models from different tasks without training

    George Stoica, Daniel Bolya, Jakob Brandt Bjorner, Pratik Ramesh, Taylor Hearn, and Judy Hoffman. Zipit! merging models from different tasks without training. InThe Twelfth International Conference on Learning Representations, 2023

    2023

  37. [45]

    Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    Mirac Suzgun, Nathan Scales, Nathanael Schärli, Sebastian Gehrmann, Yi Tay, Hyung Won Chung, Aakanksha Chowdhery, Quoc V Le, Ed H Chi, Denny Zhou, , and Jason Wei. Challenging big-bench tasks and whether chain-of-thought can solve them.arXiv preprint arXiv:2210.09261, 2022

    arXiv

  38. [46]

    Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  39. [47]

    Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023

    MosaicML NLP Team. Introducing mpt-7b: A new standard for open-source, commercially usable llms, 2023. Accessed: 2023-05-05. 12

    2023

  40. [48]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    2023 arXiv

  41. [49]

    Knowl- edge fusion of large language models

    Fanqi Wan, Xinting Huang, Deng Cai, Xiaojun Quan, Wei Bi, and Shuming Shi. Knowl- edge fusion of large language models. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  42. [50]

    Fusechat: Knowledge fusion of chat models.arXiv preprint arXiv:2402.16107, 2024

    Fanqi Wan, Ziyi Yang, Longguang Zhong, Xiaojun Quan, Xinting Huang, and Wei Bi. Fusechat: Knowledge fusion of chat models.arXiv preprint arXiv:2402.16107, 2024

    2024 arXiv

  43. [51]

    Fusing models with complementary expertise.arXiv preprint arXiv:2310.01542, 2023

    Hongyi Wang, Felipe Maia Polo, Yuekai Sun, Souvik Kundu, Eric Xing, and Mikhail Yurochkin. Fusing models with complementary expertise.arXiv preprint arXiv:2310.01542, 2023

    2023 arXiv

  44. [52]

    Learn it or leave it: Module composition and pruning for continual learning

    Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, and Hinrich Schütze. Learn it or leave it: Module composition and pruning for continual learning. InProceedings of the 9th Workshop on Representation Learning for NLP (RepL4NLP-2024), pages 163–176, 2024

    2024

  45. [53]

    Rehearsal- free modular and compositional continual learning for language models

    Mingyang Wang, Heike Adel, Lukas Lange, Jannik Strötgen, and Hinrich Schütze. Rehearsal- free modular and compositional continual learning for language models. InProceedings of the 2024 Conference of the North American Chapter of the Association for Computational Linguistics: ...

    2024

  46. [54]

    A systematic review of machine learning applications in infectious disease prediction, diagnosis, and outbreak forecasting

    Yiting Wang, Jiachen Zhong, and Rohan Kumar. A systematic review of machine learning applications in infectious disease prediction, diagnosis, and outbreak forecasting. 2025

    2025

  47. [55]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  48. [56]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    2024

  49. [57]

    Adamerging: Adaptive model merging for multi-task learning.arXiv preprint arXiv:2310.02575, 2023

    Enneng Yang, Zhenyi Wang, Li Shen, Shiwei Liu, Guibing Guo, Xingwei Wang, and Dacheng Tao. Adamerging: Adaptive model merging for multi-task learning.arXiv preprint arXiv:2310.02575, 2023

    2023 arXiv

  50. [58]

    Yi: Open foundation models by 01

    Alex Young, Bei Chen, Chao Li, Chengen Huang, Ge Zhang, Guanwei Zhang, Guoyin Wang, Heng Li, Jiangcheng Zhu, Jianqun Chen, et al. Yi: Open foundation models by 01. ai.arXiv preprint arXiv:2403.04652, 2024

    2024 arXiv

  51. [59]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  52. [60]

    Rethinking token reduction for state space models.arXiv preprint arXiv:2410.14725, 2024

    Zheng Zhan, Yushu Wu, Zhenglun Kong, Changdi Yang, Yifan Gong, Xuan Shen, Xue Lin, Pu Zhao, and Yanzhi Wang. Rethinking token reduction for state space models.arXiv preprint arXiv:2410.14725, 2024

    2024 arXiv

  53. [61]

    Towards the law of capacity gap in distilling language models

    Chen Zhang, Dawei Song, Zheyu Ye, and Yan Gao. Towards the law of capacity gap in distilling language models. 2023

    2023

  54. [62]

    Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    Jinghan Zhang, Junteng Liu, Junxian He, et al. Composing parameter-efficient modules with arithmetic operation.Advances in Neural Information Processing Systems, 36:12589–12610, 2023

    2023

  55. [63]

    Alpacare:instruction-tuned large language models for medical application, 2023

    Xinlu Zhang, Chenxin Tian, Xianjun Yang, Lichang Chen, Zekun Li, and Linda Ruth Petzold. Alpacare:instruction-tuned large language models for medical application, 2023

    2023

  56. [64]

    7b fully open source moxin-llm – from pretraining to grpo-based reinforcement learning enhancement, 2025

    Pu Zhao, Xuan Shen, Zhenglun Kong, Yixin Shen, Sung-En Chang, Timothy Rupprecht, Lei Lu, Enfu Nan, Changdi Yang, Yumei He, Weiyan Shi, Xingchen Xu, Yu Huang, Wei Jiang, Wei Wang, Yue Chen, Yong He, and Yanzhi Wang. 7b fully open source moxin-llm – from pretraining to grpo-base...

    2025

  57. [65]

    Pruning foundation models for high accuracy without retraining.arXiv preprint arXiv:2410.15567, 2024

    Pu Zhao, Fei Sun, Xuan Shen, Pinrui Yu, Zhenglun Kong, Yanzhi Wang, and Xue Lin. Pruning foundation models for high accuracy without retraining.arXiv preprint arXiv:2410.15567, 2024

    2024 arXiv

  58. [66]

    Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, et al. Judging llm-as-a-judge with mt-bench and chatbot arena.Advances in Neural Information Processing Systems, 36:46595–46623, 2023

    2023

  59. [67]

    Enhancing thyroid disease prediction using machine learning: A comparative study of ensemble models and class balancing techniques

    Jiachen Zhong and Yiting Wang. Enhancing thyroid disease prediction using machine learning: A comparative study of ensemble models and class balancing techniques. 2025

    2025

  60. [68]

    St-moe: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906, 2022

    Barret Zoph, Irwan Bello, Sameer Kumar, Nan Du, Yanping Huang, Jeff Dean, Noam Shazeer, and William Fedus. St-moe: Designing stable and transferable sparse expert models.arXiv preprint arXiv:2202.08906, 2022. 14 A Design Details Adaptive Selection Network’s Decision-making Pro...

    2022 arXiv

  61. [70]

    Building on this foun- dation, GShard [26] and Switch Transformers [12] presented some of the first large-scale models leveraging SMoE

    introduced the concept of Sparsely-gated Mixture-of-Experts (SMoE). Building on this foun- dation, GShard [26] and Switch Transformers [12] presented some of the first large-scale models leveraging SMoE. This technique reduces computational overhead by dynamically routing inpu...

  62. [2020]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.