Pith. sign in

REVIEW 3 major objections 5 minor 65 references

CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation

T0 review · 3 major / 5 minor · reviewed 2026-08-15 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read CABS+ claims that conflict-aware sparsification followed by a gradient-free covariance-adapted search for task scaling coefficients makes multi-task model merging more accurate and cheap enough for a single V100 GPU.

desk verdict Solid incremental extension of CABS with a real efficiency win, but the headline gain over AdaMerging is inflated by an undisclosed label-regime difference and by averaging relative improvements. read the letter →

arxiv 2608.12842 v1 pith:V6BHQU57 submitted 2026-08-13 cs.AI

Yuchen Liu , Zongzhen Yang , Binhang Qi , Hailong Sun , Xiang Gao This is my paper · ORCID
classification cs.AI
keywords modelmergingtaskvectorsconflict-awaresparsificationadaptiveweightallocationCMA-ESgradient-freeoptimizationmergeabilityRelativeSynergyScore
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper claims model merging can be both more effective and dramatically cheaper by pairing conflict-aware sparsification with a gradient-free, memory-light search for task scaling coefficients. It extends the earlier CABS method, replacing exponential grid search with a CMA-ES-style Adaptive Weight Allocation and an asymmetric fitness function that keeps losses from a few tasks from dominating the coefficient search. Across 27 datasets and 5 model families, it reports average gains of 16.97% over AdaMerging and 12.93% over WUDIMerging, using less than 25% of AdaMerging's GPU memory and achieving nearly a 4x time speedup over WUDIMerging. The paper also proposes the Relative Synergy Score as a pre-merge measure of how compatible two task models are, and identifies six factors—learning rate, training epochs, task and data heterogeneity, architecture, and scale—that predict mergeability. If these results hold, model merging becomes practical on modest GPUs and can be guided by measurable compatibility checks before any merging compute is spent.

What carries the argument

The central object is the merged model $$\theta_{\text{merged}} = \theta_{\text{base}} + \sum_{t=1}^{T} \lambda_t^* \, \tilde{\tau}_t,$$ where $\tilde{\tau}_t$ are task vectors after CABS's sequential, block-wise $n{:}m$ pruning. Three pieces carry the argument. First, the sequential mask generated from task vector A and used to prune task vector B makes the pruned vectors orthogonal under the Frobenius inner product, removing the cross term from the merged update norm and enabling independent scaling of task contributions. Second, the AWA search distribution is a multivariate normal updated by CMA-ES, and its learned covariance can capture correlations between task coefficients that grid search cannot exploit. Third, the asymmetric fitness function $f_t(\lambda)$, which penalizes a task's relative loss increase with $\alpha=100$ and rewards a loss decrease with only $\beta=1$, prevents any single task with a large loss scale from dominating the search.

What would settle it

Run CABS+ on a task suite while withholding labels from the fitness evaluation, using only unlabeled inputs as AdaMerging does, and compare average task accuracy; if the method depends on labels, it should degrade toward or below AdaMerging's level, contradicting the claimed 16.97% advantage. Separately, measure CABS+ GPU memory while doubling the number of task vectors on a 7B model; if memory usage grows linearly rather than staying flat, the claimed O(1) memory advantage fails.

Watch

Extended reading notes

Core claim

CABS+ argues that the main obstacles to task-vector model merging—parameter overlap, scale differences among task losses, and exponential coefficient search—can be removed in one pipeline. Conflict-aware sparsification performs sequential pruning: task vector A is pruned with block-wise n:m sparsification, its mask is used to remove overlapping positions in task vector B, and sign selection plus averaging handles any unavoidable residual overlap. The paper states this makes the pruned task vectors orthogonal in the Frobenius inner product, eliminating the cross term in the merged update norm and letting each task's contribution be scaled independently. Adaptive Weight Allocation then optimizes the scaling coefficients with a gradient-free covariance-matrix-adaptation evolution strategy, bounding coefficients to [0.1, 2] and scoring each candidate by a relative, asymmetric loss change: a task whose loss rises is penalized 100 times more than a task whose loss falls. The paper reports that the resulting method outperforms AdaMerging and WUDIMerging by 16.97% and 12.93% on average, stays competitive with or above CABS on every tested configuration, and in large-model experiments uses less than 25% of AdaMerging's GPU memory and roughly a quarter of WUDIMerging's merging time.

Load-bearing premise

The AWA search scores each candidate by the loss of the merged model on test samples, so it requires labeled evaluation data for every task; if such labels are unavailable, the coefficient search cannot be run as described.

Editorial extensions

If this is right

  • Merging seven-billion-parameter models becomes feasible on a 32GB V100 GPU: CABS+ keeps GPU memory at inference level, reported as O(1) in the number of tasks, while AdaMerging's gradient graph grows linearly with tasks, depth, and sequence length.
  • The number of coefficient evaluations drops from $O(S^T)$ for grid search to $O(KG)$, so merging many tasks no longer requires exhaustive search over a coefficient grid.
  • The asymmetric fitness function should yield more balanced per-task performance rather than merely a higher average, because any candidate that raises a task's loss is heavily penalized.
  • Relative Synergy Score computed before merging can guide model selection: the paper's empirical study predicts that similar task semantics, similar data distributions, moderately tuned learning rates, and larger model scales all improve mergeability.
  • If the efficiency and accuracy results generalize, model merging becomes a practical substitute for retraining in multi-task deployment, especially on memory-limited hardware.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the AWA fitness function requires task-specific losses, the method as described needs labeled evaluation data for every task; an unsupervised analogue using batch entropy on unlabeled inputs would extend CABS+ to settings where labels are unavailable, and would make the comparison with AdaMerging fully apples-to-apples.
  • The orthogonality argument predicts that CABS+ should gain most when task vectors are highly conflicting; a targeted experiment merging deliberately homogeneous or overlapping task vectors would test whether the advantage shrinks as expected.
  • The RSS findings imply a practical decision rule: estimate RSS on a small validation subset before merging and decline merges with strongly negative scores, saving the compute that would otherwise be spent on destructive merges.
  • Since CMA-ES learns correlations among task coefficients, CABS+ may scale to task counts far beyond the six tested here, but population size and iteration count will likely need to grow with dimensionality; that scaling behavior is not demonstrated in the paper.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes CABS+, an extension of the prior CABS model-merging method. CABS+ replaces CABS's grid search for task-vector scaling coefficients with Adaptive Weight Allocation (AWA), a gradient-free CMA-ES-style search using boundary constraints and an asymmetric, baseline-normalized fitness function. The authors report that CABS+ outperforms AdaMerging and WUDIMerging by 16.97% and 12.93% on average across eight experimental scenarios, uses less than 25% of AdaMerging's GPU memory on a 7B-parameter model, and reduces merging time by roughly 4x relative to WUDIMerging. The paper also contributes an empirical study of factors affecting mergeability and proposes a Relative Synergy Score (RSS) for quantifying whether merging is synergistic or destructive.

Significance. If the empirical claims hold, CABS+ would be a practically useful contribution: it offers a training-free, low-memory coefficient search that scales to billion-parameter models, and it provides evidence on when merging is likely to succeed through the RSS analysis. The evaluation is broad, spanning 27 datasets, five model families, and both language and vision models, and the authors make code available. The asymmetric fitness function is a reasonable way to prevent high-loss tasks from dominating coefficient search. However, the central performance comparison is weakened by a disclosed-information asymmetry: AWA appears to optimize a supervised loss on test labels, while AdaMerging is unsupervised. The headline improvement is also computed as an average of per-scenario relative gains rather than an aggregate accuracy difference, and the coefficients are selected on the same test data used for final accuracy reporting. These issues are load-bearing for the paper's main claim, though they are addressable with additional experiments and more careful reporting.

major comments (3)
  1. [Section III-C, Eqs. (5)-(9)] The AWA fitness function is defined through L_t(theta(lambda)), a task loss evaluated on test samples, but the paper never defines L_t or states that it requires ground-truth labels. A task loss such as cross-entropy is computed against labels for every task. This is a strictly stronger information regime than AdaMerging, which the paper explicitly describes as minimizing 'unsupervised entropy on test samples' (Section II-B). The 16.97% improvement over AdaMerging may therefore reflect the availability of labeled test data rather than a better merging mechanism. The authors should disclose the label requirement, state exactly which loss is used, and, ideally, compare CABS+ with a label-free variant (e.g., entropy-based fitness) to place both methods on equal footing.
  2. [Figure 3, Section IV-F] The headline claim that CABS+ improves overall performance by 16.97% over AdaMerging is actually the unweighted average of eight per-scenario relative improvements, not an aggregate performance difference. One scenario (GPT-2, six tasks) contributes a +74.00% relative gain because AdaMerging drops to 39.49 while CABS+ reaches 68.71, and this single outlier dominates the average. The absolute differences are much smaller in other scenarios (for example, +1.07% on the Mistral LLM Leaderboard suite). The authors should report aggregate accuracies, absolute improvements, or both, and should state explicitly that 16.97% is an average of relative gains rather than an overall accuracy improvement.
  3. [Section IV-A and IV-C] Because AWA selects lambda* by minimizing a loss on test samples (Eqs. 5-9) and the final accuracy is then reported on the same test set, the reported numbers are in-sample estimates rather than unbiased estimates of generalization. This is a methodological concern for all of the main tables, not just the efficiency comparison. The authors should use a validation split for coefficient selection and a separate test split for reporting, or add a sentence explaining why this in-sample selection does not materially affect the conclusions.
minor comments (5)
  1. [Section IV-A] The paper states that each configuration is evaluated three times and the average is reported, but no variance or standard deviation is given in any table or figure. Reporting error bars or standard deviations would strengthen the stability claim that is made repeatedly.
  2. [Figure 3] The figure mixes performance bars and improvement percentages in one plot; the labels are dense and some values (e.g., the Overall Improvement row) appear to be computed as averages of the preceding columns without this being explained in the caption. The caption should define the computation.
  3. [Abstract and Section VI] The phrase 'improves overall performance by 16.97% and 12.93%' is ambiguous; it should say 'average relative improvement over eight scenarios' to avoid overstatement.
  4. [Source Code Availability] The anonymous source-code URL contains a space ('CABS Plus-70C1'); this is likely a typo and should be corrected.
  5. [Section III-C, Eq. (8)] The penalty coefficients alpha=100 and beta=1, the bounds l=0.1 and u=2, population size K=6, and generation count G=50 are chosen without sensitivity analysis or justification. Adding a small ablation or parameter-sensitivity table would help establish that the results are not sensitive to these choices.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: CABS+ is an empirical method paper whose components are defined and measured rather than derived from the claims they support.

full rationale

CABS+ does not present a formal derivation whose outputs are equivalent to its inputs. The AWA objective (Eqs. 5-9) is a stated design choice: F(lambda)=sum_t f_t(lambda) with f_t based on normalized loss changes, and the final coefficients are obtained by minimizing this objective (Eq. 18, Algorithm 1). The reported benchmark accuracies are measured after merging, and there is no equation-level identity between the AWA fitness and the reported accuracy; accuracy and the supervised loss are different quantities. The comparison with AdaMerging and WUDIMerging is an empirical result, not a consequence of an equation. The paper's reliance on the authors' own CABS method is substantial, but CABS is an externally published ICML 2025 method used as a component and baseline, so it constitutes real evidence rather than an unverified self-citation chain. RSS (Eq. 20) is definitional, but it is explicitly introduced as a quantitative metric, not as a derived scientific law; the subsequent empirical findings are supported by controlled experiments. The most serious concern, namely that AWA's loss function appears to require ground-truth labels while AdaMerging uses unsupervised entropy, is an evaluation-fairness and information-asymmetry issue rather than circularity: the improvement is not logically forced by an equation, though it may be partly explained by the stronger supervision signal. Under the stated circularity criteria (equivalence by construction or fitted parameter renamed as prediction), no specific circular step can be exhibited from the paper's text.

Assumptions & free parameters 6 free parameters · 4 assumptions · 0 invented entities

The central mechanism assumes task-vector additivity and that test loss is a usable fitness signal. The numerical hyperparameters of AWA and CABS are hand-chosen. No new entities are introduced.

free parameters (6)
  • AWA population size K = 6
    Chosen by hand; affects search coverage and computational cost.
  • AWA max generations G = 50
    Chosen by hand with early stopping after 6 non-improving iterations.
  • AWA initial step size sigma = 0.05
    Chosen by hand for the CMA-ES search.
  • AWA feasible bounds l,u = 0.1, 2
    Boundary constraints chosen by hand to keep coefficients in a reasonable range.
  • Asymmetric penalty coefficients alpha, beta = 100, 1
    Chosen by hand; large penalty for loss increases, small reward for decreases.
  • CABS sparsity ratio = 0.90 small models, 0.75 large models
    Inherited from CABS; chosen by hand and applied uniformly.
assumptions (4)
  • domain assumption Task vectors can be additively composed with a base model to produce a merged model.
    Task arithmetic is the core assumption of CABS+ and of the CABS baseline it extends.
  • domain assumption CABS pruning makes the coefficient optimization landscape smooth enough for CMA-ES to converge quickly.
    Claimed in Section III-C without quantitative evidence; the derivation is deferred to the supplementary material.
  • domain assumption Test losses are valid proxies for downstream task performance in the AWA search.
    The AWA fitness is a sum of test losses, but the paper does not discuss label availability, distribution shift, or variance.
  • domain assumption n:m block pruning retains the task-relevant parameters while removing conflicts.
    Inherited from CABS; no formal proof is given in the main text.

how reviews work

0 comments
Cite this review

Pith. "Pith review of CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation." pith.science (2026). https://pith.science/paper/V6BHQU57

@misc{pith2026260812842,
  author       = {Pith},
  title        = {Pith review of: CABS+: Efficient and Scalable Model Merging via Conflict-Aware Sparsification and Adaptive Weight Allocation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6BHQU57}},
  note         = {Machine review of arXiv:2608.12842}
}
read the original abstract

Model merging has recently attracted significant attention as a promising paradigm for constructing unified multi-task models without requiring additional retraining. However, parameter conflicts and knowledge interference across tasks often degrade merged-model performance. Prior work introduced Conflict-Aware and Balanced Sparsification (CABS), which reduces parameter interference through structured pruning and sequential masking. However, CABS relies on grid search to determine scaling coefficients, resulting in exponential time complexity, while its optimization objective can be dominated by high-performance tasks, leading to suboptimal overall performance. To address these limitations, we extend CABS and propose CABS+. Specifically, Adaptive Weight Allocation (AWA) optimizes merging coefficients via a gradient-free search scheme to reduce time complexity, while an asymmetric fitness function promotes more comprehensive performance gains across tasks. Moreover, we conduct a systematic empirical study of key factors influencing model merging performance and propose Relative Synergy Score (RSS) to quantify model mergeability and guide model selection. We compare CABS+ with state-of-the-art model merging methods, including CABS, AdaMerging, and WUDIMerging, across 27 datasets and 5 models covering large language, small-scale language, and vision models. Extensive experiments verify the effectiveness and efficiency of CABS+. Compared with AdaMerging and WUDIMerging, CABS+ improves overall performance by 16.97% and 12.93%, respectively, exhibits stronger stability and robustness across varying task numbers and model architectures, uses less than 25% of the GPU memory required by AdaMerging, and achieves nearly a 4x speedup in merging time over WUDIMerging.

Figures

Figures reproduced from arXiv: 2608.12842 by the authors.

Figure 1
Figure 1. Illustration of the overall framework of CABS+. (a) Conflict-Aware and Balanced Sparsification; (b) Adaptive Weight Allocation strategy; (c) Overall [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Performance comparison of different merging methods on RoBERTa [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Comprehensive Performance and Improvement Comparison Across Methods. [PITH_FULL_IMAGE:figures/full_fig_p011_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

65 extracted references · 44 canonical work pages

  1. [1]

    Task vector quantization for memory-efficient model merging,

    Y . Kim, S. Lee, A. Jung, B. Ryu, and S. Hong, “Task vector quantization for memory-efficient model merging,” inProceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), 2025, pp. 20 105– 20 115

    work page 2025

  2. [2]

    Parameter competition balancing for model merging,

    G. Duet al., “Parameter competition balancing for model merging,” Advances in Neural Information Processing Systems (NeurIPS), vol. 37, pp. 84 746–84 776, 2024

    work page 2024

  3. [3]

    Toward Effective Model Merging in Semantic Segmen- tation,

    H. Chenet al., “Toward Effective Model Merging in Semantic Segmen- tation,”IEEE Transactions on Neural Networks and Learning Systems (TNNLS), vol. 37, no. 4, pp. 1948–1962, Apr. 2026

    work page 1948

  4. [4]

    Deep Model Fusion: A Survey,

    W. Li, Y . Peng, M. Zhang, L. Ding, H. Hu, and L. Shen, “Deep Model Fusion: A Survey,”IEEE Transactions on Neural Networks and Learning Systems (TNNLS), pp. 1–17, 2025

    work page 2025

  5. [5]

    Dataless Knowledge Fusion by Merging Weights of Language Models,

    X. Jin, X. Ren, D. Preotiuc-Pietro, and P. Cheng, “Dataless Knowledge Fusion by Merging Weights of Language Models,” inThe Eleventh International Conference on Learning Representations (ICLR), Sep. 2022

    work page 2022

  6. [6]

    Evolutionary optimization of model merging recipes,

    T. Akiba, M. Shing, Y . Tang, Q. Sun, and D. Ha, “Evolutionary optimization of model merging recipes,”Nature Machine Intelligence (NMI), vol. 7, no. 2, pp. 195–204, Feb. 2025

    work page 2025

  7. [7]

    Editing models with task arithmetic,

    G. Ilharco, M. T. Ribeiro, M. Wortsman, L. Schmidt, H. Hajishirzi, and A. Farhadi, “Editing models with task arithmetic,” inThe Eleventh International Conference on Learning Representations (ICLR), Sep. 2023

    work page 2023

  8. [8]

    Adamerging: Adaptive model merging for multi- task learning,

    E. Yanget al., “Adamerging: Adaptive model merging for multi- task learning,” inThe Twelfth International Conference on Learning Representations (ICLR), May 2024

    work page 2024

Show all 65 references
  1. [9]

    Whoever started the interference should end it: Guiding data-free model merging via task vectors,

    R. Cheng, F. Xiong, Y . Wei, W. Zhu, and C. Yuan, “Whoever started the interference should end it: Guiding data-free model merging via task vectors,” inForty-second International Conference on Machine Learning (ICML), 2025

    2025

  2. [10]

    Channel merging: Preserving specialization for merged experts,

    M. Zhang, J. Liu, G. Ding, L. Ou, X. Yu, and B. Zhuang, “Channel merging: Preserving specialization for merged experts,” inProceedings of the AAAI Conference on Artificial Intelligence (AAAI), vol. 39, no. 21, 2025, pp. 22 479–22 487. 15

    2025

  3. [11]

    Mergebench: A benchmark for merging domain-specialized LLMs,

    Y . He, S. Zeng, Y . Hu, R. Yang, T. Zhang, and H. Zhao, “Mergebench: A benchmark for merging domain-specialized LLMs,” inThe Thirty- ninth Annual Conference on Neural Information Processing Systems (NeurIPS), 2026

    2026

  4. [12]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch,

    L. Yu, B. Yu, H. Yu, F. Huang, and Y . Li, “Language models are super mario: Absorbing abilities from homologous models as a free lunch,” inForty-first International Conference on Machine Learning (ICML), 2024

    2024

  5. [13]

    TIES- Merging: Resolving Interference When Merging Models,

    P. Yadav, D. Tam, L. Choshen, C. A. Raffel, and M. Bansal, “TIES- Merging: Resolving Interference When Merging Models,”Advances in Neural Information Processing Systems (NeurIPS), vol. 36, pp. 7093– 7115, Dec. 2023

    2023

  6. [14]

    Model breadcrumbs: Scaling multi- task model merging with sparse masks,

    M. Davari and E. Belilovsky, “Model breadcrumbs: Scaling multi- task model merging with sparse masks,” inEuropean Conference on Computer Vision (ECCV). Springer, 2024, pp. 270–287

    2024

  7. [15]

    Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic,

    Y . He, Y . Hu, Y . Lin, T. Zhang, and H. Zhao, “Localize-and-Stitch: Efficient Model Merging via Sparse Task Arithmetic,”Transactions on Machine Learning Research (TMLR), Oct. 2024

    2024

  8. [16]

    Pruning and quantization for deep neural network acceleration: A survey,

    T. Liang, J. Glossner, L. Wang, S. Shi, and X. Zhang, “Pruning and quantization for deep neural network acceleration: A survey,”Neuro- computing, vol. 461, pp. 370–403, Oct. 2021

    2021

  9. [17]

    Knowledge composition using task vectors with learned anisotropic scaling,

    F. Z. Zhang, P. Albert, C. Rodriguez-Opazo, A. van den Hengel, and E. Abbasnejad, “Knowledge composition using task vectors with learned anisotropic scaling,”Advances in Neural Information Processing Systems (NeurIPS), vol. 37, pp. 67 319–67 354, 2024

    2024

  10. [18]

    Learn to merge: Meta-learning for adaptive multi-task model merging,

    J. Chen, Q. Zhang, W. Zhang, X. Luo, P. S. Yu, and Z. Qiao, “Learn to merge: Meta-learning for adaptive multi-task model merging,” 2026, arxiv

    2026

  11. [19]

    Averaging weights leads to wider optima and better generalization,

    P. Izmailov, D. Podoprikhin, T. Garipov, D. Vetrov, and A. G. Wilson, “Averaging weights leads to wider optima and better generalization,” in34th Conference on Uncertainty in Artificial Intelligence 2018, UAI

    2018

  12. [20]

    Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing inference time,

    M. Wortsmanet al., “Model soups: averaging weights of multiple fine- tuned models improves accuracy without increasing inference time,” in Proceedings of the 39th International Conference on Machine Learning (ICML), Jun. 2022, pp. 23 965–23 998

    2022

  13. [21]

    Merging Models with Fisher- Weighted Averaging,

    M. S. Matena and C. A. Raffel, “Merging Models with Fisher- Weighted Averaging,”Advances in Neural Information Processing Sys- tems (NeurIPS), vol. 35, pp. 17 703–17 716, Dec. 2022

    2022

  14. [22]

    Multi-task model merging via adaptive weight disen- tanglement,

    F. Xionget al., “Multi-task model merging via adaptive weight disen- tanglement,”arXiv preprint arXiv:2411.18729, 2024

    2024 arXiv

  15. [23]

    Modeling multi-task model merging as adaptive projective gradient descent,

    Y . Wei, A. Tang, L. Shen, Z. Hu, C. Yuan, and X. Cao, “Modeling multi-task model merging as adaptive projective gradient descent,” in Proceedings of the 42nd International Conference on Machine Learning (ICML), vol. 267, 13–19 Jul 2025, pp. 66 178–66 193

    2025

  16. [24]

    Representation surgery in model merging with proba- bilistic modeling,

    Q. Weiet al., “Representation surgery in model merging with proba- bilistic modeling,” inForty-second International Conference on Machine Learning (ICML), Jun. 2025

    2025

  17. [25]

    Localizing task information for improved model merging and com- pression,

    K. Wang, N. Dimitriadis, G. Ortiz-Jimenez, F. Fleuret, and P. Frossard, “Localizing task information for improved model merging and com- pression,” inForty-first International Conference on Machine Learning (ICML), 2024

    2024

  18. [26]

    Dropout: a simple way to prevent neural networks from overfit- ting,

    N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhut- dinov, “Dropout: a simple way to prevent neural networks from overfit- ting,”The journal of machine learning research (JMLR), vol. 15, no. 1, pp. 1929–1958, 2014

    1929

  19. [27]

    BERT busters: Outlier dimensions that disrupt transformers,

    O. Kovaleva, S. Kulshreshtha, A. Rogers, and A. Rumshisky, “BERT busters: Outlier dimensions that disrupt transformers,” inFindings of the Association for Computational Linguistics: ACL-IJCNLP, 2021, pp. 3392–3405

    2021

  20. [28]

    Outlier Dimen- sions that Disrupt Transformers are Driven by Frequency,

    G. Puccetti, A. Rogers, A. Drozd, and F. Dell’Orletta, “Outlier Dimen- sions that Disrupt Transformers are Driven by Frequency,” inFindings of the Association for Computational Linguistics: EMNLP 2022, Dec. 2022, pp. 1286–1304

    2022

  21. [29]

    Outlier weighed layerwise sparsity (OWL): a missing secret sauce for pruning LLMs to high sparsity,

    L. Yinet al., “Outlier weighed layerwise sparsity (OWL): a missing secret sauce for pruning LLMs to high sparsity,” inProceedings of the 41st International Conference on Machine Learning (ICML), ser. ICML’24, vol. 235, Jul. 2024, pp. 57 101–57 115

    2024

  22. [30]

    Learning n:m fine-grained structured sparse neural networks from scratch,

    A. Zhouet al., “Learning n:m fine-grained structured sparse neural networks from scratch,” inInternational Conference on Learning Rep- resentations (ICLR), 2021

    2021

  23. [31]

    Structured Pruning Learns Compact and Accurate Models,

    M. Xia, Z. Zhong, and D. Chen, “Structured Pruning Learns Compact and Accurate Models,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), May 2022, pp. 1513–1528

    2022

  24. [32]

    Completely derandomized self- adaptation in evolution strategies,

    N. Hansen and A. Ostermeier, “Completely derandomized self- adaptation in evolution strategies,”Evolutionary computation, vol. 9, no. 2, pp. 159–195, 2001

    2001

  25. [33]

    Open llm leaderboard,

    E. Beechinget al., “Open llm leaderboard,” 2023

    2023

  26. [34]

    Mistral 7B,

    A. Q. Jianget al., “Mistral 7B,”arXiv preprint arXiv:2310.06825, 2023

    2023 arXiv

  27. [35]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P. Clarket al., “Think you have solved question answering? try arc, the ai2 reasoning challenge,”arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  28. [36]

    Hellaswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y . Bisk, A. Farhadi, and Y . Choi, “Hellaswag: Can a machine really finish your sentence?” inProceedings of the 57th Annual Meeting of the Association for Computational Linguistics (ACL), 2019, pp. 4791–4800

    2019

  29. [37]

    MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,

    Y . Wanget al., “MMLU-Pro: A More Robust and Challenging Multi-Task Language Understanding Benchmark,” 2024, https://arxiv.org/abs/2406.01574

    2024 arXiv

  30. [38]

    TruthfulQA: Measuring How Models Mimic Human Falsehoods,

    S. Lin, J. Hilton, and O. Evans, “TruthfulQA: Measuring How Models Mimic Human Falsehoods,” inProceedings of the 60th Annual Meeting of the Association for Computational Linguistics (ACL), May 2022, pp. 3214–3252

    2022

  31. [39]

    Winogrande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. L. Bras, C. Bhagavatula, and Y . Choi, “Winogrande: An adversarial winograd schema challenge at scale,”Communications of the ACM, vol. 64, no. 9, pp. 99–106, 2021

    2021

  32. [40]

    Training verifiers to solve math word problems,

    K. Cobbeet al., “Training verifiers to solve math word problems,”arXiv preprint arXiv:2110.14168, 2021

    2021 arXiv

  33. [41]

    Open llm leaderboard v2,

    C. Fourrier, N. Habib, A. Lozovskaya, K. Szafer, and T. Wolf, “Open llm leaderboard v2,” https://huggingface.co/spaces/open-llm-leaderboard/ open llm leaderboard, 2024

    2024

  34. [42]

    Qwen2. 5 technical report,

    A. Yanget al., “Qwen2. 5 technical report,”arXiv preprint arXiv:2412.15115, 2024

    2024 arXiv

  35. [43]

    Instruction-following evaluation for large language models,

    J. Zhouet al., “Instruction-following evaluation for large language models,” 2023, https://arxiv.org/abs/2311.07911

    2023 arXiv

  36. [44]

    Challenging big-bench tasks and whether chain-of- thought can solve them,

    M. Suzgunet al., “Challenging big-bench tasks and whether chain-of- thought can solve them,” 2022, https://arxiv.org/abs/2210.09261

    2022 arXiv

  37. [45]

    Measuring mathematical problem solving with the math dataset,

    D. Hendryckset al., “Measuring mathematical problem solving with the math dataset,” 2021, https://arxiv.org/abs/2103.03874

    2021 arXiv

  38. [46]

    Gpqa: A graduate-level google-proof qa benchmark,

    D. Reinet al., “Gpqa: A graduate-level google-proof qa benchmark,” 2023, https://arxiv.org/abs/2311.12022

    2023 arXiv

  39. [47]

    Musr: Testing the limits of chain-of-thought with multistep soft reasoning,

    Z. Sprague, X. Ye, K. Bostrom, S. Chaudhuri, and G. Durrett, “Musr: Testing the limits of chain-of-thought with multistep soft reasoning,” 2024, https://arxiv.org/abs/2310.16049

    2024 arXiv

  40. [48]

    A framework for few-shot language model evaluation,

    L. Gaoet al., “A framework for few-shot language model evaluation,” 2024

    2024

  41. [49]

    Roberta: A robustly optimized bert pretraining approach,

    Y . Liu, “Roberta: A robustly optimized bert pretraining approach,”arXiv preprint arXiv:1907.11692, 2019

    1907 arXiv

  42. [50]

    Language models are unsupervised multitask learn- ers,

    A. Radfordet al., “Language models are unsupervised multitask learn- ers,”OpenAI blog, vol. 1, no. 8, p. 9, 2019

    2019

  43. [51]

    Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. Bowman, “Glue: A multi-task benchmark and analysis platform for natural lan- guage understanding,” inProceedings of the 2018 EMNLP Workshop BlackboxNLP: Analyzing and Interpreting Neural Networks for NLP, 2018, pp. 353–355

    2018

  44. [52]

    Neural network accept- ability judgments,

    A. Warstadt, A. Singh, and S. R. Bowman, “Neural network accept- ability judgments,”Transactions of the Association for Computational Linguistics, vol. 7, pp. 625–641, 2019

    2019

  45. [53]

    The repeval 2017 shared task: Multi-genre natural language inference with sentence representations,

    N. Nangia, A. Williams, A. Lazaridou, and S. Bowman, “The repeval 2017 shared task: Multi-genre natural language inference with sentence representations,” inProceedings of the 2nd workshop on evaluating vector space representations for NLP, 2017, pp. 1–10

    2017

  46. [54]

    Automatically constructing a corpus of sen- tential paraphrases,

    B. Dolan and C. Brockett, “Automatically constructing a corpus of sen- tential paraphrases,” inThird international workshop on paraphrasing (IWP2005), 2005

    2005

  47. [55]

    The pascal recognising textual entailment challenge,

    I. Dagan, O. Glickman, and B. Magnini, “The pascal recognising textual entailment challenge,” inMachine learning challenges workshop. Springer, 2005, pp. 177–190

    2005

  48. [56]

    The second pascal recognising textual entailment challenge,

    R. Bar-Haimet al., “The second pascal recognising textual entailment challenge,” inProceedings of the second PASCAL challenges workshop on recognising textual entailment, vol. 1. Citeseer, 2006

    2006

  49. [57]

    The third pascal recognizing textual entailment challenge,

    D. Giampiccolo, B. Magnini, I. Dagan, and W. B. Dolan, “The third pascal recognizing textual entailment challenge,” inProceedings of the ACL-PASCAL workshop on textual entailment and paraphrasing, 2007, pp. 1–9

    2007

  50. [58]

    The fifth pascal recognizing textual entailment challenge

    L. Bentivogli, P. Clark, I. Dagan, and D. Giampiccolo, “The fifth pascal recognizing textual entailment challenge.”TAC, vol. 7, no. 8, p. 1, 2009

    2009

  51. [59]

    Recursive deep models for semantic compositionality over a sentiment treebank,

    R. Socheret al., “Recursive deep models for semantic compositionality over a sentiment treebank,” inProceedings of the 2013 conference on empirical methods in natural language processing (EMNLP), 2013, pp. 1631–1642. 16

    2013

  52. [60]

    Race: Large-scale reading comprehension dataset from examinations,

    G. Lai, Q. Xie, H. Liu, Y . Yang, and E. Hovy, “Race: Large-scale reading comprehension dataset from examinations,” inProceedings of the 2017 Conference on Empirical Methods in Natural Language Processing (EMNLP), 2017, pp. 785–794

    2017

  53. [61]

    Squad: 100,000+ questions for machine comprehension of text,

    P. Rajpurkar, “Squad: 100,000+ questions for machine comprehension of text,”arXiv preprint arXiv:1606.05250, 2016

    2016 arXiv

  54. [62]

    FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion,

    A. Tanget al., “FusionBench: A Unified Library and Comprehensive Benchmark for Deep Model Fusion,”Journal of Machine Learning Research (JMLR), vol. 26, no. 307, pp. 1–38, 2025

    2025

  55. [63]

    raids-lab/crater,

    “raids-lab/crater,” https://github.com/raids-lab/crater., 2026

    2026

  56. [64]

    Beyond task vectors: Selective task arithmetic based on importance metrics,

    T. Bowen, L. Songning, W. Jiemin, S. Zhihao, G. Shiming, and Y . Yutao, “Beyond task vectors: Selective task arithmetic based on importance metrics,”arXiv preprint arXiv:2411.16139, 2024

    2024 arXiv

  57. [2018]

    Association For Uncertainty in Artificial Intelligence (AUAI), 2018, pp. 876–885

    2018

Pith tools

Reviewed August 15, 2026 · model on record in the stance chip above.