Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching

T0 review · 3 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that pruning an LLM can be reformulated as searching over layer removals, selections, and merges across a family of finetuned checkpoints, preserving 97.3% of a Llama2-13B's benchmark score while removing 25% of parameters.

desk verdict A practical pruning method that stitches layers from fine-tuned variants; the core idea holds up, but the headline 97.3% is a best-of-three point estimate with calibration leakage, so treat it as promising rather than pinned down. read the letter →

arxiv 2506.20480 v1 pith:R6II4XOZ submitted 2025-06-25 cs.CL

Guinan Su , Li Shen , Lu Yin , Shiwei Liu , Yanwu Yang , Jonas Geiping This is my paper
classification cs.CL
keywords structuredpruninglayermergingzero-orderoptimizationmodelLLMcompressionfinetunedvariantsParetoLlama-2
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper argues that the best way to prune a large language model is not to prune a single model but to cut and stitch layers drawn from a family of finetuned variants of the same base model. Each finetuned variant accentuates a different ability, so a smaller model can be assembled from the strongest pieces of several siblings through layer removal, layer selection, and layer merging. The authors pose this assembly as a zero-order optimization problem and solve it with a multi-fidelity Bayesian search guided by accuracy on a small calibration set. If the claim holds, structured pruning can remove roughly a quarter of a model's parameters and retain about 97% of its benchmark performance, without the expensive post-training that ordinary pruning methods require. On the tested Llama-2 models the method outperforms four existing pruning baselines across a 14-task evaluation suite.

What carries the argument

The carrier of the argument is the configuration vector $\omega = \{r, \{c_i\}, \{m_i\}, \{h_i\}\}$: a binary removal vector $r$ over all layer positions, a selection vector $c_i$ telling which finetuned variant supplies each retained position (with the base model as default), and a merge method $m_i$ with hyperparameters $h_i$ for positions fed by multiple variants. This turns pruning into a black-box search over a finite but huge space. The objective is a Chebyshev scalarization of per-task accuracies on four calibration datasets, and the search is run by a multi-fidelity Bayesian optimizer that spends small budgets on most candidates and large budgets only on the survivors. This machinery is what lets the method find configurations that remove redundant later layers, import specialized layers from different finetunes, and merge the imported layers instead of merely concatenating them.

What would settle it

Take one of the reported pruned architectures, permute which finetuned variant supplies each retained layer, and evaluate on the full 14-task suite; if random permutations match the optimized selection's average score, then layer-level transplantability and the search are not actually carrying the result.

Watch

Extended reading notes

Core claim

The paper's central claim is that an LLM's capabilities survive aggressive layer removal when the remaining layer positions are filled by layers taken from, or merged across, finetuned siblings of the same base model. Each finetuned variant accentuates a different capability, and task-arithmetic merging combines layer-level weight vectors so that the compressed model inherits pieces of all of them. The search is formulated as a zero-order optimization over a configuration space of removal, selection, and merge hyperparameters, evaluated by a multi-objective accuracy signal on a small calibration set. On Llama2-13B the best found models keep 97.3% of the dense model's average benchmark score while dropping 25% of parameters; on Llama2-7B the retention is 92.2%, and on Llama-3-8B it is 84.55% after removing 9 of 32 layers. These results are reported without post-training, outperforming the four compared pruning baselines on the 14-task suite.

Load-bearing premise

A layer taken from one finetuned version of the base model can be dropped into another version, or merged with its corresponding layer, without functionally corrupting the model.

Editorial extensions

If this is right

  • The pruned models need no post-training: the 13B model holds 97.3% of the dense model's average score immediately after removing 10 of 40 layers.
  • Bigger models are more compressible this way: at similar pruning ratios, Llama2-13B keeps 97.3% of performance while Llama2-7B keeps 92.2%.
  • Layer merging is the load-bearing operation: enabling it raises the 7B average from 44.83 (removal only) and 43.20 (selection without merging) to 48.55.
  • The recipe transfers across generations: on Llama-3-8B with 9 of 32 layers removed, the method retains 84.55% of performance versus 62.79% for the best single-model baseline.
  • The search returns a Pareto front of pruned models, so a practitioner can choose among configurations that trade off different task abilities.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper: the same cutting-and-stitching recipe should transfer to any architecture with finetuned checkpoints, but search cost grows with the number of candidates, so cheaper surrogates or warm starts are the natural next step.
  • Beyond the paper: the ablation gap between removal-only, selection-without-merging, and full merging suggests that finetuned variants store partially complementary skills at the layer level; that structure could be mapped directly to understand where abilities live in a network.
  • Beyond the paper: a testable extension is to stack this method with quantization, since parameter-count reduction and bit-width reduction are independent axes and their deployment gains should multiply.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper proposes GPTailor, a structured-pruning method that searches over layer removal, layer selection among a family of fine-tuned checkpoints of the same base model, and layer-wise task-arithmetic merging. The search is formulated as a zero-order multi-objective optimization problem solved with the SMAC multi-fidelity optimizer. Experiments on Llama2-7B/13B and Llama3-8B, evaluated on 14 OpenCompass tasks, report that the compressed Llama2-13B model retains 97.3% of the best dense candidate's average performance while removing about 25% of layers, and that the approach outperforms LLM-Pruner, SliceGPT, LaCo, and ShortGPT without post-training. Ablations isolate the contributions of layer removal, layer selection, layer merging, and the calibration objective.

Significance. If the quantitative results hold, the paper makes a useful conceptual contribution: instead of pruning a single checkpoint, one can treat a family of fine-tuned variants as a pool of layers and optimize which layers to keep, transplant, or merge. The manuscript has clear strengths: the code is released, the baseline comparisons cover multiple scenarios (single-model pruning, pruning-then-merging, merging-then-pruning), and the ablation suite is thoughtfully designed (LR-only, LS+LR, FL-merge, Single-obj, PPL-obj, and the avg* column). These strengths support the qualitative claim that cross-variant layer search with merging helps. The quantitative headline figures are less firmly established, because they are best-of-three point estimates selected with a calibration set that overlaps four of the fourteen evaluation tasks, and no variance or repeated-search information is reported.

major comments (3)
  1. [§4.2, Table 1; abstract; conclusion] The headline retention numbers are not computed against the model named as "original" in the paper. The 97.3% figure is 54.33/55.86, where 55.86 is the dense Code variant, not the base Llama-2-13B (55.11); similarly, 92.2% is 48.55/52.63, where 52.63 is the dense LM variant, not the base Llama-2-7B (45.47). The abstract and conclusion present these as retention of "original performance," which is misleading; for the 7B model the ratio to the base model is above 100%. Please state the retention denominator explicitly and either compute the stated claim against the base model or rephrase the claim as retention relative to the best fine-tuned candidate.
  2. [§3.3, Eq. (2); §4.1; §4.2] The reported retention uses the best of three Pareto-optimal configurations (48.55 for 7B and 54.33 for 13B), and those configurations were selected by a search objective evaluated on calibration samples from PIQA, WSC, CSQA, and MMLU. The avg* column removes those four tasks from the reported average, but the selected configurations were still optimized on them, so the remaining ten tasks are not independent holdouts. No repeated search seeds or standard deviations are reported, and the text says the three Pareto configurations were "randomly selected" while the narrative uses the maximum of the three. As a result, the specific values 97.3% and 92.2% are optimistic point estimates rather than established retention rates. Please report all Pareto solutions, add multiple search seeds, and either use a disjoint calibration set or demonstrate stability of the conclusions to the calibration-task choice.
  3. [§4.3, Table 2] The efficiency claim is not quantified. Table 2 reports only the distribution of trials across budget levels, but the 22% of trials at the largest budget consume the majority of the total calibration samples (110×1000 versus 207×100 plus 183×300 plus 110×1000), and the paper does not report wall-clock time, GPU-hours, or inference cost per budget. The statement that this allocation "significantly increase[s] the chance of discovering superior configurations under the same computational budget" needs a direct cost measurement to be verifiable.
minor comments (5)
  1. [§3.2, Eq. (1)] The search-space cardinality formula is written as a single product over the retained-layer positions, but the product over {i : r_i = 0} depends on the particular removal vector r. As a closed-form expression the formula is inconsistent and should be revised or replaced by a sum over valid removal vectors.
  2. [§4.4] The Perplexity-objective experiment uses 1500 WikiText examples as calibration data, while the Single-objective experiment uses 1000 MMLU validation examples; the text says these are "similarly sized," which is inaccurate. Please correct the wording or use equal calibration sizes.
  3. [§4.2, Table 1] The paper should state explicitly whether the three reported Ours rows are three independent Pareto solutions or multiple seeds of the same search; this matters for interpreting the spread, for example the XSum values 12.99, 3.46, and 13.20 for the 7B model.
  4. [Appendix D] For LLM-Pruner and SliceGPT, only merging-then-pruning is evaluated, not pruning-then-merging, because of the stated alignment difficulties. This is reasonable, but the narrative in §4.2 saying that baselines were re-run under all three scenarios should be qualified to avoid overstating the comparison.
  5. [Figure 3] The x-axis label "Prune Ratio" uses percentages from 0 to 75, while the text describes 0–37.5% as the low-ratio range; also, the "Ours 0%" point is not the unmodified base model but the result of search with no layer removal. Please clarify both points.

Circularity Check

0 steps flagged · score 2.0 of 10

No significant circularity: the search/evaluation chain is self-contained, and the only self-citation is a non-load-bearing initialization heuristic.

full rationale

Walking the claimed derivation chain—search space (Sec. 3.2), multi-objective calibration objective (Eq. 2), SMAC search (Algorithm 1), and the OpenCompass evaluation (Table 1)—we find no step that is equivalent to its inputs by construction. The pruned models are not derived from the reported benchmarks; they are selected by maximizing Eq. (2) on PIQA/WSC/CSQA/MMLU calibration samples, and the paper explicitly acknowledges the resulting overlap by reporting the avg* column that excludes those four tasks. Because the optimization objective is a Chebyshev scalarization of four task accuracies rather than the 14-task average, the headline 97.3% figure is an evaluation outcome, not a tautology; the independent content (cross-task transfer and superiority over baselines) is carried by the avg* and ablation comparisons. The only self-citation, [34] (Su and Geiping), is used to justify random-middle-layer initialization with the claim that 'models are relatively robust to changes in these intermediate layers'; this is a non-load-bearing heuristic, and the central method does not rest on it. No uniqueness theorem or ansatz is imported from the authors' prior work. The paper's self-reported check ('To ensure our results were not biased by our calibration data, we also calculate an avg*') and its disclosure that three Pareto-optimal configurations were selected for reporting are evaluation-protocol caveats, not circular derivation steps. The central claims are therefore supported by independent benchmark evaluations, and the paper merits only the minor score associated with a non-load-bearing self-citation.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The method introduces no new theoretical entities, but it relies on several domain assumptions: layer interchangeability across fine-tuned variants, layer-level task-arithmetic merging, calibration-accuracy proxy, and middle-layer redundancy. The searched merge factors and output scales are fitted to calibration data and are part of the reported configurations.

free parameters (3)
  • Per-layer task arithmetic merge factor lambda = Searched in [0.5, 1.0] per merged layer (e.g., 0.64, 0.60, 0.50 in Tables 12 and 13)
    The optimizer selects this factor for every layer position where multiple candidate layers are merged; it is fit to calibration accuracy and is part of the reported architecture.
  • Per-layer output scale = Values such as 1.05, 0.92, and 1.19 in Tables 12 and 13
    The architecture tables list an Output Scale per retained layer that is not defined or derived in the method section; if used at inference it is an additional searched or hand-adjusted parameter.
  • Sparsity target s = 9/32 for 7B and 8B, 10/40 for 13B
    Set to match ShortGPT and LaCo settings for comparison, not derived from the method; it controls all reported results.
assumptions (5)
  • domain assumption Fine-tuned variants share a common base, so layers can be transplanted across models without alignment or dimension mismatch
    The search space directly inserts layers from candidate models into a pruned model (Section 3.2, selection vector c_i, Figure 2); this requires cross-model layer compatibility.
  • domain assumption Layer-wise task arithmetic merging preserves functionality
    Merging in the search space uses task-arithmetic combination at the layer level (Sections 3.2 and Appendix C), assuming individual layer weight differences can be scaled and summed.
  • domain assumption Accuracy on the calibration tasks is a valid proxy for preserving broad capabilities
    The objective optimizes accuracy on PIQA, WSC, CSQA, and MMLU (Section 3.3); the paper relies on this to select configurations that then score well on the full 14-task suite.
  • domain assumption Middle-layer removal is a safe initialization for the search
    Section 4.1 states 'we use models with randomly removed middle layers as starting points', relying on middle-layer redundancy literature (ShortGPT, deeper layers).
  • domain assumption OpenCompass evaluation accurately reflects model capability
    All performance claims are averages over 14 tasks from OpenCompass (Section 4.1); if this suite is unrepresentative, the retention percentages change.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching." pith.science (2026). https://pith.science/paper/R6II4XOZ

@misc{pith2026250620480,
  author       = {Pith},
  title        = {Pith review of: GPTailor: Large Language Model Pruning Through Layer Cutting and Stitching},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/R6II4XOZ}},
  note         = {Machine review of arXiv:2506.20480}
}
abstract

Large language models (LLMs) have shown remarkable capabilities in language understanding and generation. However, such impressive capability typically comes with a substantial model size, which presents significant challenges in deployment and inference. While structured pruning of model parameters offers a promising way to reduce computational costs at deployment time, current methods primarily focus on single model pruning. In this work, we develop a novel strategy to compress models by strategically combining or merging layers from finetuned model variants, which preserves the original model's abilities by aggregating capabilities accentuated in different finetunes. We pose the optimal tailoring of these LLMs as a zero-order optimization problem, adopting a search space that supports three different operations: (1) Layer removal, (2) Layer selection from different candidate models, and (3) Layer merging. Our experiments demonstrate that this approach leads to competitive model pruning, for example, for the Llama2-13B model families, our compressed models maintain approximately 97.3\% of the original performance while removing $\sim25\%$ of parameters, significantly outperforming previous state-of-the-art methods. The code is available at https://github.com/Guinan-Su/auto-merge-llm.

Figures

Figures reproduced from arXiv: 2506.20480 by the authors.

Figure 1
Figure 1. Our Approach: Model Pruning through Cutting and Stitching. We achieve competitive model pruning performance by running a zero-order search that tailors layers based on a shared pool of finetuned variants of the original model, selecting and stitching layers if necessary. The model finetunes accentuate task-specific skills, allowing us to merge key components into a smaller model, maintaining, for example, 97% of cap… view at source ↗
Figure 2
Figure 2. (a) Structure of our best-performing 7B-pruned model. The model integrates layers from [PITH_FULL_IMAGE:figures/full_fig_p007_2.png] view at source ↗
Figure 3
Figure 3. Performance Comparison Across Dif￾ferent Pruning Ratios. 4.4 Which Parts of the Search Space are Critical ? To determine where the benefits of our approach come from, we designed ablation experiments to evaluate the contribution of different components in our search space. As our framework supports: (1) selectively choosing layers from different candidate models, (2) layer merging, and (3) layer removal, we conducte… view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. When Fewer Layers Break More Chains: Layer Pruning Harms Test-Time Scaling in LLMs

    cs.LG 2025-10 conditional novelty 6.0 of 10

    Removing just one or two layers from a reasoning-tuned LLM destroys its test-time scaling, and neither LoRA nor full fine-tuning restores it.

Reference graph

Works this paper leans on

62 extracted references · 14 canonical work pages · cited by 1 Pith paper

  1. [1]

    Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timo- thée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. Llama: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, Lama Ahmad, Ilge Akkaya, Floren- cia Leoni Aleman, Diogo Almeida, Janko Altenschmidt, Sam Altman, Shyamal Anadkat, et al. Gpt-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    arXiv 2023

  3. [3]

    Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna

    Wei-Lin Chiang, Zhuohan Li, Ziqing Lin, Ying Sheng, Zhanghao Wu, Hao Zhang, Lianmin Zheng, Siyuan Zhuang, Yonghao Zhuang, Joseph E Gonzalez, et al. Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality.See https://vicuna. lmsys. org (accessed 14 April 2023), 2(3):6, 2023

    2023

  4. [4]

    Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    Jared Kaplan, Sam McCandlish, Tom Henighan, Tom B Brown, Benjamin Chess, Rewon Child, Scott Gray, Alec Radford, Jeffrey Wu, and Dario Amodei. Scaling laws for neural language models.arXiv preprint arXiv:2001.08361, 2020

    arXiv 2001

  5. [5]

    Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    Jordan Hoffmann, Sebastian Borgeaud, Arthur Mensch, Elena Buchatskaya, Trevor Cai, Eliza Rutherford, Diego de Las Casas, Lisa Anne Hendricks, Johannes Welbl, Aidan Clark, et al. Training compute-optimal large language models.arXiv preprint arXiv:2203.15556, 2022

    arXiv 2022

  6. [6]

    Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    Jason Wei, Yi Tay, Rishi Bommasani, Colin Raffel, Barret Zoph, Sebastian Borgeaud, Dani Yogatama, Maarten Bosma, Denny Zhou, Donald Metzler, et al. Emergent abilities of large language models.arXiv preprint arXiv:2206.07682, 2022

    arXiv 2022

  7. [7]

    Learning and generalization in overparame- terized neural networks, going beyond two layers.Advances in neural information processing systems, 32, 2019

    Zeyuan Allen-Zhu, Yuanzhi Li, and Yingyu Liang. Learning and generalization in overparame- terized neural networks, going beyond two layers.Advances in neural information processing systems, 32, 2019

    work page 2019

  8. [8]

    A convergence theory for deep learning via over-parameterization

    Zeyuan Allen-Zhu, Yuanzhi Li, and Zhao Song. A convergence theory for deep learning via over-parameterization. InInternational conference on machine learning, pages 242–252. PMLR, 2019

    2019

Show all 62 references
  1. [9]

    Train big, then compress: Rethinking model size for efficient training and inference of transformers

    Zhuohan Li, Eric Wallace, Sheng Shen, Kevin Lin, Kurt Keutzer, Dan Klein, and Joey Gon- zalez. Train big, then compress: Rethinking model size for efficient training and inference of transformers. InInternational Conference on machine learning, pages 5958–5968. PMLR, 2020

    2020

  2. [10]

    Sparsegpt: Massive language models can be accurately pruned in one-shot

    Elias Frantar and Dan Alistarh. Sparsegpt: Massive language models can be accurately pruned in one-shot. InInternational Conference on Machine Learning, pages 10323–10337. PMLR, 2023

    2023

  3. [11]

    Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078, 2023

    Tim Dettmers, Ruslan Svirschevski, Vage Egiazarian, Denis Kuznedelev, Elias Frantar, Saleh Ashkboos, Alexander Borzunov, Torsten Hoefler, and Dan Alistarh. Spqr: A sparse-quantized representation for near-lossless llm weight compression.arXiv preprint arXiv:2306.03078, 2023

    2023 arXiv

  4. [12]

    Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023

    Mengzhou Xia, Tianyu Gao, Zhiyuan Zeng, and Danqi Chen. Sheared llama: Accelerating language model pre-training via structured pruning.arXiv preprint arXiv:2310.06694, 2023

    2023 arXiv

  5. [13]

    Shortened llama: A simple depth pruning for large language models

    Bo-Kyeong Kim, Geonmin Kim, Tae-Ho Kim, Thibault Castells, Shinkook Choi, Junho Shin, and Hyoung-Kyu Song. Shortened llama: A simple depth pruning for large language models. arXiv preprint arXiv:2402.02834, 11, 2024

    2024 arXiv

  6. [14]

    Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    Xinyin Ma, Gongfan Fang, and Xinchao Wang. Llm-pruner: On the structural pruning of large language models.Advances in neural information processing systems, 36:21702–21720, 2023

    2023

  7. [15]

    Disco: Distilling counterfactuals with large language models.arXiv preprint arXiv:2212.10534, 2022

    Zeming Chen, Qiyue Gao, Antoine Bosselut, Ashish Sabharwal, and Kyle Richardson. Disco: Distilling counterfactuals with large language models.arXiv preprint arXiv:2212.10534, 2022. 11

    2022 arXiv

  8. [16]

    Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes.arXiv preprint arXiv:2305.02301, 2023

    Cheng-Yu Hsieh, Chun-Liang Li, Chih-Kuan Yeh, Hootan Nakhost, Yasuhisa Fujii, Alexander Ratner, Ranjay Krishna, Chen-Yu Lee, and Tomas Pfister. Distilling step-by-step! outperform- ing larger language models with less training data and smaller model sizes.arXiv preprint arXiv:...

    2023 arXiv

  9. [17]

    Distilling reasoning capabilities into smaller language models.Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, 2023

    Kumar Shridhar, Alessandro Stolfo, and Mrinmaya Sachan. Distilling reasoning capabilities into smaller language models.Findings of the Association for Computational Linguistics: ACL 2023, pages 7059–7073, 2023

    2023

  10. [18]

    Zephyr: Direct distillation of lm alignment.arXiv preprint arXiv:2310.16944, 2023

    Lewis Tunstall, Edward Beeching, Nathan Lambert, Nazneen Rajani, Kashif Rasul, Younes Belkada, Shengyi Huang, Leandro V on Werra, Clémentine Fourrier, Nathan Habib, et al. Zephyr: Direct distillation of lm alignment.arXiv preprint arXiv:2310.16944, 2023

    arXiv

  11. [19]

    Zeroquant: Efficient and affordable post-training quantization for large-scale transformers

    Zhewei Yao, Reza Yazdani Aminabadi, Minjia Zhang, Xiaoxia Wu, Conglong Li, and Yuxiong He. Zeroquant: Efficient and affordable post-training quantization for large-scale transformers. Advances in Neural Information Processing Systems, 35:27168–27183, 2022

    2022

  12. [20]

    A survey of quantization methods for efficient neural network inference

    Amir Gholami, Sehoon Kim, Zhen Dong, Zhewei Yao, Michael W Mahoney, and Kurt Keutzer. A survey of quantization methods for efficient neural network inference. InLow-power computer vision, pages 291–326. Chapman and Hall/CRC, 2022

    2022

  13. [21]

    Qlora: Efficient finetuning of quantized llms.Advances in neural information processing systems, 36:10088– 10115, 2023

    Tim Dettmers, Artidoro Pagnoni, Ari Holtzman, and Luke Zettlemoyer. Qlora: Efficient finetuning of quantized llms.Advances in neural information processing systems, 36:10088– 10115, 2023

    2023

  14. [22]

    Shortgpt: Layers in large language models are more redundant than you expect.arXiv preprint arXiv:2403.03853, 2024

    Xin Men, Mingyu Xu, Qingyu Zhang, Bingning Wang, Hongyu Lin, Yaojie Lu, Xianpei Han, and Weipeng Chen. Shortgpt: Layers in large language models are more redundant than you expect.arXiv preprint arXiv:2403.03853, 2024

    2024 arXiv

  15. [23]

    Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing inference time

    Mitchell Wortsman, Gabriel Ilharco, Samir Ya Gadre, Rebecca Roelofs, Raphael Gontijo-Lopes, Ari S Morcos, Hongseok Namkoong, Ali Farhadi, Yair Carmon, Simon Kornblith, et al. Model soups: averaging weights of multiple fine-tuned models improves accuracy without increasing infe...

    2022

  16. [24]

    Smac3: A versatile bayesian optimization package for hyperparameter optimization.Journal of Machine Learning Research, 23(54):1–9, 2022

    Marius Lindauer, Katharina Eggensperger, Matthias Feurer, André Biedenkapp, Difan Deng, Carolin Benjamins, Tim Ruhkopf, René Sass, and Frank Hutter. Smac3: A versatile bayesian optimization package for hyperparameter optimization.Journal of Machine Learning Research, 23(54):1–9, 2022

    2022

  17. [25]

    Slicegpt: Compress large language models by deleting rows and columns

    Saleh Ashkboos, Maximilian L Croci, Marcelo Gennari do Nascimento, Torsten Hoefler, and James Hensman. Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024, 2024

    2024 arXiv

  18. [26]

    Laco: Large language model pruning via layer collapse

    Yifei Yang, Zouying Cao, and Hai Zhao. Laco: Large language model pruning via layer collapse. arXiv preprint arXiv:2402.11187, 2024

    2024 arXiv

  19. [27]

    Structural pruning of pre-trained language models via neural architecture search.arXiv preprint arXiv:2405.02267, 2024

    Aaron Klein, Jacek Golebiowski, Xingchen Ma, Valerio Perrone, and Cedric Archambeau. Structural pruning of pre-trained language models via neural architecture search.arXiv preprint arXiv:2405.02267, 2024

    2024 arXiv

  20. [28]

    Weight averaging for neural networks and local resampling schemes

    Joachim Utans. Weight averaging for neural networks and local resampling schemes. In Proc. AAAI-96 Workshop on Integrating Multiple Learned Models. AAAI Press, pages 133–138. Citeseer, 1996

    1996

  21. [29]

    Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    Gabriel Ilharco, Marco Tulio Ribeiro, Mitchell Wortsman, Suchin Gururangan, Ludwig Schmidt, Hannaneh Hajishirzi, and Ali Farhadi. Editing models with task arithmetic.arXiv preprint arXiv:2212.04089, 2022

    2022 arXiv

  22. [30]

    Sampling generative networks.arXiv preprint arXiv:1609.04468, 2016

    Tom White. Sampling generative networks.arXiv preprint arXiv:1609.04468, 2016

    2016 arXiv

  23. [31]

    Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024

    Prateek Yadav, Derek Tam, Leshem Choshen, Colin A Raffel, and Mohit Bansal. Ties-merging: Resolving interference when merging models.Advances in Neural Information Processing Systems, 36, 2024. 12

    2024

  24. [32]

    Language models are super mario: Absorbing abilities from homologous models as a free lunch

    Le Yu, Bowen Yu, Haiyang Yu, Fei Huang, and Yongbin Li. Language models are super mario: Absorbing abilities from homologous models as a free lunch. InForty-first International Conference on Machine Learning, 2024

    2024

  25. [33]

    Evolutionary optimization of model merging recipes.arXiv preprint arXiv:2403.13187, 2024

    Takuya Akiba, Makoto Shing, Yujin Tang, Qi Sun, and David Ha. Evolutionary optimization of model merging recipes.arXiv preprint arXiv:2403.13187, 2024

    2024 arXiv

  26. [34]

    Fine, i’ll merge it myself: A multi-fidelity framework for automated model merging.arXiv preprint arXiv:2502.04030, 2025

    Guinan Su and Jonas Geiping. Fine, i’ll merge it myself: A multi-fidelity framework for automated model merging.arXiv preprint arXiv:2502.04030, 2025

    2025 arXiv

  27. [35]

    Joshua Knowles. Parego: A hybrid algorithm with on-line landscape approximation for expen- sive multiobjective optimization problems.IEEE transactions on evolutionary computation, 10(1):50–66, 2006

    2006

  28. [36]

    Random forests.Machine learning, 45:5–32, 2001

    Leo Breiman. Random forests.Machine learning, 45:5–32, 2001

    2001

  29. [37]

    Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    OpenCompass Contributors. Opencompass: A universal evaluation platform for foundation models.https://github.com/open-compass/opencompass, 2023

    2023

  30. [38]

    Clue: A chinese language understanding evaluation benchmark.arXiv preprint arXiv:2004.05986, 2020

    Liang Xu, Hai Hu, Xuanwei Zhang, Lu Li, Chenjie Cao, Yudong Li, Yechen Xu, Kai Sun, Dian Yu, Cong Yu, et al. Clue: A chinese language understanding evaluation benchmark.arXiv preprint arXiv:2004.05986, 2020

    2004 arXiv

  31. [39]

    Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can a machine really finish your sentence?arXiv preprint arXiv:1905.07830, 2019

    1905 arXiv

  32. [40]

    Piqa: Reasoning about phys- ical commonsense in natural language

    Yonatan Bisk, Rowan Zellers, Jianfeng Gao, Yejin Choi, et al. Piqa: Reasoning about phys- ical commonsense in natural language. InProceedings of the AAAI conference on artificial intelligence, volume 34, pages 7432–7439, 2020

    2020

  33. [41]

    Chid: A large-scale chinese idiom dataset for cloze test.arXiv preprint arXiv:1906.01265, 2019

    Chujie Zheng, Minlie Huang, and Aixin Sun. Chid: A large-scale chinese idiom dataset for cloze test.arXiv preprint arXiv:1906.01265, 2019

    1906 arXiv

  34. [42]

    The winograd schema challenge.KR, 2012:13th, 2012

    Hector J Levesque, Ernest Davis, and Leora Morgenstern. The winograd schema challenge.KR, 2012:13th, 2012

    2012

  35. [43]

    Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    Alon Talmor, Jonathan Herzig, Nicholas Lourie, and Jonathan Berant. Commonsenseqa: A ques- tion answering challenge targeting commonsense knowledge.arXiv preprint arXiv:1811.00937, 2018

    2018 arXiv

  36. [44]

    Boolq: Exploring the surprising difficulty of natural yes/no questions

    Christopher Clark, Kenton Lee, Ming-Wei Chang, Tom Kwiatkowski, Michael Collins, and Kristina Toutanova. Boolq: Exploring the surprising difficulty of natural yes/no questions. arXiv preprint arXiv:1905.10044, 2019

    1905 arXiv

  37. [45]

    Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    Dan Hendrycks, Collin Burns, Steven Basart, Andy Zou, Mantas Mazeika, Dawn Song, and Jacob Steinhardt. Measuring massive multitask language understanding.arXiv preprint arXiv:2009.03300, 2020

    2009 arXiv

  38. [46]

    Cmmlu: Measuring massive multitask language understanding in chinese

    Haonan Li, Yixuan Zhang, Fajri Koto, Yifei Yang, Hai Zhao, Yeyun Gong, Nan Duan, and Timothy Baldwin. Cmmlu: Measuring massive multitask language understanding in chinese. arXiv preprint arXiv:2306.09212, 2023

    2023 arXiv

  39. [47]

    Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017

    Guokun Lai, Qizhe Xie, Hanxiao Liu, Yiming Yang, and Eduard Hovy. Race: Large-scale reading comprehension dataset from examinations.arXiv preprint arXiv:1704.04683, 2017

    2017 arXiv

  40. [48]

    Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.arXiv preprint arXiv:1808.08745, 2018

    Shashi Narayan, Shay B Cohen, and Mirella Lapata. Don’t give me the details, just the summary! topic-aware convolutional neural networks for extreme summarization.arXiv preprint arXiv:1808.08745, 2018

    2018 arXiv

  41. [49]

    Investigating prior knowledge for challenging chinese machine reading comprehension.Transactions of the Association for Computational Linguistics, 8:141–155, 2020

    Kai Sun, Dian Yu, Dong Yu, and Claire Cardie. Investigating prior knowledge for challenging chinese machine reading comprehension.Transactions of the Association for Computational Linguistics, 8:141–155, 2020. 13

    2020

  42. [50]

    Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    Xiang Yue, Xingwei Qu, Ge Zhang, Yao Fu, Wenhao Huang, Huan Sun, Yu Su, and Wenhu Chen. Mammoth: Building math generalist models through hybrid instruction tuning.arXiv preprint arXiv:2309.05653, 2023

    2023 arXiv

  43. [51]

    llama-2-coder-7b (revision d30d193), 2023

    Manuel Romero. llama-2-coder-7b (revision d30d193), 2023

    2023

  44. [52]

    Wizardlm: Empowering large language models to follow complex instructions

    Can Xu, Qingfeng Sun, Kai Zheng, Xiubo Geng, Pu Zhao, Jiazhan Feng, Chongyang Tao, and Daxin Jiang. Wizardlm: Empowering large language models to follow complex instructions. arXiv preprint arXiv:2304.12244, 2023

    2023 arXiv

  45. [53]

    Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023

    Haipeng Luo, Qingfeng Sun, Can Xu, Pu Zhao, Jianguang Lou, Chongyang Tao, Xiubo Geng, Qingwei Lin, Shifeng Chen, and Dongmei Zhang. Wizardmath: Empowering mathematical rea- soning for large language models via reinforced evol-instruct.arXiv preprint arXiv:2308.09583, 2023

    2023 arXiv

  46. [54]

    Code alpaca: An instruction-following llama model for code generation

    Sahil Chaudhary. Code alpaca: An instruction-following llama model for code generation. https://github.com/sahil280114/codealpaca, 2023

    2023

  47. [55]

    Shallow-deep networks: Understanding and mitigating network overthinking

    Yigitcan Kaya, Sanghyun Hong, and Tudor Dumitras. Shallow-deep networks: Understanding and mitigating network overthinking. InInternational conference on machine learning, pages 3301–3310. PMLR, 2019

    2019

  48. [56]

    The unreasonable ineffectiveness of the deeper layers.arXiv preprint arXiv:2403.17887, 2024

    Andrey Gromov, Kushal Tirumala, Hassan Shapourian, Paolo Glorioso, and Daniel A Roberts. The unreasonable ineffectiveness of the deeper layers.arXiv preprint arXiv:2403.17887, 2024

    2024 arXiv

  49. [57]

    Pointer sentinel mixture models, 2016

    Stephen Merity, Caiming Xiong, James Bradbury, and Richard Socher. Pointer sentinel mixture models, 2016

    2016

  50. [58]

    The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, et al. The llama 3 herd of models.arXiv preprint arXiv:2407.21783, 2024

    2024 arXiv

  51. [59]

    Code-llama-3-8b, 2023

    Ajibawa. Code-llama-3-8b, 2023

    2023

  52. [60]

    Mathcoder: Seamless code integration in LLMs for enhanced mathematical reasoning

    Zimu Lu, Aojun Zhou, Zimu Lu, Sichun Luo, Weikang Shi, Renrui Zhang, Linqi Song, Mingjie Zhan, and Hongsheng Li. Mathcoder: Seamless code integration in LLMs for enhanced mathematical reasoning. InThe Twelfth International Conference on Learning Representations, 2024

    2024

  53. [61]

    Mathcoder2: Better math reasoning from continued pretraining on model- translated mathematical code, 2024

    Zimu Lu, Aojun Zhou, Ke Wang, Houxing Ren, Weikang Shi, Junting Pan, Mingjie Zhan, and Hongsheng Li. Mathcoder2: Better math reasoning from continued pretraining on model- translated mathematical code, 2024

    2024

  54. [62]

    Goldilocks

    Jack W Rae, Anna Potapenko, Siddhant M Jayakumar, and Timothy P Lillicrap. Compressive transformers for long-range sequence modelling.arXiv preprint arXiv:1911.05507, 2019. A Baseline To ensure fair comparison, we applied various baseline pruning methods including LLM- Pruner(...

    2020 arXiv

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.