Pith. sign in

REVIEW 4 major objections 5 minor 61 references

Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper shows that a multimodal LLM can be compressed by up to 60% via structural pruning and recovery training while retaining 78–84% of its original benchmark performance.

desk verdict Useful empirical map of MLLM pruning and recovery, with real but fixable weaknesses in calibration and claims. read the letter →

arxiv 2507.20749 v1 pith:53QNGHKK submitted 2025-07-28 cs.CL cs.CV

Yiran Huang , Lukas Thede , Massimiliano Mancini , Wenjia Xu , Zeynep Akata This is my paper
classification cs.CLcs.CV
keywords multimodallargelanguagemodelsstructuralpruninglayerwisewidthwiseknowledgedistillationmodelcompressionsupervisedfinetuningdata-efficientrecovery
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper asks whether an already-trained multimodal large language model can be shrunk directly—by cutting entire transformer layers or by removing attention heads and MLP neurons—and then patched back up with cheap recovery training, instead of training a small model from scratch. It argues that the best pruning style depends on the budget: widthwise pruning holds up better when compute or data are scarce, while layerwise pruning edges ahead once recovery training is available at moderate compression. It further claims that below about 20% compression, retraining only the multimodal projector is enough, and that supervised finetuning combined with L2 matching of hidden states gives the most reliable recovery at every compression level. Taken together, the results sketch a practical decision chart: up to 60% compression is survivable, retaining roughly 78–84% of average benchmark performance, and as little as 5% of the original training data can suffice below 50% compression.

What carries the argument

The argument is carried by two structural pruning mechanisms applied to the language backbone. Layerwise pruning ranks transformer layers with the Block Influence score—the cosine distance between a layer's input and output hidden states on a calibration set—and deletes the lowest-ranked layers. Widthwise pruning builds a dependency graph of attention heads and MLP neurons inside each layer, so that any neuron whose only upstream or downstream connection is pruned goes with it, and ranks these groups with a first-order Taylor approximation of the loss change; groups with the lowest summed importance are removed. Recovery is then driven by three complementary losses: supervised cross-entropy finetuning (on the projector alone, or the projector plus a low-rank adapted LLM), logits-based knowledge distillation with forward or reverse KL, and L2 matching of the final-layer hidden states against the original unpruned teacher. The central quantity that ties the whole study together is the compression ratio—the fraction of language-model parameters removed—and the paper maps every pruning and recovery choice against it.

What would settle it

Take a fixed model and compression ratio, draw several different 10-sample calibration sets from the same training data, and rerun the pruning and recovery pipeline for each draw; if the chosen layers or dependency groups and the resulting benchmark scores vary substantially across draws, the reported ordering of pruning and recovery strategies is not stable.

Watch

Extended reading notes

Core claim

The paper's central claim is that structural pruning of the language-model backbone, followed by lightweight recovery training, is a viable and flexible route to compressing MLLMs, and that the right choice of pruning paradigm and recovery recipe depends on the compression ratio and available resources. Concretely, widthwise pruning (removing attention heads and MLP units via dependency groups) preserves more performance than layerwise pruning (dropping entire transformer blocks) when no recovery training is affordable; once recovery is allowed, layerwise pruning is slightly better below roughly 40% compression and widthwise pruning takes over above that. The paper further claims that finetuning only the multimodal projector restores performance at small compression ratios because pruning mainly damages vision–language alignment there, while supervised finetuning of the projector plus LLM, combined with hidden-state L2 distillation from the original teacher, is the strongest recovery recipe across all ratios. With this recipe, the paper reports retaining over 95% of original performance at compression ratios below 30% and 78–84% average benchmark performance at 60% compression, and matching full-data recovery with only 5% of the training data when compression stays below about 50%.

Load-bearing premise

All pruning decisions are made from importance scores computed on only 10 randomly chosen calibration samples, so if those ten samples do not represent the visual instruction distribution, the comparisons between pruning methods and recovery recipes could reflect sampling noise rather than real structural redundancy.

Editorial extensions

If this is right

  • Practitioners can compress an existing 7B or 3B MLLM by up to 60% and keep roughly 78–84% of average benchmark performance, without training a small model from scratch.
  • At compression ratios below about 20%, updating only the multimodal projector recovers most lost performance, so recovery costs little more than a projection-layer finetune.
  • Combining supervised finetuning with hidden-state L2 distillation from the original model is the dependable recovery recipe across all compression levels, whereas distillation alone can collapse at high ratios.
  • When compression stays below roughly 50%, as little as 5% of the original visual instruction data matches full-data recovery, sharply cutting the cost of compression.
  • The resulting decision rule—widthwise pruning without recovery for low-resource settings, layerwise with recovery at moderate ratios, widthwise with distillation at high ratios—transfers to a third architecture in the appendix, supporting its generality.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension the paper does not run is to make the calibration set adaptive: instead of 10 fixed samples, drawing a small batch of calibration samples per layer or per group could stabilise importance estimates and push the safe compression threshold higher.
  • Because hidden-state L2 matching only used the final layer, and the appendix shows diminishing returns from matching more layers, one could test layerwise mapping with a learned projection, which might matter when widthwise pruning changes hidden dimensions and layer counts unequally.
  • The reported latency penalty for quantization alone, versus the balanced latency when pruning is combined with quantization, suggests a combined pruning-plus-quantization pipeline as a natural next step for deployment under strict latency budgets.
  • The claim that 5% of data suffices below 50% compression is measured on visual instruction tuning; an untested extension is whether the same data efficiency holds for specialized domains, where the small calibration set would need to cover the target distribution.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper asks whether structural pruning of the language-model backbone of multimodal LLMs, followed by lightweight recovery training, can yield accurate compressed models. It compares layerwise (depthwise) and widthwise pruning on LLaVA-v1.5-7B, Bunny-v1.0-3B, and Mini-InternVL-Chat-4B, and evaluates recovery via projector-only finetuning, projector+LLM finetuning, logits-based KD, and hidden-state L2 distillation. The main empirical claims are: widthwise pruning preserves more accuracy without recovery in low-resource settings; projector-only finetuning suffices below 20% compression; supervised finetuning combined with hidden-state L2 distillation gives the best recovery across ratios; and 5% of the training data suffices for compression below 50%. The paper distills these findings into a practitioner decision flow and reports performance retention of 83–84% at 60% compression.

Significance. If the claims hold, the paper provides actionable, practical guidance for compressing existing MLLMs without retraining from smaller language models, which is currently a gap in the literature. The study's strengths include a clean head-to-head comparison of two pruning paradigms, recovery ablations over data fractions, and a generalization check on a third architecture (InternVL). The decision-flow summary (Figure 1) is a useful artifact for practitioners. The main limitations are the reliance on a single 10-sample calibration draw for all pruning decisions and some imprecise reporting of compression ratios and data-efficiency claims; these affect the reliability of the headline recommendations but are, in principle, fixable with additional experiments and careful rewriting.

major comments (4)
  1. [Section 4 (Experimental Setup) and all downstream tables] The calibration set for both importance scores is fixed to n=10 randomly selected samples (Eqs. 2 and 4–5), with no seed, no repeated draws, and no variance reporting. Since these scores determine exactly which layers/neurons are pruned at every compression ratio, every subsequent comparison — widthwise vs. layerwise, projector-only vs. joint FT, FT vs. KD, and the data-efficiency curves in Fig. 4 — is conditional on one arbitrary 10-sample draw from a 665k/695k dataset. Table 1 already shows a ranking reversal at 45% compression on LLaVA without recovery (depthwise 38.60% vs widthwise 35.49%), which contradicts the 'widthwise is better' summary and could be sampling noise. Please report results over multiple calibration draws (e.g., 5–10 seeds) with standard deviations, or at minimum demonstrate that the pruning masks are stable across different 10-sample draws.
  2. [Abstract and Section 4.4 (Data Efficient Recovery)] The abstract's claim that 'effective recovery can be achieved with as little as 5% of the original training data, while retaining over 95% of the original performance' conflates two different reference points. Section 4.4 and Figure 4 measure performance relative to full-data recovery training, not relative to the uncompressed model. The y-axis label 'Avg. Rel. Performance vs. Full Data (%)' makes this explicit, and the takeaway later states 'performance comparable to full data training.' The 95% figure is therefore not evidence that 5% data recovers 95% of the original model's accuracy. Please correct the abstract and any other places that state the claim as 'original performance,' or provide a separate plot showing recovery relative to the uncompressed baseline.
  3. [Section 4 (Experimental Setup), Table 1, and Table 8] The compression ratio definition is internally inconsistent. The text says 'the compression ratio (Ratio) indicates the proportion of remaining language model parameters compared to the pre-pruning state,' but Table 1 lists a 15% ratio for a model whose size drops from 7.0B to 6.3B, which would correspond to 90% remaining parameters, not 15%. Moreover, if 15% of the LLM parameters (6.7B of the 7.0B total) were pruned, the expected total size would be about 6.0B, not 6.3B. This ambiguity affects every quantitative claim in the paper. Please clarify whether 'Ratio' is the fraction of parameters removed or the fraction remaining, and ensure the reported sizes and ratios are mutually consistent.
  4. [Section 4.2 and Figure 2] The claim that 'at a compression ratio of 40%, finetuning both the projector and the LLM restores more than 80% of the original model's performance' is not directly supported by a data point at 40%; the closest measurements are at 30% and 45%. Since the decision flow in Figure 1 uses 40% as a boundary between layerwise and widthwise strategies, please either report results at 40% or explicitly state that the 40% threshold is an interpolation between the 30% and 45% measurements.
minor comments (5)
  1. [Section 4.3, Table 3] The text states that coupling FT with KD 'provides consistent gains of 3 to 23 percentage points,' but in Table 3 the RKL+FT gain over FT alone at 60% compression is only 0.47 percentage points (69.85 vs 69.38), which falls outside the stated range. Please revise the sentence to reflect the actual gain values.
  2. [Appendix A.2, Table 6] The compression ratios in Table 6 (12.8%, 25.5%, 39.0%, 51.8%) differ from the nominal 15%, 30%, 45%, 60% used elsewhere. Please explain whether these are the actual achieved ratios for the widthwise-pruned Bunny model and reconcile them with the main-text ratios.
  3. [Section 4, Table 8] The note in Table 8 repeats the ambiguous compression-ratio language ('proportion of remaining LLM parameters compared to the pre-pruning state'), which conflicts with the reported sizes. Please harmonize the definition across all tables and the main text.
  4. [Section 5.2, Table 4] The text says LLM.int8() incurs 'a fourfold increase in latency,' but Table 4 shows 398 ms vs 105 ms, which is 3.8×, and the ± values further qualify the comparison. Please use the exact ratio or say 'approximately fourfold.'
  5. [Throughout] There are several typos and missing spaces, e.g., 'multimoal' in reference [3], 'LlaV' in the Figure 6 caption, and 'on on' in Eq. (3) ('Ni is dependent on on Ni'). A careful proofread is recommended.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical benchmarking study whose recommendations directly summarize measured results; no derivation reduces to its inputs.

full rationale

The manuscript does not derive a formal prediction from an input definition. Its central claims (widthwise pruning better in low-resource settings, projector-only finetuning sufficient below 20% compression, FT plus L2 distillation best for recovery, and 5% data sufficing below 50% compression) are direct readings of benchmark tables and figures (Tables 1, 3, 8; Figures 2-4). The pruning importance scores in Eqs. (2) and (4)-(5) are computed on a calibration set and used to select which structures to remove, but the evaluations are performed on unseen multimodal benchmarks; the reported metrics are not defined in terms of those importance scores, so there is no self-definitional reduction and no fitted-input-called-prediction step. The decision flowchart in Fig. 1 is a summary of the same experimental grid rather than an independent derivation, which is in-sample description rather than circularity. The only author self-citation is [14] (MultiFlow, co-authored by M. Mancini) in a related-work list of pruning methods; it is not load-bearing for any conclusion. The n=10 calibration sample and the absence of repeated draws are robustness or statistical limitations, and the abstract's '95% of original performance' phrasing refers to performance relative to full-data recovery in Sec. 4.4 rather than the uncompressed model; both are correctness or reporting concerns, not circular steps.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The paper rests on standard pruning heuristics (BI score, Taylor importance) and a very small calibration set. The key practical findings do not depend on a fitted theoretical model, so the parameter burden is mainly experimental hyperparameters that are underreported.

free parameters (4)
  • calibration dataset size = 10 samples
    Number of randomly selected samples used to compute importance scores for pruning; not varied or justified, and central to pruning decisions.
  • distillation loss weights (alpha, beta, gamma) = not reported
    Weights balancing SFT, logits KD, and hidden-state matching in the total loss; no values given in the main text.
  • LoRA rank and learning rate for LLM finetuning = not reported
    Hyperparameters for recovery finetuning; not specified, affects the recovery results.
  • KD temperature = 2.0
    Temperature for logits-based distillation, stated in experiments but not optimized.
assumptions (4)
  • domain assumption Layers that cause larger changes in hidden states have greater influence on model performance (Block Influence score).
    Section 3.1 states this as the key assumption behind layerwise pruning.
  • ad hoc to paper The 10-sample calibration set is representative of the full visual instruction dataset.
    Section 4: pruning importance is computed from 10 random samples, with no evidence of stability.
  • domain assumption Finetuning the vision encoder does not improve performance and is omitted.
    Section 3.2 cites [29] to keep the vision encoder frozen.
  • standard math Taylor expansion (Eq. 4) adequately approximates the importance of each weight group.
    Standard first-order approximation used in pruning; assumes loss smoothness.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study." pith.science (2026). https://pith.science/paper/53QNGHKK

@misc{pith2026250720749,
  author       = {Pith},
  title        = {Pith review of: Investigating Structural Pruning and Recovery Techniques for Compressing Multimodal Large Language Models: An Empirical Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/53QNGHKK}},
  note         = {Machine review of arXiv:2507.20749}
}
read the original abstract

While Multimodal Large Language Models (MLLMs) demonstrate impressive capabilities, their substantial computational and memory requirements pose significant barriers to practical deployment. Current parameter reduction techniques primarily involve training MLLMs from Small Language Models (SLMs), but these methods offer limited flexibility and remain computationally intensive. To address this gap, we propose to directly compress existing MLLMs through structural pruning combined with efficient recovery training. Specifically, we investigate two structural pruning paradigms--layerwise and widthwise pruning--applied to the language model backbone of MLLMs, alongside supervised finetuning and knowledge distillation. Additionally, we assess the feasibility of conducting recovery training with only a small fraction of the available data. Our results show that widthwise pruning generally maintains better performance in low-resource scenarios with limited computational resources or insufficient finetuning data. As for the recovery training, finetuning only the multimodal projector is sufficient at small compression levels (< 20%). Furthermore, a combination of supervised finetuning and hidden-state distillation yields optimal recovery across various pruning levels. Notably, effective recovery can be achieved with as little as 5% of the original training data, while retaining over 95% of the original performance. Through empirical study on two representative MLLMs, i.e., LLaVA-v1.5-7B and Bunny-v1.0-3B, this study offers actionable insights for practitioners aiming to compress MLLMs effectively without extensive computation resources or sufficient data.

Figures

Figures reproduced from arXiv: 2507.20749 by the authors.

Figure 1
Figure 1. Compression decision flow for MLLMs. The left panel presents a decision flowchart that guides the choice of pruning and recovery based on resource avail￾ability and compression ratio requirements.(i) widthwise pruning only (no recovery) in extremely low-resource settings; (ii) layerwise pruning with MM-projector fine-tuning for moderate compression (≤ 40%); and (iii) widthwise pruning + knowledge distilla￾tion for h… view at source ↗
Figure 2
Figure 2. Comparison of pruning and finetuning strategies on two MLLMs. The plot shows the average relative performance under three scenarios: pruning only, pruning followed by finetuning the projector, and pruning followed by finetuning both the pro￾jector and the LLM. For smaller compression ratios (<20%), finetuning only the pro￾jector effectively recovers performance. For larger compression ratios, jointly finetuning the … view at source ↗
Figure 3
Figure 3. Comparison of different distillation recovery strategies (KL loss, RKL loss, L2 loss, and their combinations) for Bunny and LlaVA models pruned with widthwise pruning. The plot shows the relative performance improvement of each strategy over standard finetuning across various compression ratios. The results demonstrate that distillation helps recover more performance than finetuning alone, with the L2 loss component… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Comparison of recovery performance using different percentages of training data (100%, 20%, 10%, and 5%) for finetuning and distillation after pruning across Bunny and Llava models. For smaller compression ratios, even a small percentage of the training data (as low as…
Figure 5
Figure 5. Figure 5: and [PITH_FULL_IMAGE:figures/full_fig_p019_5.png]
Figure 6
Figure 6. Figure 6: Comparison of L2 and RKL distillation strategies with and without additional fine-tuning loss for Bunny and LLaV models compressed by layerwise pruning. The plot shows performance differences relative to standard fine-tuning across varying com￾pression ratios. 0 10 20 …
Figure 7
Figure 7. Figure 7: Comparison of different distillation recovery strategies (KL loss, RKL loss, L2 loss, and their combinations) for Bunny and LlaVA models pruned with layerwise pruning. The plot shows the relative performance improvement of each strategy over standard fine-tuning across…

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

61 extracted references · 14 canonical work pages

  1. [1]

    arXiv preprint arXiv:2401.15024 (2024)

    Ashkboos, S., Croci, M.L., Nascimento, M.G.d., Hoefler, T., Hensman, J.: Slicegpt: Compress large language models by deleting rows and columns. arXiv preprint arXiv:2401.15024 (2024)

    arXiv 2024

  2. [2]

    Bai, H., Zhang, W., Hou, L., Shang, L., Jin, J., Jiang, X., Liu, Q., Lyu, M., King, I.: Binarybert: Pushing the limit of bert quantization (2021), https://arxiv.org/ abs/2012.15701

    arXiv 2021

  3. [3]

    Bo, L., Peiyuan, Z., Kaichen, Z., Fanyi, P., Xinrun, D., Yuhao, D., Haotian, L., Yuanhan, Z., Ge, Z., Chunyuan, L., Ziwei, L.: Lmms-eval: Accelerating the development of large multimoal models (March 2024), https://github.com/ EvolvingLMMs-Lab/lmms-eval

    work page 2024

  4. [4]

    arXiv preprint arXiv:2404.16821 (2024)

    Chen, Z., Wang, W., Tian, H., Ye, S., Gao, Z., Cui, E., Tong, W., Hu, K., Luo, J., Ma, Z., et al.: How far are we to gpt-4v? closing the gap to commercial multimodal models with open-source suites. arXiv preprint arXiv:2404.16821 (2024)

    arXiv 2024

  5. [5]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Chen, Z., Wu, J., Wang, W., Su, W., Chen, G., Xing, S., Zhong, M., Zhang, Q., Zhu, X., Lu, L., et al.: Internvl: Scaling up vision foundation models and aligning for generic visual-linguistic tasks. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 24185–24198 (2024)

    2024

  6. [6]

    Chiang, W.L., Li, Z., Lin, Z., Sheng, Y., Wu, Z., Zhang, H., Zheng, L., Zhuang, S., Zhuang, Y., Gonzalez, J.E., Stoica, I., Xing, E.P.: Vicuna: An open-source chatbot impressing gpt-4 with 90%* chatgpt quality (March 2023), https://lmsys.org/blog/ 2023-03-30-vicuna/

    work page 2023

  7. [7]

    arXiv preprint arXiv:2312.16886 (2023)

    Chu, X., Qiao, L., Lin, X., Xu, S., Yang, Y., Hu, Y., Wei, F., Zhang, X., Zhang, B., Wei, X., et al.: Mobilevlm: A fast, reproducible and strong vision language assistant for mobile devices. arXiv preprint arXiv:2312.16886 (2023)

    arXiv 2023

  8. [8]

    Dery, L., Kolawole, S., Kagy, J.F., Smith, V., Neubig, G., Talwalkar, A.: Everybody prune now: Structured pruning of llms with only forward passes (2024), https: //arxiv.org/abs/2402.05406

    arXiv 2024

Show all 61 references
  1. [9]

    int8 (): 8-bit matrix multiplicationfortransformersatscale.AdvancesinNeuralInformationProcessing Systems 35, 30318–30332 (2022)

    Dettmers, T., Lewis, M., Belkada, Y., Zettlemoyer, L.: Gpt3. int8 (): 8-bit matrix multiplicationfortransformersatscale.AdvancesinNeuralInformationProcessing Systems 35, 30318–30332 (2022)

    2022

  2. [10]

    Ding, X., Ding, G., Guo, Y., Han, J.: Centripetal sgd for pruning very deep convo- lutional networks with complicated structure (2019), https://arxiv.org/abs/1904. 03837

    2019

  3. [11]

    Dong, X., Chen, S., Pan, S.J.: Learning to prune deep neural networks via layer- wise optimal brain surgeon (2017), https://arxiv.org/abs/1705.07565

    2017 arXiv

  4. [12]

    arXiv preprint arXiv:1909.11556 (2019)

    Fan, A., Grave, E., Joulin, A.: Reducing transformer depth on demand with struc- tured dropout. arXiv preprint arXiv:1909.11556 (2019)

    2019 arXiv

  5. [13]

    In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition

    Fang,G.,Ma,X.,Song,M.,Mi,M.B.,Wang,X.:Depgraph:Towardsanystructural pruning. In: Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. pp. 16091–16101 (2023)

    2023

  6. [14]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Farina, M., Mancini, M., Cunegatti, E., Liu, G., Iacca, G., Ricci, E.: Multiflow: Shifting towards task-agnostic vision-language pruning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 16185– 16195 (2024)

    2024

  7. [15]

    Frankle, J., Carbin, M.: The lottery ticket hypothesis: Finding sparse, trainable neural networks (2019), https://arxiv.org/abs/1803.03635

    2019 arXiv

  8. [16]

    International Journal of Computer Vision 129(6), 1789–1819 (Mar Pruning and Recovery Techniques for Compressing MLLMs 15 2021)

    Gou, J., Yu, B., Maybank, S.J., Tao, D.: Knowledge distillation: A sur- vey. International Journal of Computer Vision 129(6), 1789–1819 (Mar Pruning and Recovery Techniques for Compressing MLLMs 15 2021). https://doi.org/10.1007/s11263-021-01453-z, http://dx.doi.org/10.1007/ s...

    2021 doi

  9. [18]

    Gu, Y., Dong, L., Wei, F., Huang, M.: Minillm: Knowledge distillation of large language models (2024), https://arxiv.org/abs/2306.08543

    2024 arXiv

  10. [19]

    arXiv preprint arXiv:2402.11530 (2024)

    He, M., Liu, Y., Wu, B., Yuan, J., Wang, Y., Huang, T., Zhao, B.: Efficient mul- timodal learning from data-centric perspective. arXiv preprint arXiv:2402.11530 (2024)

    2024 arXiv

  11. [20]

    Hinton, G., Vinyals, O., Dean, J.: Distilling the knowledge in a neural network (2015), https://arxiv.org/abs/1503.02531

    2015 arXiv

  12. [21]

    arXiv preprint arXiv:1904.09751 (2019)

    Holtzman, A., Buys, J., Du, L., Forbes, M., Choi, Y.: The curious case of neural text degeneration. arXiv preprint arXiv:1904.09751 (2019)

    2019 arXiv

  13. [22]

    Hsu, Y.C., Hua, T., Chang, S., Lou, Q., Shen, Y., Jin, H.: Language model com- pression with weighted low-rank factorization (2022), https://arxiv.org/abs/2207. 00112

    2022

  14. [24]

    arXiv preprint arXiv:2106.09685 (2021)

    Hu, E.J., Shen, Y., Wallis, P., Allen-Zhu, Z., Li, Y., Wang, S., Wang, L., Chen, W.: Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685 (2021)

    2021 arXiv

  15. [25]

    In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition

    Hudson, D.A., Manning, C.D.: Gqa: A new dataset for real-world visual reasoning and compositional question answering. In: Proceedings of the IEEE/CVF confer- ence on computer vision and pattern recognition. pp. 6700–6709 (2019)

    2019

  16. [26]

    Microsoft Research Blog (2023)

    Javaheripi, M., Bubeck, S., Abdin, M., Aneja, J., Bubeck, S., Mendes, C.C.T., Chen, W., Del Giorno, A., Eldan, R., Gopi, S., et al.: Phi-2: The surprising power of small language models. Microsoft Research Blog (2023)

    2023

  17. [27]

    Master’s thesis, University of Washington (2024)

    Jiang, F.: Identifying and mitigating vulnerabilities in llm-integrated applications. Master’s thesis, University of Washington (2024)

    2024

  18. [28]

    Jiao,X.,Yin,Y.,Shang,L.,Jiang,X.,Chen,X.,Li,L.,Wang,F.,Liu,Q.:Tinybert: Distilling bert for natural language understanding (2020), https://arxiv.org/abs/ 1909.10351

    2020 arXiv

  19. [29]

    arXiv preprint arXiv:2402.07865 (2024)

    Karamcheti, S., Nair, S., Balakrishna, A., Liang, P., Kollar, T., Sadigh, D.: Pris- matic vlms: Investigating the design space of visually-conditioned language models. arXiv preprint arXiv:2402.07865 (2024)

    2024 arXiv

  20. [30]

    Lan, Z., Chen, M., Goodman, S., Gimpel, K., Sharma, P., Soricut, R.: Albert: A lite bert for self-supervised learning of language representations (2020), https: //arxiv.org/abs/1909.11942

    2020 arXiv

  21. [31]

    Lee, N., Ajanthan, T., Gould, S., Torr, P.H.S.: A signal propagation perspective for pruning neural networks at initialization (2020), https://arxiv.org/abs/1906.06307

    2020 arXiv

  22. [32]

    Li, H., Kadav, A., Durdanovic, I., Samet, H., Graf, H.P.: Pruning filters for efficient convnets (2017), https://arxiv.org/abs/1608.08710

    2017 arXiv

  23. [33]

    arXiv preprint arXiv:2305.10355 (2023)

    Li, Y., Du, Y., Zhou, K., Wang, J., Zhao, W.X., Wen, J.R.: Evaluating object hal- lucination in large vision-language models. arXiv preprint arXiv:2305.10355 (2023)

    2023 arXiv

  24. [34]

    org/abs/2011.00593

    Liang, K.J., Hao, W., Shen, D., Zhou, Y., Chen, W., Chen, C., Carin, L.: Mixkd: Towards efficient distillation of large-scale language models (2021), https://arxiv. org/abs/2011.00593

    2021 arXiv

  25. [35]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Liu, H., Li, C., Li, Y., Lee, Y.J.: Improved baselines with visual instruction tuning. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 26296–26306 (2024) 16 Huang et al

    2024

  26. [36]

    org/abs/2304.08485

    Liu, H., Li, C., Wu, Q., Lee, Y.J.: Visual instruction tuning (2023), https://arxiv. org/abs/2304.08485

    2023 arXiv

  27. [37]

    Liu, L., Zhang, S., Kuang, Z., Zhou, A., Xue, J.H., Wang, X., Chen, Y., Yang, W., Liao, Q., Zhang, W.: Group fisher pruning for practical network compression (2021), https://arxiv.org/abs/2108.00708

    2021 arXiv

  28. [38]

    Advances in Neural Information Processing Systems 35, 2507–2521 (2022)

    Lu, P., Mishra, S., Xia, T., Qiu, L., Chang, K.W., Zhu, S.C., Tafjord, O., Clark, P., Kalyan, A.: Learn to explain: Multimodal reasoning via thought chains for science question answering. Advances in Neural Information Processing Systems 35, 2507–2521 (2022)

    2022

  29. [39]

    Advances in neural information processing systems36, 21702–21720 (2023)

    Ma, X., Fang, G., Wang, X.: Llm-pruner: On the structural pruning of large lan- guage models. Advances in neural information processing systems36, 21702–21720 (2023)

    2023

  30. [40]

    arXiv preprint arXiv:1910.06360 (2019)

    McCarley, J., Chakravarti, R., Sil, A.: Structured pruning of a bert-based question answering model. arXiv preprint arXiv:1910.06360 (2019)

    2019 arXiv

  31. [41]

    arXiv preprint arXiv:2403.03853 (2024)

    Men, X., Xu, M., Zhang, Q., Wang, B., Lin, H., Lu, Y., Han, X., Chen, W.: Shortgpt: Layers in large language models are more redundant than you expect. arXiv preprint arXiv:2403.03853 (2024)

    2024 arXiv

  32. [42]

    Michel, P., Levy, O., Neubig, G.: Are sixteen heads really better than one? Ad- vances in neural information processing systems32 (2019)

    2019

  33. [43]

    Park, S., Lee, J., Mo, S., Shin, J.: Lookahead: A far-sighted alternative of magnitude-based pruning (2020), https://arxiv.org/abs/2002.04809

    2020 arXiv

  34. [44]

    arXiv preprint arXiv:2404.16637 (2024)

    Popp, N., Metzen, J.H., Hein, M.: Zero-shot distillation for image encoders: How to make effective use of synthetic data. arXiv preprint arXiv:2404.16637 (2024)

    2024 arXiv

  35. [45]

    In: International conference on machine learning

    Radford, A., Kim, J.W., Hallacy, C., Ramesh, A., Goh, G., Agarwal, S., Sastry, G., Askell, A., Mishkin, P., Clark, J., et al.: Learning transferable visual models from natural language supervision. In: International conference on machine learning. pp. 8748–8763. PMLR (2021)

    2021

  36. [46]

    Computer Speech & Language77, 101429 (2023)

    Sajjad, H., Dalvi, F., Durrani, N., Nakov, P.: On the effect of dropping layers of pre-trained transformer models. Computer Speech & Language77, 101429 (2023)

    2023

  37. [47]

    Sanh, V., Debut, L., Chaumond, J., Wolf, T.: Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter (2020), https://arxiv.org/abs/1910.01108

    2020 arXiv

  38. [48]

    Sanh, V., Wolf, T., Rush, A.M.: Movement pruning: Adaptive sparsity by fine- tuning (2020), https://arxiv.org/abs/2005.07683

    2020 arXiv

  39. [49]

    Sun, S., Cheng, Y., Gan, Z., Liu, J.: Patient knowledge distillation for bert model compression (2019), https://arxiv.org/abs/1908.09355

    2019 arXiv

  40. [50]

    arXiv preprint arXiv:2302.13971 (2023)

    Touvron, H., Lavril, T., Izacard, G., Martinet, X., Lachaux, M.A., Lacroix, T., Rozière, B., Goyal, N., Hambro, E., Azhar, F., et al.: Llama: Open and efficient foundation language models. arXiv preprint arXiv:2302.13971 (2023)

    2023 arXiv

  41. [51]

    arXiv preprint arXiv:1905.09418 (2019)

    Voita, E., Talbot, D., Moiseev, F., Sennrich, R., Titov, I.: Analyzing multi-head self-attention: Specialized heads do the heavy lifting, the rest can be pruned. arXiv preprint arXiv:1905.09418 (2019)

    2019 arXiv

  42. [52]

    Wang, W., Wei, F., Dong, L., Bao, H., Yang, N., Zhou, M.: Minilm: Deep self- attention distillation for task-agnostic compression of pre-trained transformers (2020), https://arxiv.org/abs/2002.10957

    2020 arXiv

  43. [53]

    Xia, M., Gao, T., Zeng, Z., Chen, D.: Sheared llama: Accelerating language model pre-training via structured pruning (2024), https://arxiv.org/abs/2310.06694

    2024 arXiv

  44. [54]

    org/abs/2402.13116 Pruning and Recovery Techniques for Compressing MLLMs 17

    Xu, X., Li, M., Tao, C., Shen, T., Cheng, R., Li, J., Xu, C., Tao, D., Zhou, T.: A survey on knowledge distillation of large language models (2024), https://arxiv. org/abs/2402.13116 Pruning and Recovery Techniques for Compressing MLLMs 17

    2024 arXiv

  45. [55]

    In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition

    Yang, C., An, Z., Huang, L., Bi, J., Yu, X., Yang, H., Diao, B., Xu, Y.: Clip-kd: An empirical study of clip model distillation. In: Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. pp. 15952–15962 (2024)

    2024

  46. [56]

    Yao, Z., Aminabadi, R.Y., Zhang, M., Wu, X., Li, C., He, Y.: Zeroquant: Efficient and affordable post-training quantization for large-scale transformers (2022), https: //arxiv.org/abs/2206.01861

    2022 arXiv

  47. [57]

    arXiv preprint arXiv:2306.13549 (2023)

    Yin, S., Fu, C., Zhao, S., Li, K., Sun, X., Xu, T., Chen, E.: A survey on multimodal large language models. arXiv preprint arXiv:2306.13549 (2023)

    2023 arXiv

  48. [58]

    org/abs/1909.08174

    You, Z., Yan, K., Ye, J., Ma, M., Wang, P.: Gate decorator: Global filter pruning method for accelerating deep convolutional neural networks (2019), https://arxiv. org/abs/1909.08174

    2019 arXiv

  49. [59]

    In: Proceedings of CVPR (2024)

    Yue, X., Ni, Y., Zhang, K., Zheng, T., Liu, R., Zhang, G., Stevens, S., Jiang, D., Ren, W., Sun, Y., Wei, C., Yu, B., Yuan, R., Sun, R., Yin, M., Zheng, B., Yang, Z., Liu, Y., Huang, W., Sun, H., Su, Y., Chen, W.: Mmmu: A massive multi- discipline multimodal understanding and ...

    2024

  50. [60]

    In: 2019 Fifth Workshop on Energy Efficient Machine Learn- ing and Cognitive Computing - NeurIPS Edition (EMC2-NIPS)

    Zafrir, O., Boudoukh, G., Izsak, P., Wasserblat, M.: Q8bert: Quantized 8bit bert. In: 2019 Fifth Workshop on Energy Efficient Machine Learn- ing and Cognitive Computing - NeurIPS Edition (EMC2-NIPS). IEEE (Dec 2019). https://doi.org/10.1109/emc2-nips53020.2019.00016, http://dx...

    2019

  51. [61]

    In: Proceedings of the IEEE/CVF International Conference on Computer Vision

    Zhai, X., Mustafa, B., Kolesnikov, A., Beyer, L.: Sigmoid loss for language im- age pre-training. In: Proceedings of the IEEE/CVF International Conference on Computer Vision. pp. 11975–11986 (2023)

    2023

  52. [62]

    arXiv preprint arXiv:2403.06199 (2024)

    Zhu, M., Zhu, Y., Liu, X., Liu, N., Xu, Z., Shen, C., Peng, Y., Ou, Z., Feng, F., Tang, J.: A comprehensive overhaul of multimodal assistant with small language models. arXiv preprint arXiv:2403.06199 (2024)

    2024 arXiv

  53. [63]

    In: Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited

    Zhu, Y., Zhu, M., Liu, N., Xu, Z., Peng, Y.: Llava-phi: Efficient multi-modal assis- tant with small language model. In: Proceedings of the 1st International Workshop on Efficient Multimedia Computing under Limited. pp. 18–22 (2024) 18 Huang et al. A Appendix A.1 Model archite...

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.