Pith. sign in

REVIEW 4 major objections 7 minor 53 references

MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Low-rank LLM compression improves sharply when the compression ratio is allocated per sublayer and per weight matrix, and MGAA does this allocation in two forward passes with no task-specific search.

desk verdict Useful plug-and-play allocation for low-rank LLM compression; the intra-sublayer energy balancing is the real novelty, but the cosine-similarity heuristic needs harder evidence than it gets. read the letter →

arxiv 2507.03294 v1 pith:6GYT74RT submitted 2025-07-04 cs.LG cs.AI

Guangyan Li , Yongqiang Tang , Wensheng Zhang This is my paper · ORCID
classification cs.LGcs.AI
keywords low-rankcompressionparameterallocationlargelanguagemodelscosinesimilarityenergyretentionplug-and-playmodulezero-shotperplexitymodel
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Low-rank compression of large language models usually spends the same compression ratio on every weight matrix, even though some layers and matrices can absorb far more compression than others. MGAA allocates the budget at two granularities: between transformer sublayers, using the cosine similarity between each sublayer's input and output activations as an importance score, and within each sublayer, assigning ranks so that every weight matrix retains the same fraction of its energy. On LLaMA1 at a 50% compression ratio this lowers zero-shot perplexity from 33.21 with PCA to 22.02, a 33.7% improvement, and the same allocation rules improve several existing low-rank methods when plugged into them. If the paper is right, where the compression budget is spent matters as much as how each matrix is decomposed.

What carries the argument

The core machinery is the pair of allocation formulas layered on top of PCA low-rank approximation. The sublayer score is $I = \frac{1}{L}\sum_{m=1}^{L} \frac{X_m^\top Y_m}{\|X_m\|_2\|Y_m\|_2}$, converted into a compression ratio by Z-score normalization and a scaling coefficient via $p = \alpha z + p_t$, then corrected for the parameter-count imbalance between FFN and MHA sublayers. The intra-sublayer rule builds the cumulative normalized energy vector $c_i = \sum_{j \le i} \hat\lambda_j$ from the eigenvalue decomposition $YY^\top = U\Lambda U^\top$, with $\hat\lambda_j = \lambda_j / \sum_i \lambda_i$, and solves a rank-budget optimization that maximizes total retained energy while keeping the retained energy fractions of all matrices within $\epsilon$ of each other. Because the PCA compression loss is $\sum_{i=r+1}^{d_{\mathrm{out}}} \lambda_i$, equalizing retained energy fractions is the mechanism that equalizes per-matrix loss and turns each matrix's eigenvalue spectrum into a rank allocation.

What would settle it

Compress the same model at the same budget twice, once with MGAA's cosine-similarity ordering of sublayer compression ratios and once with the same multiset of ratios randomly shuffled across sublayers; if the shuffled model matches MGAA's perplexity, the resemblance scores carry no information.

Watch

Extended reading notes

Core claim

The paper's central claim is that the quality of a low-rank-compressed LLM is governed largely by the allocation of the rank budget, not only by the decomposition method. It proposes a two-level allocation that needs no downstream validation: a sublayer's importance is measured by the mean cosine similarity between its input and output columns, with lower similarity meaning a larger feature transformation and thus a smaller compression ratio, and within a sublayer each matrix is given a rank so that its retained energy fraction, computed from the normalized eigenvalues of the output autocorrelation matrix, is equal across matrices subject to the sublayer's rank budget. In the PCA feature-space setting the compression loss equals the sum of the truncated eigenvalues, so equalizing retained energy fractions equalizes per-matrix loss. The paper reports that combining the two rules beats uniform PCA on every tested backbone, scales to LLaMA1-13B, and improves ASVD, AWSVD, AFM, and Joint Rank-k Approximation when added to them.

Load-bearing premise

The load-bearing premise is that how much a sublayer's output resembles its input, measured by cosine similarity, faithfully indicates how much compression that sublayer can survive; the paper motivates this qualitatively and never isolates the link in an experiment before relying on it.

Editorial extensions

If this is right

  • At the same 50% parameter budget, MGAA cuts LLaMA1 perplexity from 33.21 to 22.02 relative to PCA, a 33.7% reduction, and the gap widens at higher compression ratios.
  • The allocation runs in two forward passes on task-agnostic calibration data, replacing heuristic search that requires repeated evaluations on validation sets.
  • MGAA works as a plug-and-play module: applying it to SVD, ASVD, AWSVD, AFM, and Joint Rank-k Approximation improves each method's perplexity and reasoning accuracy.
  • The method transfers to larger models and to multimodality: LLaMA1-13B improves up to 70% compression, and LLaVA-1.6 gains on several visual-question benchmarks.
  • Energy-equalization predicts that matrices with slower-accumulating eigenvalue spectra, such as $W_v$, keep higher ranks, while fast-decaying spectra like $W_k$ are compressed more aggressively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to reuse the cosine-similarity importance score as a sensitivity proxy for quantization bit-widths or pruning block sizes, where the same task-agnostic, evaluation-free property is desirable.
  • If energy-equalization is the active ingredient, the intra-sublayer allocation could be computed from weight spectra alone, potentially eliminating the need for calibration data in that step.
  • The gap between MGAA and an oracle that allocates ranks by true measured perplexity sensitivity would quantify how much the cosine-similarity proxy leaves on the table.
  • The Z-score and scaling-coefficient formula assumes compression ratios vary smoothly with importance; a rank-based or clipped transform might be more robust for outlier sublayers such as the first and last layers.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper proposes MGAA, a parameter allocation method for low-rank compression of LLMs, operating at two granularities: (i) between sublayers, compression ratios are set according to the cosine similarity between each sublayer's input and output activations (Eqs. 12-14), under the heuristic that lower similarity indicates greater transformation complexity and hence higher importance; and (ii) within each sublayer, ranks are allocated to weight matrices so that retained energy ratios are roughly equal across matrices, subject to a rank budget (Eq. 19). The method is designed as a plug-and-play module for existing low-rank methods such as PCA, ASVD, AWSVD, and AFM. Experiments on LLaMA1, Vicuna, LLaMA3, Mistral, and LLaVA report consistent perplexity and downstream-accuracy improvements over uniform and heuristic allocation baselines, with the headline result being a 33.7% perplexity reduction over PCA on LLaMA1 at 50% compression.

Significance. If the claims hold, MGAA would be a useful and practical contribution: it is computationally cheap (requiring only forward-pass activations), task-agnostic at compression time, and demonstrably compatible with several low-rank approximation paradigms. The paper's core PCA loss derivation (Eq. 11) is standard and correct, and the empirical comparisons are broad, covering multiple model families, compression ratios, and both language modeling and reasoning benchmarks. The plug-and-play experiments with AFM, ASVD, AWSVD, and Joint Rank-k are valuable and support the generalization claim. The weaknesses are that two load-bearing heuristics (cosine-similarity importance and energy-balance equality) rest on limited empirical justification, and the main hyperparameter alpha is tuned on the same evaluation suites used for the headline comparisons, which tempers the strength of the empirical claims.

major comments (4)
  1. [§3.2, Eqs. (12)-(14)] The cosine-similarity importance measure is not causally validated. The paper gives a qualitative argument (simpler transformations imply more redundancy) and shows end-task improvements, but no experiment isolates whether the ordering of sublayer compression ratios according to cosine similarity is what drives the gains, as opposed to merely using a non-uniform allocation. Table 6 compares cosine against Euclidean, Manhattan, and normalized Euclidean distances, but all are functions of the same activation vectors and do not test the directional claim. Please add an ablation that uses a random or reversed sublayer ordering while keeping the same variance of compression ratios; this would directly test whether the similarity signal, rather than non-uniformity alone, is responsible for the improvement.
  2. [§4.4.6, Table 9] The hyperparameter alpha is selected by evaluating exactly the WikiText2, PTB, and C4 perplexities and the seven-dataset average accuracy that are also used for the main comparisons in Tables 2, 4, and 5. This means the headline results are conditional on a hyperparameter fit to the evaluation suites, which contradicts the abstract's claim that the method compresses 'without task-specific evaluations in the compression process.' Please fix alpha using a held-out validation set (or a separate validation split of the calibration data) and report the main results with that fixed alpha; if Table 9 is intended as sensitivity analysis rather than selection, state this clearly and show that the main conclusions hold for a range of alpha values without choosing the best.
  3. [§3.3, Eq. (19)] The energy-balance constraint is justified primarily by the 'Fixed Energy' row of Table 1, but that row is itself produced by enforcing the same energy-balance principle; the argument is therefore partly circular. The paper should provide independent evidence that equalizing retained energy ratios is what yields the improvement, rather than the larger total retained energy that the constraint happens to induce. Concretely, please compare Eq. (19) against an unconstrained allocation that maximizes total retained energy under the same rank budget, and report results for several values of the balance threshold epsilon (which is currently a free parameter not analyzed in the experiments).
  4. [Table 1] The 'Fixed Rank' and 'Fixed Energy' experiments in Table 1 are reported without variance, repeated runs, or any statistical measure, yet these observations are the empirical foundation for the intra-sublayer allocation scheme. Since the claim is that performance is 'notably comparable' under equal energy retention, please report standard deviations across multiple calibration-data samples or at least a small number of repeated runs; without this, it is difficult to assess whether the observed differences (e.g., Wv perplexity of 7.18 vs 5.75 under fixed rank) are significant or whether the fixed-energy improvements could be noise.
minor comments (7)
  1. [Title] The title contains a typo: 'fof' should be 'for'.
  2. [§2.3] The sentence 'These approachs measure the sensitivity...' should read 'These approaches measure...'.
  3. [Eq. (6)] Equation (6) is missing a closing parenthesis: the expression for Pi(k) ends with 'f(x, Wi(k)),' with no closing parenthesis for the F function.
  4. [Table 5] The AFM rows are duplicated verbatim in Table 5; one of the two blocks should presumably list a different baseline (e.g., AWSVD or another method), and the table caption should clarify which method each block corresponds to.
  5. [Table 3 header] The table header says 'compression rations'; this should be 'compression ratios'.
  6. [§4.6] The text uses 'multi-model' where 'multimodal' is intended (e.g., 'multi-model LLMs' and 'multi-modal model'); please standardize the terminology.
  7. [Figure 1 caption] The caption says the left figure illustrates allocation 'based on the eigenvalues of their input and output features,' but Eq. (12) uses cosine similarity of the raw activations, not eigenvalues; please align the caption with the actual computation.

Circularity Check

1 steps flagged · score 4.0 of 10

MGAA's central allocation mechanism is not circular, but the headline results are partially tuned: the hyperparameter α is selected on the same evaluation benchmarks used to report MGAA's superiority.

  1. fitted input called prediction [Section 4.4.6 'Analysis of Hyperparameters', Table 9; used with main results in Section 4.2, Table 2.]
    "when α = 0.35, the model achieves optimal performance across multiple metrics, with the lowest perplexity on WikiText2 (24.54) and C4 (50.22) datasets, as well as the highest average accuracy (44.59) on downstream tasks. Based on our comprehensive analysis of both perplexity metrics and downstream task performance, we recommend setting α = 0.35 as the optimal configuration for model compression."

    The scalar α in Eq. (14) sets the magnitude of sublayer compression-ratio differentiation. The paper selects α by optimizing exactly the metrics used in the main evaluation (WikiText2/PTB/C4 perplexity and the seven-dataset reasoning average), as shown in Table 9. The headline MGAA results in Table 2 are then reported on those same metrics for a configuration whose α was chosen to maximize performance on them. The reported optimum is therefore a fitted input to the evaluation rather than an independent prediction; the advantage is not forced across all α (Table 9 shows MGAA-like L-PCA beats PCA even at α=0.10), so this is partial, not total, circularity.

full rationale

No load-bearing self-citation or self-definitional reduction is present: reference [25] is a baseline (AWSVD), not a uniqueness theorem, and the method does not rename a known result. The sublayer cosine-similarity rule (Eq. 12-14) is an openly stated heuristic, and the energy-balance rule (Eq. 19) is an empirical design constraint supported by Table 1's Fixed Energy experiment; imposing equal energy retention is a construction choice, not a disguised prediction of the final perplexity. The main circularity-adjacent issue is the selection of α=0.35 on the test benchmarks, which conditions the reported gains on a hyperparameter fitted to the evaluation suite. Because the components are also ablated (L-PCA, E-PCA), applied across several backbones and compression ratios, and combined plug-and-play with other low-rank methods, the central claim retains substantial independent empirical content; hence the score is moderate rather than high.

Assumptions & free parameters 4 free parameters · 6 assumptions · 0 invented entities

The method rests on two empirical heuristics (cosine-similarity importance and energy-balanced retention), one tuned hyperparameter alpha, two unspecified optimization hyperparameters (epsilon and the lower rank bound), and a calibration-data representativeness assumption. No invented entities are introduced. The central claim is grounded in experiments, but the key relationships between the heuristics and performance are asserted rather than derived.

free parameters (4)
  • alpha (scaling coefficient) = 0.35
    Chosen from Table 9 on LLaMA1 at 50% compression; not derived and no validation split is described.
  • epsilon (energy balance threshold) = unspecified
    Appears in Eq. (19) as the allowed spread of retained energy ratios; no value or search method is given.
  • Lower bound on parameter retention ratio = unspecified
    Mentioned in Section 3.3 as a safeguard but never quantified; affects the feasible set of Eq. (19).
  • Calibration data volume = 128 sequences x 512 tokens for LLMs; 256 sequences x 3000 for LLaVA
    Selected by the authors; Tables 7 and 8 show results depend on this choice, so it is a free input to the method.
assumptions (6)
  • ad hoc to paper Lower cosine similarity between a sublayer's input and output activation implies higher sublayer importance and warrants a lower compression ratio.
    Introduced in Section 3.2 around Eqs. (12)-(14) with no supporting evidence beyond a heuristic argument; the central allocation rule rests on it.
  • domain assumption Maintaining equal retained energy ratios across weight matrices within a sublayer preserves model performance.
    Inferred from Table 1 (Fixed Energy rows) in Section 3.3; used to justify the constraint in Eq. (19). No error bars or statistical test support the inference.
  • domain assumption Calibration activations (128 sequences of 512 tokens from WikiText2 and Alpaca) are representative enough to estimate sublayer importance and matrix energy distributions for unseen test sets.
    Used throughout Section 3 and evaluated only via ablations in Tables 7-8; the paper itself shows single-domain calibration overfits.
  • standard math The compression loss of PCA low-rank approximation equals the sum of truncated eigenvalues of the output autocorrelation matrix.
    Eq. (11) follows from linear algebra; used to tie energy retention to compression loss.
  • domain assumption Eq. (19) is a solvable convex optimization with a known algorithm.
    The paper states it is a typical convex optimization problem in Section 3.3 but does not provide the solver, and the discrete rank variables and balance constraint make convexity questionable.
  • standard math The rank formula r = din * dout * (1-p) / (din + dout) from Eq. (5) is used without stating its derivation.
    Standard formula equating parameter reduction under low-rank factorization; treated as given.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs." pith.science (2026). https://pith.science/paper/6GYT74RT

@misc{pith2026250703294,
  author       = {Pith},
  title        = {Pith review of: MGAA: Multi-Granular Adaptive Allocation fof Low-Rank Compression of LLMs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/6GYT74RT}},
  note         = {Machine review of arXiv:2507.03294}
}
read the original abstract

The enormous parameter scale of large language models (LLMs) has made model compression a research hotspot, which aims to alleviate computational resource demands during deployment and inference. As a promising direction, low-rank approximation technique has made remarkable achievements. Nevertheless, unfortunately, the vast majority of studies to low-rank approximation compression generally apply uniform compression ratios across all weight matrices, while disregarding their inherently differentiated impacts on the model's performance. Although a few recent work attempts to employ heuristic search strategies to achieve the optimal parameter allocation, such strategies are computationally inefficient and lose the generalization ability in the era of LLMs. In this study, we propose a novel parameter Multi-Granular Adaptive Allocation (MGAA) method, which can adaptively allocate parameters between and within sublayers without task-specific evaluations in the compression process. MGAA consists of two components: 1) Among different sublayers, it assigns compression ratios based on their cosine similarity between inputs and outputs, allowing for a more tailored compression in sublayers with varying degrees of importance, and 2) Within each sublayer, it allocates different compression ratios to weight matrices based on their energy distribution characteristics, ensuring a consistent energy retention ratio while optimizing compression efficiency. Comprehensive evaluations of MGAA across multiple LLMs backbone models and benchmark datasets demonstrate its superior performance. Additionally, we apply our MGAA to multimodal model LLaVA, exhibiting remarkable performance improvements.

Figures

Figures reproduced from arXiv: 2507.03294 by the authors.

Figure 1
Figure 1. An overview of multi-granular adaptive allocation.The left figure illustrates the allocation of compression ratios to [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. The cosine similarity of different sublayers in the [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗
Figure 3
Figure 3. The influence of parameters α and pt on the normal￾ized output p. normalization to standardize the distribution of cosine sim￾ilarities across sublayers: z = I − E[I] p Var[I] . (13) 2) A scaling coefficient α is introduced to modulate the sen￾sitivity of compression ratio. This hyperparameter governs the magnitude of sublayer compression variations, where larger values of α induce more pronounced compression ratio … view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: The cumulative energy vectors of the four weight matrices in the MHA sublayer are shown in the left and the [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: At an overall compression of 50%, the sublayer [PITH_FULL_IMAGE:figures/full_fig_p011_5.png]
Figure 6
Figure 6. Figure 6: Under the 50% compression ratio, the LLaMA1 model [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

53 extracted references · 41 canonical work pages

  1. [1]

    Llama: Open and efficient foundation language models,

    H. Touvron, T. Lavril, G. Izacard, X. Martinet, M.-A. Lachaux, T. Lacroix, B. Rozi`ere, N. Goyal, E. Hambro, F. Azharet al., “Llama: Open and efficient foundation language models,” arXiv preprint arXiv:2302.13971, 2023

    arXiv 2023

  2. [2]

    Opt: Open pre-trained transformer language models,

    S. Zhang, S. Roller, N. Goyal, M. Artetxe, M. Chen, S. Chen, C. Dewan, M. Diab, X. Li, X. V . Lin, T. Mihaylov, M. Ott, S. Shleifer, K. Shuster, D. Simig, P . S. Koura, A. Sridhar, T. Wang, and L. Zettle- moyer, “Opt: Open pre-trained transformer language models,” arXiv preprint arXiv:2205.01068, 2022

    arXiv 2022

  3. [3]

    Mistral 7b,

    A. Q. Jiang, A. Sablayrolles, A. Mensch, C. Bamford, D. S. Chaplot, D. de las Casas, F. Bressand, G. Lengyel, G. Lample, L. Saulnier, L. R. Lavaud, M.-A. Lachaux, P . Stock, T. L. Scao, T. Lavril, T. Wang, T. Lacroix, and W. E. Sayed, “Mistral 7b,” arXiv preprint arXiv:2310.06825, 2023

    arXiv

  4. [4]

    Chain-of-thought prompting elicits reasoning in large language models,

    J. Wei, X. Wang, D. Schuurmans, M. Bosma, F. Xia, E. Chi, Q. V . Le, D. Zhou et al., “Chain-of-thought prompting elicits reasoning in large language models,” in Proc. NeurIPS, 2022

    work page 2022

  5. [5]

    Gptq: Ac- curate post-training quantization for generative pre-trained trans- formers,

    E. Frantar, S. Ashkboos, T. Hoefler, and D. Alistarh, “Gptq: Ac- curate post-training quantization for generative pre-trained trans- formers,” in Proc. ICLR, 2023

    work page 2023

  6. [6]

    Awq: Activation-aware weight quantization for llm compression and acceleration,

    J. Lin, J. Tang, H. Tang, S. Yang, W.-M. Chen, W.-C. Wang, G. Xiao, X. Dang, C. Gan, and S. Han, “Awq: Activation-aware weight quantization for llm compression and acceleration,” in Proc. ML- Sys, 2024

    work page 2024

  7. [7]

    SmoothQuant: Accurate and efficient post-training quantization for large language models,

    G. Xiao, J. Lin, M. Seznec, H. Wu, J. Demouth, and S. Han, “SmoothQuant: Accurate and efficient post-training quantization for large language models,” in Proc. ICML, 2023

    work page 2023

  8. [8]

    Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,

    T. Dettmers, M. Lewis, Y. Belkada, and L. Zettlemoyer, “Gpt3.int8(): 8-bit matrix multiplication for transformers at scale,” in Proc. NeurIPS, 2022

    work page 2022

Show all 53 references
  1. [9]

    Optimization-based post-training quantization with bit-split and stitching,

    P . Wang, W. Chen, X. He, Q. Chen, Q. Liu, and J. Cheng, “Optimization-based post-training quantization with bit-split and stitching,” IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  2. [10]

    Llm-pruner: On the structural pruning of large language models,

    X. Ma, G. Fang, and X. Wang, “Llm-pruner: On the structural pruning of large language models,” in Proc. NeurIPS, 2023

    2023

  3. [11]

    Fluctuation-based adaptive structured pruning for large language models,

    Y. An, X. Zhao, T. Yu, M. Tang, and J. Wang, “Fluctuation-based adaptive structured pruning for large language models,” in Proc. AAAI, 2024

    2024

  4. [12]

    Slicegpt: Compress large language models by delet- ing rows and columns,

    S. Ashkboos, M. L. Croci, M. G. d. Nascimento, T. Hoefler, and J. Hensman, “Slicegpt: Compress large language models by delet- ing rows and columns,” in Proc. ICLR, 2024

    2024

  5. [13]

    Influence function based second-order channel pruning: Evaluating true loss changes for pruning is possible without retraining,

    H. Cheng, M. Zhang, and J. Q. Shi, “Influence function based second-order channel pruning: Evaluating true loss changes for pruning is possible without retraining,” IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  6. [14]

    Performance-aware approximation of global channel pruning for multitask cnns,

    H. Ye, B. Zhang, T. Chen, J. Fan, and B. Wang, “Performance-aware approximation of global channel pruning for multitask cnns,” IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  7. [15]

    Can language models teach? teacher explanations improve student performance via personal- ization,

    S. Saha, P . Hase, and M. Bansal, “Can language models teach? teacher explanations improve student performance via personal- ization,” in Proc. NeurIPS, 2023

    2023

  8. [16]

    Scott: Self- consistent chain-of-thought distillation,

    P . Wang, Z. Wang, Z. Li, Y. Gao, B. Yin, and X. Ren, “Scott: Self- consistent chain-of-thought distillation,” in Proc. ACL, 2023

    2023

  9. [17]

    Unpacking the gap box against data-free knowledge distillation,

    Y. Wang, B. Qian, H. Liu, Y. Rui, and M. Wang, “Unpacking the gap box against data-free knowledge distillation,” IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  10. [18]

    Compressing transformers: Features are low- rank, but weights are not!

    H. Yu and J. Wu, “Compressing transformers: Features are low- rank, but weights are not!” in Proc. AAAI, vol. 37, 2023

    2023

  11. [19]

    Asvd: Activation-aware singular value decomposition for compressing large language models,

    Z. Yuan, Y. Shang, Y. Song, Q. Wu, Y. Yan, and G. Sun, “Asvd: Activation-aware singular value decomposition for compressing large language models,” arXiv preprint arXiv:2312.05821, 2024

    2024 arXiv

  12. [20]

    Svd-llm: Truncation- aware singular value decomposition for large language model compression,

    X. Wang, Y. Zheng, Z. Wan, and M. Zhang, “Svd-llm: Truncation- aware singular value decomposition for large language model compression,” arXiv preprint arXiv:2403.07378, 2024

    2024 arXiv

  13. [21]

    Deeptensor: Low-rank tensor decomposition with deep network priors,

    V . Saragadam, R. Balestriero, A. Veeraraghavan, and R. G. Bara- niuk, “Deeptensor: Low-rank tensor decomposition with deep network priors,” IEEE Trans. Pattern Anal. Mach. Intell., 2024

    2024

  14. [22]

    Compressing pre-trained language models by matrix decomposition

    M. Noach and Y. Goldberg, “Compressing pre-trained language models by matrix decomposition.” in Proc. AACL, 2020

    2020

  15. [23]

    Drone: Data-aware low-rank compression for large nlp models,

    P . H. Chen, H.-F. Yu, I. S. Dhillon, and C.-J. Hsieh, “Drone: Data-aware low-rank compression for large nlp models,” in Proc. NeurIPS, 2021

    2021

  16. [24]

    Lan- guage model compression with weighted low-rank factorization,

    Y.-C. Hsu, T. Hua, S. Chang, Q. Lou, Y. Shen, and H. Jin, “Lan- guage model compression with weighted low-rank factorization,” in Proc. ICLR, 2022. 14

    2022

  17. [25]

    Lorap: Transformer sub-layers deserve differentiated structured compression for large language models,

    G. Li, Y. Tang, and W. Zhang, “Lorap: Transformer sub-layers deserve differentiated structured compression for large language models,” in Proc. ICML, 2024

    2024

  18. [26]

    Lord: Low rank decompo- sition of monolingual code llms for one-shot compression,

    A. Kaushal, T. Vaidhya, and I. Rish, “Lord: Low rank decompo- sition of monolingual code llms for one-shot compression,” arXiv preprint arXiv:2309.14021, 2023

    2023 arXiv

  19. [27]

    Shortgpt: Layers in large language models are more redundant than you expect,

    X. Men, M. Xu, Q. Zhang, B. Wang, H. Lin, Y. Lu, X. Han, and W. Chen, “Shortgpt: Layers in large language models are more redundant than you expect,” arXiv preprint arXiv:2403.03853, 2024

    2024 arXiv

  20. [28]

    The optimal bert surgeon: Scalable and accurate second-order pruning for large language models,

    E. Kurtic, D. Campos, T. Nguyen, E. Frantar, M. Kurtz, B. Fineran, M. Goin, and D. Alistarh, “The optimal bert surgeon: Scalable and accurate second-order pruning for large language models,” in Proc. EMNLP, 2022

    2022

  21. [29]

    Sparsegpt: Massive language models can be accurately pruned in one-shot,

    E. Frantar and D. Alistarh, “Sparsegpt: Massive language models can be accurately pruned in one-shot,” in Proc. ICML, 2023

    2023

  22. [30]

    Pointer sentinel mixture models,

    S. Merity, C. Xiong, J. Bradbury, and R. Socher, “Pointer sentinel mixture models,” arXiv preprint arXiv:1609.07843, 2016

    2016 arXiv

  23. [31]

    Stanford alpaca: An instruction- following llama model,

    R. Taori, I. Gulrajani, T. Zhang, Y. Dubois, X. Li, C. Guestrin, P . Liang, and T. B. Hashimoto, “Stanford alpaca: An instruction- following llama model,” 2023

    2023

  24. [32]

    Boolq: Exploring the surprising difficulty of natu- ral yes/no questions,

    C. Clark, K. Lee, M.-W. Chang, T. Kwiatkowski, M. Collins, and K. Toutanova, “Boolq: Exploring the surprising difficulty of natu- ral yes/no questions,” in Proc. NAACL, 2019

    2019

  25. [33]

    Piqa: Reasoning about physical commonsense in natural language,

    Y. Bisk, R. Zellers, R. Le bras, J. Gao, and Y. Choi, “Piqa: Reasoning about physical commonsense in natural language,” in Proc. AAAI, 2020, p. 7432–7439

    2020

  26. [34]

    Hel- laswag: Can a machine really finish your sentence?

    R. Zellers, A. Holtzman, Y. Bisk, A. Farhadi, and Y. Choi, “Hel- laswag: Can a machine really finish your sentence?” in Proc. ACL, 2019

    2019

  27. [35]

    Wino- grande: An adversarial winograd schema challenge at scale,

    K. Sakaguchi, R. Le Bras, C. Bhagavatula, and Y. Choi, “Wino- grande: An adversarial winograd schema challenge at scale,” in Proc. AAAI, 2020, p. 8732–8740

    2020

  28. [36]

    Think you have solved question answering? try arc, the ai2 reasoning challenge,

    P . Clark, I. Cowhey, O. Etzioni, T. Khot, A. Sabharwal, C. Schoenick, and O. Tafjord, “Think you have solved question answering? try arc, the ai2 reasoning challenge,” arXiv preprint arXiv:1803.05457, 2018

    2018 arXiv

  29. [37]

    Can a suit of armor conduct electricity? a new dataset for open book question answering,

    T. Mihaylov, P . Clark, T. Khot, and A. Sabharwal, “Can a suit of armor conduct electricity? a new dataset for open book question answering,” in Proc. EMNLP, 2018

    2018

  30. [38]

    A framework for few-shot language model evaluation,

    L. Sutawika, L. Gao, H. Schoelkopf, S. Biderman, J. Tow, B. Abbasi, ben fattori, C. Lovering, farzanehnakhaee70, J. Phang, A. Thite, Fazz, Aflah, N. Muennighoff, T. Wang, sdtblck, nopperl, gakada, tttyuntian, researcher2, Chris, J. Etxaniz, Z. Kasner, Khalid, J. Hsu, AndyZwei,...

    2023

  31. [39]

    Data-free weight compress and denoise for large language mod- els,

    R. Peng, Y. Zhou, Q. Guo, Y. Gao, H. Yan, X. Qiu, and D. Lin, “Data-free weight compress and denoise for large language mod- els,” arXiv preprint arXiv:2402.16319, 2024

    2024 arXiv

  32. [40]

    Exploring the limits of transfer learning with a unified text-to-text transformer,

    C. Raffel, N. Shazeer, A. Roberts, K. Lee, S. Narang, M. Matena, Y. Zhou, W. Li, and P . J. Liu, “Exploring the limits of transfer learning with a unified text-to-text transformer,”Journal of Machine Learning Research, vol. 21, pp. 1–67, 2020

    2020

  33. [41]

    Building a large annotated corpus of english: the penn treebank,

    M. Marcus, “Building a large annotated corpus of english: the penn treebank,” Computational Linguistics, 1993

    1993

  34. [42]

    Improved baselines with visual instruction tuning,

    H. Liu, C. Li, Y. Li, and Y. J. Lee, “Improved baselines with visual instruction tuning,” in Proc. CVPR, 2024, pp. 26 296–26 306

    2024

  35. [43]

    Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality,

    W.-L. Chiang, Z. Li, Z. Lin, Y. Sheng, Z. Wu, H. Zhang, L. Zheng, S. Zhuang, Y. Zhuang, J. E. Gonzalez et al. , “Vicuna: An open- source chatbot impressing gpt-4 with 90%* chatgpt quality,” See https://vicuna. lmsys. org (accessed 14 April 2023) , 2023

    2023

  36. [44]

    Mak- ing the v in vqa matter: Elevating the role of image understanding in visual question answering,

    Y. Goyal, T. Khot, D. Summers-Stay, D. Batra, and D. Parikh, “Mak- ing the v in vqa matter: Elevating the role of image understanding in visual question answering,” in Proc. CVPR, 2017, pp. 6904–6913

    2017

  37. [45]

    Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,

    X. Yue, Y. Ni, K. Zhang, T. Zheng, R. Liu, G. Zhang, S. Stevens, D. Jiang, W. Ren, Y. Sunet al., “Mmmu: A massive multi-discipline multimodal understanding and reasoning benchmark for expert agi,” in Proc. CVPR, 2024, pp. 9556–9567

    2024

  38. [46]

    Learn to explain: Multimodal reason- ing via thought chains for science question answering,

    P . Lu, S. Mishra, T. Xia, L. Qiu, K.-W. Chang, S.-C. Zhu, O. Tafjord, P . Clark, and A. Kalyan, “Learn to explain: Multimodal reason- ing via thought chains for science question answering,” in Proc. NeurIPS, vol. 35, 2022, pp. 2507–2521

    2022

  39. [47]

    Mme: A com- prehensive evaluation benchmark for multimodal large language models,

    C. Fu, P . Chen, Y. Shen, Y. Qin, M. Zhang, X. Lin, Z. Qiu, W. Lin, J. Yang, X. Zheng, K. Li, X. Sun, R. Ji, and T. Lab, “Mme: A com- prehensive evaluation benchmark for multimodal large language models,” arXiv preprint arXiv:2306.13394v4, 2024

    2024 arXiv

  40. [48]

    Chartqa: A benchmark for question answering about charts with visual and logical reasoning,

    A. Masry, D. X. Long, J. Q. Tan, S. Joty, and E. Hoque, “Chartqa: A benchmark for question answering about charts with visual and logical reasoning,” in Proc. ACL, 2022

    2022

  41. [49]

    Infographicvqa,

    M. Mathew, V . Bagal, R. Tito, D. Karatzas, E. Valveny, and C. Jawa- har, “Infographicvqa,” in Proc. WACV, 2021

    2021

  42. [50]

    Ai2d-rst: a multimodal corpus of 1000 primary school science diagrams,

    T. Hiippala, M. Alikhani, J. Haverinen, T. Kalliokoski, E. Lo- gacheva, S. Orekhova, A. Tuomainen, M. Stone, and J. A. Bateman, “Ai2d-rst: a multimodal corpus of 1000 primary school science diagrams,” Language Resources and Evaluation , vol. 55, pp. 661 – 688, 2019

    2019

  43. [51]

    Vizwiz grand challenge: Answering visual questions from blind people,

    D. Gurari, Q. Li, A. J. Stangl, A. Guo, C. Lin, K. Grauman, J. Luo, and J. P . Bigham, “Vizwiz grand challenge: Answering visual questions from blind people,” in Proc. CVPR, 2018, pp. 3608–3617

    2018

  44. [52]

    Towards vqa models that can read,

    A. Singh, V . Natarjan, M. Shah, Y. Jiang, X. Chen, D. Batra, D. Parikh, and M. Rohrbach, “Towards vqa models that can read,” in Proc. CVPR, 2019, pp. 8317–8326

    2019

  45. [53]

    Lmms-eval: Accelerating the devel- opment of large multimoal models,

    B. Li, P . Zhang, K. Zhang, F. Pu, X. Du, Y. Dong, H. Liu, Y. Zhang, G. Zhang, C. Li, and Z. Liu, “Lmms-eval: Accelerating the devel- opment of large multimoal models,” March 2024

    2024

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.