Pith. sign in

REVIEW 3 major objections 4 minor 50 references

Gradient-based Fine-Tuning through Pre-trained Model Regularization

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GRFT updates only whole rows or columns of a weight matrix chosen by squared-gradient sums, and the authors report it beats GPS, Adapter, and LoRA while updating as little as 0.30% of parameters on VTAB.

desk verdict A modest PEFT improvement with a broken proof as written; worth refereeing for the empirical core, but the theory needs a rewrite. read the letter →

arxiv 2507.00016 v1 pith:VW3EH4XK submitted 2025-06-14 cs.LG cs.AIcs.CV

Xuanbo Liu , Liu Liu , Fuxiang Wu , Fusheng Hao , Xianglong Liu This is my paper · ORCID
classification cs.LGcs.AIcs.CV MSC 68T07
keywords parameter-efficientfine-tuninggradient-basedparameterselectionrow-andcolumn-wisemaskssquared-gradientrankingL2regularizationknowledgetransferViTlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper proposes GRFT, a way to fine-tune large pre-trained models by updating only selected whole rows or columns of each weight matrix, rather than scattered individual weights. The selection rule is to keep the rows or columns with the largest sums of squared gradients, computed once from the pre-trained model under a supervised contrastive loss. The authors argue, through a first-order Taylor approximation of the loss decrease, that these rows or columns give the largest reduction in loss per update, and they add an L2 penalty that pulls the selected weights back toward their pre-trained values to preserve learned knowledge. The reported empirical result is that GRFT beats the sparse-selection GPS baseline and standard PEFT methods such as Adapter and LoRA, while updating only 1.22% of parameters on FGVC and 0.30% on VTAB, and 0.08% on LLaMA3-1B text tasks.

What carries the argument

The central object is the binary row/column mask $M$ built from the gradient matrix $H = \nabla L_{\mathrm{scl}}(W_{\mathrm{pre}})$. Each row score is $S_i = \sum_{j=1}^n h_{ij}^2$ (or the analogous column score), and the mask keeps the $k$ rows or columns with the largest scores. The argument-carrying identity is the Taylor approximation $\Delta L \approx -\alpha_t \|\nabla L \odot M\|^2$, which says the loss decrease is governed by the squared gradient mass retained by the mask; maximizing that mass is why large-gradient rows or columns are chosen. The second machinery piece is the regularized objective $L_R = L_{\mathrm{cross}} + \lambda \sum_{l \in R} \|W^l - W^l_{\mathrm{pre}}\|^2$, which anchors selected parameters near their pre-trained values.

What would settle it

Midway through fine-tuning, recompute the per-row squared-gradient ranking on the same data and compare it with the frozen mask's selected rows; if the agreement is low and switching to the currently top-ranked rows improves validation accuracy, the fixed-mask assumption is false. A control run with masks recomputed every epoch would directly show whether the one-step Taylor-optimal choice stays optimal.

Watch

Extended reading notes

Core claim

GRFT's central claim is that, instead of scattering trainable weights across a matrix, one should update whole rows or columns of the pre-trained weight matrix, choosing the rows or columns whose gradient has the largest squared sum. These gradients are evaluated once, before fine-tuning, using a supervised contrastive loss on the pre-trained model, and a binary mask freezes all other weights. The paper derives this rule from a first-order Taylor expansion of the loss: minimizing the loss change after one gradient step is equivalent to maximizing the inner product of the gradient with its masked version, which favours parameters with the largest squared gradients; aggregating by row or column then selects the largest row or column sums. An L2 regularization term on the last layers, patch embedding, and classification head keeps the updated weights close to their pre-trained values to preserve transfer knowledge. The paper reports that this combination outperforms GPS, Adapter, LoRA, VPT, and SSF on FGVC and VTAB with ViT-B/16, updating 1.22% and 0.30% of parameters respectively, and beats LoRA on GLUE tasks with LLaMA3-1B while updating 0.08%.

Load-bearing premise

The mask is computed once from the pre-trained model's gradients before training and never changes, on the assumption that the rows or columns giving the largest one-step loss decrease at initialization remain the best parameters to update for the entire fine-tuning run.

Editorial extensions

If this is right

  • On the paper's results, a ViT-B/16 can be adapted to fine-grained classification tasks with only 1.22% of its parameters updated, improving on full fine-tuning's 89.44% average accuracy by reaching 91.33%.
  • Storing the mask becomes a list of row or column indices instead of a matrix the size of the weights, which removes the main storage drawback of GPS.
  • Because updates are dense along whole rows or columns, the method avoids sparse-matrix gradient computation and is easier to run on standard hardware.
  • Adding L2 regularization to the last layers, patch embedding, and classifier head improves accuracy over no norm and over L1 regularization on the FGVC benchmark.
  • The method transfers to decoder-only language models: on LLaMA3-1B it updates only 0.08% of parameters and outperforms LoRA on the RTE subset of GLUE.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • An untested extension is to recompute the row or column scores periodically during training instead of once at initialization; the paper's one-step argument only covers the first gradient update, so adaptive masks are a natural next test.
  • The same squared-gradient logic could be fed by a second-order quantity such as the diagonal Fisher information; if the underlying claim is that high-curvature directions matter, such a variant would be a sharper version of the same idea.
  • Because selection is per layer and independent, GRFT could be combined with per-layer budgets or with LoRA in selected layers, which the paper does not explore.
  • The data-subset trick of choosing the split with minimal loss for mask computation is only ablated for up to seven subsets; a natural further test is whether this selection generalizes to larger or noisier datasets.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper proposes GRFT, a parameter-efficient fine-tuning method that selects entire rows or columns of a pre-trained model's weight matrices based on the sum of squared gradients computed under a supervised contrastive loss, freezes the remaining parameters with a mask, and adds an L2 regularization term pulling updated weights toward the pre-trained initialization. The authors claim a theoretical optimality for selecting rows or columns with the largest squared-gradient sums, and they report accuracy improvements over GPS, LoRA, adapters, and other baselines on FGVC, VTAB-1k, and three GLUE tasks, with substantially reduced parameter counts.

Significance. The empirical direction is useful: structured (row/column) selection is storage- and hardware-friendlier than sparse per-neuron selection, and the ablations in Table 4 give some evidence that row/column selection can match or slightly exceed sparse selection in their setup. The L2 regularization toward pre-trained weights is a sensible and simple mechanism, and the hyperparameter tables are a helpful contribution for reproducibility. However, the paper's central theoretical claim is not established as written: the optimization in Eq. (3.1.4) uses a per-column sparsity constraint that does not imply the row-sum selection rule of Eqs. (3.1.5)-(3.1.6), and the one-step Taylor justification does not cover a mask fixed for 100 training epochs under Adam. The empirical results are therefore currently supported only as a heuristic, and the 'theoretically demonstrate' claim would need to be substantially revised or re-proven.

major comments (3)
  1. [Sec. 3.1, Eq. (3.1.4)] The optimization problem in Eq. (3.1.4) constrains each column to have at most k nonzeros, i.e., ||M_{.,j}||_0 <= k for every column j. Under this constraint the optimal mask selects, independently in each column, the k entries with the largest squared gradients, which is exactly the GPS-style sparse selection pattern, not whole-row selection. The algorithm's row-sum rule in Eqs. (3.1.5)-(3.1.6) corresponds to a different constraint, namely that all selected entries of a row are either fully masked or fully unmasked. A concrete counterexample exposes the gap: for G = [[100,1],[1,100]] and k=1, the per-column optimum of Eq. (3.1.4) leaves squared error 2, while the top-row mask of Eqs. (3.1.5)-(3.1.6) leaves squared error 10001. Thus the row-selection rule is not the optimum of the stated objective, and the claimed theoretical optimality is not proven as written.
  2. [Sec. 3.1, Eqs. (3.1.1)-(3.1.3) and Algorithm 1] The Taylor-expansion argument in Eqs. (3.1.1)-(3.1.3) justifies a mask choice for a single gradient descent step from the initial pre-trained parameters. Algorithm 1, however, computes the mask once at initialization using the SCL loss and then fixes it for the entire training run (up to 100 epochs) while optimizing the cross-entropy plus L2 objective with Adam. The paper does not show that the parameters with the largest initial squared gradients remain the most important throughout training, nor that a mask optimal for one step is optimal for the full trajectory. This is a load-bearing assumption because the entire method depends on the frozen mask. An experiment comparing the fixed mask with periodically recomputed masks, or at least with masks computed from mid-training gradients, would be needed to support the claim.
  3. [Sec. 3.1, Eq. (3.1.7) vs Algorithm 1] Equation (3.1.7) writes the update as plain masked gradient descent, W_{t+1} - W_t = -eta_t grad L(W_t) \odot M, but Algorithm 1 uses Adam on the masked gradient, with the coordinate-wise normalization in lines 11-15. Adam's per-parameter scaling changes which coordinates receive effective updates, so the one-step Taylor optimality of a mask under plain gradient descent does not directly transfer to the implemented optimizer. The paper should either provide an analysis for the actual update rule or explicitly frame the theoretical result as a heuristic motivation for the empirical method.
minor comments (4)
  1. [Sec. 4.2] The dataset descriptions contain errors: NABirds is described as 'a dataset contains 102 flower species with 8,189 images,' which is a description of Oxford Flowers, not NABirds; please correct this and double-check the other dataset statistics.
  2. [Sec. 4.1] The text says each downstream task is trained for '100 iterations' while Table 6 and Algorithm 1 indicate 100 epochs; this wording should be harmonized to avoid confusion.
  3. [Tables 1 and 2] Several entries in Tables 1 and 2 are missing spaces or are visually concatenated (e.g., '38.879.7' in Table 2), making the tables hard to read; formatting should be cleaned.
  4. [Sec. 4.4.3] The data-subsets procedure selects the mask from the subset with minimal loss, but the paper does not compare this against a random subset or the full dataset; the ablation in Fig. 2(d) shows sensitivity but not that the minimal-loss criterion is the cause of improvement. Please add such a comparison or soften the claim.

Circularity Check

1 steps flagged · score 6.0 of 10

GRFT's central theorem—rows/columns with the highest sum of squared gradients are optimal—is the definitional argmin of Eq. (3.1.4) only under an unstated row-support constraint; under the per-column constraint actually written there, the objective's optimum is GPS-style per-column selection, which contradicts the row rule.

  1. self definitional [Section 3.1 'The Computation of Gradient Mask', Eqs. (3.1.4)-(3.1.6); Abstract claim; Algorithm 1 lines 2-5.]
    "We theoretically demonstrate that the rows or columns with the highest sum of squared gradients are optimal for updating. ... Therefore, our proposed approach is to select the entire row or column, which is theoretically justified to select the rows with the largest squared sums of the gradients. ... minM ∥∇Lscl(Wpre) − ∇Lscl(Wpre) ⊙ M∥2 2 (3.1.4) s.t.∥M,j∥0 ≤ k, ∀j ∈ [n]."

    Eq. (3.1.4) constrains each column to at most k nonzeros, whose one-step optimum is GPS-style top-k entries per column, not whole rows. For G=[[100,1],[1,100]], k=1, the feasible optimum is the diagonal mask (squared error 2), while the top-row mask leaves error 10001, so the stated objective cannot justify row selection. The only constraint under which selecting top rows by S_i=Σ_j h_ij^2 (Eqs. 3.1.5–3.1.6) is the exact argmin is an unstated row-support mask M_ij=1{i∈T}, |T|=k; there ||G−G⊙M||²=Σ_{i∉T} S_i, so the claimed 'theoretical demonstration' is an algebraic identity.

full rationale

The paper's empirical core is self-contained: GRFT is benchmarked against GPS*, LoRA, Adapter, VPT, SSF, SPT and full fine-tuning on FGVC, VTAB-1k and GLUE (Tables 1–3), and those comparisons are independent of the optimality theorem in Section 3.1. There is no load-bearing self-citation: GPS [49] and the SCL loss [22] are external prior work, and no uniqueness or optimality result is imported from the present authors' earlier papers. The circularity found is confined to the theory section, where the abstract's stronger claim—rows or columns with the highest sum of squared gradients are optimal—reduces by construction: for a row-support mask M_ij=1{i∈T}, the objective of Eq. (3.1.4) equals a constant minus Σ_{i∈T} S_i, so the top-row-sum rule (Eqs. 3.1.5–3.1.6) is the definitional argmin of that objective, and the row-support shape itself is an adopted ansatz rather than a derived conclusion. Moreover, the constraint actually written in Eq. (3.1.4) (at most k nonzeros per column) does not encode row support, and its true one-step optimum is per-column (GPS-style) selection, with an explicit 2x2 counterexample showing the row rule is not optimal under the stated constraint. Per-dataset hyperparameters (rows/columns k, λ, data-subset count) are chosen to maximize validation accuracy (Appendix A.3), standard tuning rather than fitted-input-called-prediction, so no additional circularity arises there. Overall: partial circularity in the central theoretical claim, with the empirical evaluation remaining independent, hence score 6.

Assumptions & free parameters 4 free parameters · 4 assumptions · 0 invented entities

The central claim depends on several per-dataset hyperparameters (k, lambda, regular layer count, data subset count) and on the row-vs-column orientation choice, which is not reported per dataset. No entirely new entities are introduced: the mask is a standard binary tensor. The assumptions concerning the validity of one-step Taylor selection, the informativeness of initial SCL gradients, and the minimal-loss subset heuristic are load-bearing and untested.

free parameters (4)
  • Row/Column selection count k = 1 to 26 depending on dataset (Tables 6, 7)
    Number of rows or columns selected per layer; tuned per dataset on validation, directly controls the parameter budget shown in Table 8.
  • L2 regularization strength lambda = 0 to 1e-3 depending on dataset (Tables 6, 7)
    Regularization coefficient tuned per dataset (Fig. 2(b)); zero for some VTAB tasks.
  • Regular layer count = 1 to 'all' depending on dataset (Tables 6, 7)
    Number of layers receiving the L2 penalty toward pre-trained weights; tuned per dataset.
  • Data subsets number n = 1 to 7 depending on dataset (Tables 6, 7)
    Number of random training-data partitions used to pick the minimal-loss subset for mask computation; tuned per dataset (Fig. 2(d)).
assumptions (4)
  • domain assumption The one-step Taylor expansion of the loss (Eq. 3.1.1) with masked gradient descent is a valid proxy for final fine-tuning performance.
    The selection principle is derived from a single gradient step, but the mask is fixed for 100 epochs; the paper does not test whether one-step optimality transfers to the full training trajectory.
  • domain assumption Gradients of the SCL loss on the pre-trained model at initialization indicate which parameters matter for the downstream task.
    The mask is computed from gradients of the pre-trained model before training (Algorithm 1, line 1); no evidence is given that these gradients remain informative after many updates.
  • ad hoc to paper The training subset with minimal loss is 'higher quality' and yields better masks.
    Stated as a hypothesis in Section 4.4.3 and used to choose the data for mask computation; no validation or theory is provided for this heuristic.
  • domain assumption L2 regularization toward pre-trained weights improves knowledge transfer and generalization without significantly harming adaptation.
    Common regularization assumption; the ablation in Table 5 shows L2 helps slightly (91.25 vs 91.10 without norm), but the mechanism is not analyzed.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Gradient-based Fine-Tuning through Pre-trained Model Regularization." pith.science (2026). https://pith.science/paper/VW3EH4XK

@misc{pith2026250700016,
  author       = {Pith},
  title        = {Pith review of: Gradient-based Fine-Tuning through Pre-trained Model Regularization},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VW3EH4XK}},
  note         = {Machine review of arXiv:2507.00016}
}
read the original abstract

Large pre-trained models have demonstrated extensive applications across various fields. However, fine-tuning these models for specific downstream tasks demands significant computational resources and storage. One fine-tuning method, gradient-based parameter selection (GPS), focuses on fine-tuning only the parameters with high gradients in each neuron, thereby reducing the number of training parameters. Nevertheless, this approach increases computational resource requirements and storage demands. In this paper, we propose an efficient gradient-based and regularized fine-tuning method (GRFT) that updates the rows or columns of the weight matrix. We theoretically demonstrate that the rows or columns with the highest sum of squared gradients are optimal for updating. This strategy effectively reduces storage overhead and improves the efficiency of parameter selection. Additionally, we incorporate regularization to enhance knowledge transfer from the pre-trained model. GRFT achieves state-of-the-art performance, surpassing existing methods such as GPS, Adapter Tuning, and LoRA. Notably, GRFT requires updating only 1.22% and 0.30% of the total parameters on FGVC and VTAB datasets, respectively, demonstrating its high efficiency and effectiveness. The source code will be released soon.

Figures

Figures reproduced from arXiv: 2507.00016 by the authors.

Figure 1
Figure 1. The overall pipeline of our approach. This approach primarily consists of two steps. Step 1: Gradients Selection. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Impacts of different setting. From left to right: (a) Impacts of different numbers of selected rows on performance. (b) [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. The differences selection methods between GPS and GRFT. The illustration depicts the parameter selection when [PITH_FULL_IMAGE:figures/full_fig_p012_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

50 extracted references · 40 canonical work pages

  1. [1]

    Cheng, J

    G. Cheng, J. Han, and X. Lu. Remote sensing image scene classification: Benchmark and state of the art. Proceedings of the IEEE, 105(10):1865–1883, Oct. 2017

    work page 2017

  2. [2]

    J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, Jun 2009

    work page 2009

  3. [3]

    Dosovitskiy, L

    A. Dosovitskiy, L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Minderer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby. An image is worth 16x16 words: Transformers for image recognition at scale. arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition, Oct 2020

    work page 2020

  4. [4]

    Fei-Fei, R

    L. Fei-Fei, R. Fergus, and P. Perona. Learning generative visual models from few training examples: An incremental bayesian approach tested on 101 object categories. In 2004 Conference on Computer Vision and Pattern Recognition Workshop, pages 178–178, 2004

    work page 2004

  5. [5]

    Frankle and M

    J. Frankle and M. Carbin. The lottery ticket hypothesis: Finding sparse, trainable neural networks. arXiv: Learning,arXiv: Learning, Mar 2018

    work page 2018

  6. [6]

    R. M. French. Catastrophic forgetting in connectionist networks. Trends in Cognitive Sciences, 3(4):128–135, 1999

    work page 1999

  7. [7]

    T. Gale, E. Elsen, and S. Hooker. The state of sparsity in deep neural networks. arXiv: Learning,arXiv: Learning, Feb 2019

    work page 2019

  8. [8]

    Grattafiori and A

    A. Grattafiori and A. D. etc. The llama 3 herd of models, 2024

    work page 2024

Show all 50 references
  1. [9]

    S. Han, H. Mao, and W. Dally. Deep compression: Compressing deep neural networks with pruning, trained quantization and huffman coding. arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition, Oct 2015

    2015

  2. [10]

    S. Han, J. Pool, J. Tran, and W. Dally. Learning both weights and connections for efficient neural networks. arXiv: Neural and Evolutionary Computing,arXiv: Neural and Evolutionary Computing, Jun 2015

    2015

  3. [11]

    Hayou, N

    S. Hayou, N. Ghosh, and B. Yu. Lora+: Efficient low rank adaptation of large models. arXiv preprint arXiv:2402.12354, 2024

    2024 arXiv

  4. [12]

    H. He, J. Cai, J. Zhang, D. Tao, and B. Zhuang. Sensitivity-aware visual parameter-efficient fine-tuning, 2023

    2023

  5. [13]

    K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into rectifiers: Surpassing human-level performance on imagenet classification. In Proceedings of the IEEE international conference on computer vision, pages 1026–1034, 2015

    2015

  6. [14]

    A. E. Hoerl and R. W. Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 42:80 – 86, 2000

    2000

  7. [15]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrzebski, B. Morrone, Q. de Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter-efficient transfer learning for nlp, 2019

    2019

  8. [16]

    Houlsby, A

    N. Houlsby, A. Giurgiu, S. Jastrz˛ ebski, B. Morrone, Q. Laroussilhe, A. Gesmundo, M. Attariyan, and S. Gelly. Parameter- efficient transfer learning for nlp. International Conference on Machine Learning,International Conference on Machine Learning, Jan 2019. 9

    2019

  9. [17]

    E. J. Hu, Y . Shen, P. Wallis, Z. Allen-Zhu, Y . Li, S. Wang, L. Wang, and W. Chen. Lora: Low-rank adaptation of large language models. arXiv preprint arXiv:2106.09685, 2021

    2021 arXiv

  10. [18]

    M. Jia, L. Tang, B.-C. Chen, C. Cardie, S. Belongie, B. Hariharan, and S.-N. Lim. Visual prompt tuning. In European Conference on Computer Vision, pages 709–727. Springer, 2022

    2022

  11. [19]

    Johnson, B

    J. Johnson, B. Hariharan, L. van der Maaten, L. Fei-Fei, C. L. Zitnick, and R. Girshick. Clevr: A diagnostic dataset for compositional language and elementary visual reasoning. In CVPR, 2017

    2017

  12. [20]

    Karimi Mahabadi, J

    R. Karimi Mahabadi, J. Henderson, and S. Ruder. Compacter: Efficient low-rank hypercomplex adapter layers. Advances in Neural Information Processing Systems, 34:1022–1035, 2021

    2021

  13. [21]

    Khosla, N

    A. Khosla, N. Jayadevaprakash, B. Yao, and L. Fei-Fei. Novel dataset for fine-grained image categorization. In First Workshop on Fine-Grained Visual Categorization, IEEE Conference on Computer Vision and Pattern Recognition, Colorado Springs, CO, June 2011

    2011

  14. [22]

    Khosla, P

    P. Khosla, P. Teterwak, C. Wang, A. Sarna, Y . Tian, P. Isola, A. Maschinot, C. Liu, and D. Krishnan. Supervised contrastive learning. Neural Information Processing Systems,Neural Information Processing Systems, Apr 2020

    2020

  15. [23]

    Kingma and J

    D. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv: Learning,arXiv: Learning, Dec 2014

    2014

  16. [24]

    Krause, M

    J. Krause, M. Stark, J. Deng, and L. Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pages 554–561, 2013

    2013

  17. [25]

    Krizhevsky, G

    A. Krizhevsky, G. Hinton, et al. Learning multiple layers of features from tiny images. 2009

    2009

  18. [26]

    Kruschke and J

    J. Kruschke and J. Movellan. Benefits of gain: speeded learning and minimal hidden layers in back-propagation networks. IEEE Transactions on Systems, Man, and Cybernetics, 21(1):273–280, Jan 1991

    1991

  19. [27]

    Kurtic, E

    E. Kurtic, E. Frantar, and D. Alistarh. Ziplm: Inference-aware structured pruning of language models, 2023

    2023

  20. [28]

    H. Li, A. Kadav, I. Durdanovic, H. Samet, and H. Graf. Pruning filters for efficient convnets. arXiv: Computer Vision and Pattern Recognition,arXiv: Computer Vision and Pattern Recognition, Aug 2016

    2016

  21. [29]

    J. Li, J. Xu, S. Huang, Y . Chen, W. Li, J. Liu, Y . Lian, J. Pan, L. Ding, H. Zhou, et al. Large language model inference acceleration: A comprehensive hardware perspective. arXiv preprint arXiv:2410.04466, 2024

    2024 arXiv

  22. [30]

    Q. Li. Parameter efficient fine-tuning on selective parameters for transformer-based pre-trained models. In 2024 IEEE International Conference on Multimedia and Expo (ICME), pages 1–6, 2024

    2024

  23. [31]

    D. Lian, D. Zhou, J. Feng, and X. Wang. Scaling & shifting your features: A new baseline for efficient model tuning, 2023

    2023

  24. [32]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019

    2019

  25. [33]

    H. Lu, M. Ding, Y . Huo, G. Yang, Z. Lu, M. Tomizuka, and W. Zhan. Uniadapter: Unified parameter-efficient transfer learning for cross-modal modeling. arXiv preprint arXiv:2302.06605, 2023

    2023 arXiv

  26. [34]

    Nilsback and A

    M.-E. Nilsback and A. Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing, Dec 2008

    2008

  27. [35]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, and F. L. Aleman.etc. Gpt-4 technical report, 2024

    2024

  28. [36]

    Ouyang, X

    W. Ouyang, X. Wang, C. Zhang, and X. Yang. Factors in finetuning deep model for object detection with long-tail distribution. In 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Jun 2016

    2016

  29. [37]

    Qiang, R

    R. Qiang, R. Zhang, and P. Xie. Bilora: A bi-level optimization framework for overfitting-resilient low-rank adaptation of large pre-trained models. arXiv preprint arXiv:2403.13037, 2024

    2024 arXiv

  30. [38]

    Toneva, A

    M. Toneva, A. Sordoni, R. T. d. Combes, A. Trischler, Y . Bengio, and G. J. Gordon. An empirical study of example forgetting during deep neural network learning. arXiv preprint arXiv:1812.05159, 2018

    2018 arXiv

  31. [39]

    Touvron, L

    H. Touvron, L. Martin, K. Stone, P. Albert, A. Almahairi, Y . Babaei, N. Bashlykov, S. Batra, P. Bhargava, and S. Bhosale.etc. Llama 2: Open foundation and fine-tuned chat models, 2023

    2023

  32. [40]

    Van Horn, S

    G. Van Horn, S. Branson, R. Farrell, S. Haber, J. Barry, P. Ipeirotis, P. Perona, and S. Belongie. Building a bird recognition app and large scale dataset with citizen scientists: The fine print in fine-grained dataset collection. In Proceedings of the IEEE Conference on Compu...

    2015

  33. [41]

    B. S. Veeling, J. Linmans, J. Winkens, T. Cohen, and M. Welling. Rotation equivariant CNNs for digital pathology. June 2018

    2018

  34. [42]

    C. Wah, S. Branson, P. Welinder, P. Perona, and S. J. Belongie. The caltech-ucsd birds-200-2011 dataset. 2011

    2011

  35. [43]

    A. Wang, A. Singh, J. Michael, F. Hill, O. Levy, and S. R. Bowman. Glue: A multi-task benchmark and analysis platform for natural language understanding, 2019

    2019

  36. [44]

    H. Wang, J. Chang, X. Luo, J. Sun, Z. Lin, and Q. Tian. Lion: Implicit vision prompt tuning. arXiv preprint arXiv:2303.09992, 2023

    2023 arXiv

  37. [45]

    W. Wen, C. Wu, Y . Wang, Y . Chen, and H. Li. Learning structured sparsity in deep neural networks.Neural Information Processing Systems,Neural Information Processing Systems, Aug 2016

    2016

  38. [46]

    E. B. Zaken, S. Ravfogel, and Y . Goldberg. Bitfit: Simple parameter-efficient fine-tuning for transformer-based masked language-models, 2022

    2022

  39. [47]

    X. Zhai, J. Puigcerver, A. Kolesnikov, P. Ruyssen, C. Riquelme, M. Lucic, J. Djolonga, A. S. Pinto, M. Neumann, A. Dosovitskiy, L. Beyer, O. Bachem, M. Tschannen, M. Michalski, O. Bousquet, S. Gelly, and N. Houlsby. A large-scale study of representation learning with the visua...

    2020

  40. [48]

    Zhang, J

    R. Zhang, J. Han, C. Liu, P. Gao, A. Zhou, X. Hu, S. Yan, P. Lu, H. Li, and Y . Qiao. Llama-adapter: Efficient fine-tuning of language models with zero-init attention. arXiv preprint arXiv:2303.16199, 2023. 10

    2023 arXiv

  41. [49]

    Zhang, Q

    Z. Zhang, Q. Zhang, Z. Gao, R. Zhang, E. Shutova, S. Zhou, and S. Zhang. Gradient-based parameter selection for efficient fine-tuning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 28566–28577, 2024

    2024

  42. [50]

    intrinsic rank,

    K. Zhou, J. Yang, C. C. Loy, and Z. Liu. Learning to prompt for vision-language models. International Journal of Computer Vision, 130(9):2337–2348, 2022. A. Appendix A.1. Baseline Description A.1.1 GPS GPS (Gradient-based Parameter Selection) is an innovative Parameter-Efficie...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.