Pith. sign in

REVIEW 4 major objections 6 minor 55 references

ChebyMA treats each weight matrix as a smooth surface, proving a polynomial convergence bound and beating low-rank adapters on matched-budget tests.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review

2026-08-01 18:08 UTC pith:TXLZQHUT

load-bearing objection A novel parameterization for PEFT with impressive but narrow empirical results; the theoretical guarantee rests on an unverified smoothness assumption, so it's a solid major-revision candidate rather than a desk reject. the 4 major comments →

arxiv 2607.17377 v2 pith:TXLZQHUT submitted 2026-07-19 cs.LG

Chebyshev Manifold Adaptation

Jiawen Li This is my paper
classification cs.LG
keywords parameter-efficient fine-tuningChebyshev polynomialslow-rank adaptationfunction approximationweight parameterizationPareto frontexpressivity
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper proposes ChebyMA, a parameter-efficient fine-tuning method that represents a weight-update matrix as a sum of Chebyshev polynomial surfaces over learnable row and column coordinates. The paper's core claim is an approximation guarantee: any weight matrix that is a pointwise sample of a smooth function can be reconstructed by a single ChebyMA surface with Frobenius error shrinking like O(d^{-r}) in the polynomial degree d. Because the parameterization is nonlinear, ChebyMA is claimed to be strictly more expressive than LoRA and its variants, which the paper shows are constrained special cases. On CIFAR-10/100, SST-2, and AG News, ChebyMA gives a consistently better parameter-accuracy Pareto front than LoRA, TLoRA, and StelLA at matched budgets, with the biggest gains on CIFAR-100.

Core claim

ChebyMA parameterizes a weight update as W = sum_{k=1}^S T(u_k) A_k T(v_k)^T, where T maps learnable coordinate vectors to Chebyshev basis evaluations and A_k are trainable coefficient matrices. The paper proves that if the target ΔW* samples an r-times differentiable function at ideal coordinates, then a single surface of degree d admits parameters with reconstruction error ||ΔW* - Ŵ||_F ≤ √(mn)·O(d^{-r}), a polynomial convergence rate. It further argues via Kolmogorov n-width intuition that multiple surfaces (S>1) expand the hypothesis space and let different surfaces specialize on different structural components. Experiments show consistent Pareto dominance over LoRA, TLoRA, and StelLA ac

What carries the argument

The central object is the Chebyshev hypersurface construction: W = T(u) A T(v)^T, with T(x) the matrix of first d+1 Chebyshev polynomials evaluated at learnable coordinates u, v, and A a (d+1)x(d+1) coefficient matrix. Each entry is a bivariate polynomial in the two coordinates, so the weight matrix is a scalar field over row-column index space. Chebyshev bases are chosen for their minimax optimality, stable three-term recurrence, and orthogonality; multi-surface superposition yields effective rank up to S(d+1), and LoRA is recovered as a constrained degree-1 surface with three coefficients zeroed.

Load-bearing premise

The theorem assumes the true weight update is exactly a pointwise sample of an r-times differentiable function at ideal coordinates that the learnable coordinates can recover; if real updates are not smooth along index dimensions, the guarantee is vacuous.

What would settle it

Train ChebyMA on a weight matrix whose entries are drawn independently at random (or are adversarially non-smooth) and check whether the reconstruction error still decays like d^{-r}. The paper's own TF-IDF control, where the advantage disappeared when embeddings lost smooth structure, points to this as the decisive check.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the theorem holds, any sufficiently smooth target update can be approximated with error that drops polynomially in d, giving a principled capacity knob.
  • LoRA and three-factor variants are special cases of degree-1 ChebyMA, so ChebyMA at d≥1 dominates them in expressivity at equal or fewer parameters.
  • The parameter cost ratio ChebyMA/LoRA ≈ 1/(d+1) + (d+1)/(m+n) means d=2 gives roughly a 3x parameter reduction at matched effective rank.
  • At high parameter budgets the smoothness prior becomes restrictive, so the advantage is confined to low-to-mid budgets, as the experiments show.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If the smoothness prior holds for attention weight updates in transformer layers, applying ChebyMA inside attention (not just classification heads) could extend the Pareto gains to the dominant fine-tuning regime; the paper leaves this as future work.
  • A testable prediction: the advantage over LoRA should scale with the smoothness of the weight update, measurable by the decay of Chebyshev coefficients; layers with quickly decaying coefficients should benefit most.
  • The construction generalizes to higher-dimensional tensors (e.g., 4D convolution kernels), so the same surface idea may compress convolution filters more compactly than low-rank decompositions.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes ChebyMA, a parameter-efficient adaptation method that represents a weight-update matrix as a sum of S Chebyshev-polynomial surfaces, W = Σ_k T(u_k) A_k T(v_k)^T, where u_k and v_k are learnable row/column coordinates and A_k are trainable coefficient matrices. The main theoretical result, Theorem 1, is an existence guarantee: if the target ΔW* is exactly a pointwise sample of an r-times differentiable function F* at 'ideal' coordinates, then a single-degree-d ChebyMA surface has a parameter assignment achieving Frobenius error √(mn)·O(d^{-r}). The paper also gives an informal Kolmogorov n-width argument for multi-surface models, a proposition that degree-d parameterizations strictly contain lower-degree ones, and experiments on CIFAR-10/100, AG News, and SST-2 claiming consistently superior parameter-accuracy Pareto fronts over LoRA, TLoRA, and StelLA.

Significance. The central idea—treating weight matrices as continuous fields over learnable index coordinates and approximating them with Chebyshev bases—is interesting and potentially useful. Theorem 1 is correct as an existence result under its stated smoothness/coordinate premise, and the parameter-efficiency analysis is clean. The paper also ships a concrete, vectorized implementation and reports reproducibility details (seeds, optimizer, architecture). If the smoothness premise can be verified for real pretrained-feature adaptation, the contribution would be a meaningful new PEFT paradigm. However, the paper's strongest claim—that the experiments 'validate its solid theoretical foundation'—is not currently supported, because the premise of Theorem 1 is not tested and the offered control is confounded. The multi-manifold theory is also only heuristic, despite being the version used in the best experimental configurations.

major comments (4)
  1. [Theoretical Analysis, Theorem 1 (Steps 1 and 3); Abstract] Theorem 1 is conditional on two strong premises: (i) ΔW* is exactly F*(x*_i, y*_j) for an r-smooth F*, and (ii) the feasible parameter assignment may use the ideal coordinates (x*_i, y*_j). The paper never verifies premise (i) on actual learned weight updates, and gradient descent is not shown to find or approximate the assignment in (ii). Therefore the Abstract's statement that the experimental results 'validate its solid theoretical foundation' is not justified. The TF-IDF control in the Discussion is confounded: replacing DistilBERT embeddings with bag-of-words changes the input distribution, feature dimension, and task setup, so it cannot isolate index-smoothness. Please add a direct test—e.g., train a full linear head, then measure Chebyshev approximation error on natural vs. randomly permuted row/column orders—or substantially weaken the theory-to-experiment claims.
  2. [Method, Multi-Surface Superposition; Theoretical Analysis, Multi-Manifold Superposition] All best-performing configurations in Tables 2–4 use S>1, yet there is no formal statement or proof for the multi-surface model. The text offers only 'Kolmogorov n-width intuition,' which does not imply that the ChebyMA parameterization benefits from S surfaces; n-width is a statement about optimal linear/subspace approximation, not about this nonconvex parameterization. Please either supply a concrete approximation bound for sums of S Chebyshev surfaces even under simplifying assumptions, or explicitly label the S>1 discussion as a heuristic. As written, the claim that the paper 'demonstrate[s] the expressive advantages of multi-manifold superposition' is unsupported.
  3. [Method, Parameter Efficiency Analysis; Experiments, Table 2] The parameter-efficiency argument compares representations at equal effective rank, but the experiments compare at equal parameter count. At the ∼650-parameter operating point, ChebyMA (S=1, d=2) can realize rank up to 3 while LoRA r=1 has rank 1; similar rank mismatches occur at other budgets. The reported Pareto gains may therefore reflect higher rank capacity per parameter rather than Chebyshev smoothness. Please add a control that equates rank (e.g., LoRA with r=3 matched in parameter count, or a low-rank baseline with extra random factors) or analyze the learned ChebyMA surfaces to show the gains are due to the polynomial structure. Without this, the theoretical efficiency ratio 1/(d+1)+(d+1)/(m+n) is not the quantity measured in the experiments.
  4. [Experiments, Table 3] Table 3's 'matched parameter budgets' are not actually matched. In the ∼650-budget block, TLoRA is listed with 4576 parameters while ChebyMA has 649/656; in the ∼1300 block, TLoRA has 1024 vs. ChebyMA 1312. If the Params column counts total tensor elements rather than trainable parameters, the column should be relabeled and the comparison re-run at equal trainable budgets; if it counts trainable parameters, TLoRA's r=2 entry should be 4, not 4576. The claim that 'ChebyMA achieves the highest accuracy at every matched budget' is therefore not supported by the table as printed.
minor comments (6)
  1. [Abstract / Conclusion] The phrase 'validating its solid theoretical foundation' overstates what Theorem 1 supports; see Major Comment 1. Please rephrase to something like 'consistent with the smoothness hypothesis' or provide direct validation.
  2. [Figure 2 caption] The caption appears identical to that of Figure 1 ('Overview of different methods'), but Figure 2 is a Pareto plot. Please correct the caption.
  3. [Experiments, Table 4] The sentence 'consistent with the theoretical O(d^{-r}) convergence' is not a quantitative check; any monotone improvement is consistent. Please fit or report the empirical convergence rate, or remove the claim.
  4. [Experiments, Experimental Setup] Please specify the pretraining dataset for ResNet-18 and DistilBERT, clarify the exact architecture of the 'Standard' baseline, and state whether Table 3 reports total tensor elements or trainable parameters.
  5. [References] The AG News dataset citation should be the original dataset source, and TLoRA (Islam 2026) needs full bibliographic details if it is a published work.
  6. [Conclusion] The heading 'Conclusion' appears twice. The limitations paragraph itself is useful and appreciated; the duplication is a formatting issue.

Circularity Check

0 steps flagged

No significant circularity: the approximation theorem is a standard existence proof and the experiments are not derived from fitted inputs.

full rationale

The paper's central theoretical claim (Theorem 1) is an existence bound: for any target ΔW* that is a pointwise sample of an r-times differentiable function F*, it constructs coordinate assignments u_i=x*_i, v_j=y*_j and coefficients A_pq=c_pq, then invokes the classical Chebyshev truncation bound (Trefethen 2019). This is not circular: the target is not defined in terms of ChebyMA's parameters, and the bound O(d^{-r}) is imported from standard approximation theory rather than fitted to data. Step 3's construction is the usual feasible-point argument and does not make the theorem equivalent to its input. The parameter-efficiency comparison (Table 1) is arithmetic from the parameterization formulas, not a fitted prediction. Experimental results are measured accuracy values; the claim that they 'validate' the theory is an interpretive statement, not a derivation from fitted constants. The Discussion's TF-IDF control is confounded and the smoothness premise is not independently measured, but that is a limitation in external validity or empirical support, not a circular reduction. No load-bearing self-citation, imported uniqueness theorem, or ansatz-by-citation was found; StelLA (Li et al. 2025) is used only as a baseline.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 0 invented entities

The central claim depends on the smoothness of weight updates, an unverified domain assumption. The free parameters are mainly the hyperparameters d and S, plus an initialization scheme. No new physical or model entities are introduced.

free parameters (3)
  • polynomial degree d = 1–3 across experiments
    Chosen per dataset/operating point; ablation shows strong sensitivity (14.28pp gain from d=1 to d=3). The approximation theorem assumes a fixed d; the empirical Pareto front selects d per budget.
  • number of surfaces S = 1–8 across experiments
    Chosen per dataset/operating point; ablation shows 17.04pp gain from S=1 to S=8 with diminishing returns. Multi-manifold theory is only intuition.
  • coordinate initialization noise = N(0,0.05)
    Ad hoc choice to break symmetry; the paper does not test sensitivity to this value.
axioms (4)
  • domain assumption Target weight matrix is a pointwise sample of a C^r smooth function at ideal coordinates
    Step 1 of Theorem 1; this is the central premise of the approximation guarantee. Not proven for real weight updates.
  • standard math Classical Chebyshev truncation error bound: for F∈C^r, uniform error of degree-d best approximation is O(d^{-r})
    Cited to Trefethen 2019; standard approximation theory.
  • domain assumption Pre-trained feature embeddings have smooth index structure, so weight matrices are spatially smooth
    Invoked in Discussion to explain empirical gains; supported only by a TF-IDF control where the advantage disappears.
  • domain assumption The optimizer can find the constructed feasible parameter assignment
    Theorem 1 proves existence of a low-error solution, but training must discover it; non-convex optimization may not reach it. This is implicit throughout the empirical section.

pith-pipeline@v1.3.0-alltime-deepseek · 9560 in / 13364 out tokens · 124927 ms · 2026-08-01T18:08:18.788365+00:00 · methodology

0 comments
read the original abstract

The paper presents a new parameter-efficient adaptation method called ChebyMA (Chebyshev Manifold Adaptation). ChebyMA adopts weight matrices through a multi-surface superposition of Chebyshev polynomial bases evaluated on learnable coordinates and combined via trainable coefficient matrices, replacing standard linear projections with highly expressive continuous function approximation. Theoretically, we establish an Approximation Expressivity Theorem, proving from the perspective of function approximation theory that single-manifold ChebyMA guarantees convergence in Frobenius norm error of reconstruction. Besides, drawing on Kolmogorov $n$-width intuition, we demonstrate the expressive advantages of multi-manifold superposition ($S > 1$) in decoupling high-dimensional complex features. Experimental results on Computer Vision CIFAR datasets(CIFAR-10, CIFAR-100)\cite{CIFAR} and Natural Language Processing (AG News, SST-2) datasets demonstrate that ChebyMA consistently achieves a superior parameter-accuracy Pareto front compared to standard full-parameter fine-tuning, LoRA\cite{LoRA}, TLoRA\cite{TLoRA}, and StelLA\cite{StelLA}. ChebyMA significantly outperforms other tested methods in tested datasets, validating its solid theoretical foundation for generality with purely vectorized computations.

Figures

Figures reproduced from arXiv: 2607.17377 by Jiawen Li.

Figure 1
Figure 1. Figure 1: Overview of different methods The central insight is geometric: we treat each weight ma￾trix as a scalar field over its row-column index space and approximate it through a superposition of Chebyshev poly￾nomial hypersurfaces. Details are developed in the Method section. This perspective also reveals a unifying view of exist￾ing methods. LoRA, TLoRA, and StelLA can all be un￾derstood as special cases within… view at source ↗
Figure 2
Figure 2. Figure 2: visualizes the full parameter-accuracy Pareto front across all four datasets. ChebyMA consistently defines the upper-left boundary of the achievable trade-off at low-to￾mid parameter budgets, while LoRA converges to compara￾ble performance only at high budgets approaching the full parameter count [PITH_FULL_IMAGE:figures/full_fig_p005_2.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 3 canonical work pages · 2 internal anchors

  1. [1]

    and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , keywords =

    Hu, Edward J. and Shen, Yelong and Wallis, Phillip and Allen-Zhu, Zeyuan and Li, Yuanzhi and Wang, Shean and Wang, Lu and Chen, Weizhu , keywords =. LoRA: Low-Rank Adaptation of Large Language Models , publisher =. 2021 , copyright =. doi:10.48550/ARXIV.2106.09685 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2106.09685 2021

  2. [2]

    TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models , ISBN =

    Islam, Tanvir , year =. TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models , ISBN =. doi:10.1007/978-981-95-7078-2_14 , booktitle =

    work page doi:10.1007/978-981-95-7078-2_14

  3. [3]

    StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold , publisher =

    Li, Zhizhong and Sajadmanesh, Sina and Li, Jingtao and Lyu, Lingjuan , keywords =. StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold , publisher =. 2025 , copyright =. doi:10.48550/ARXIV.2510.01938 , url =

    doi:10.48550/arxiv.2510.01938 2025

  4. [4]

    Parameter-Efficient Transfer Learning for NLP , publisher =

    Houlsby, Neil and Giurgiu, Andrei and Jastrzebski, Stanislaw and Morrone, Bruna and de Laroussilhe, Quentin and Gesmundo, Andrea and Attariyan, Mona and Gelly, Sylvain , keywords =. Parameter-Efficient Transfer Learning for NLP , publisher =. 2019 , copyright =. doi:10.48550/ARXIV.1902.00751 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1902.00751 2019

  5. [5]

    The Power of Scale for Parameter-Efficient Prompt Tuning , publisher =

    Lester, Brian and Al-Rfou, Rami and Constant, Noah , keywords =. The Power of Scale for Parameter-Efficient Prompt Tuning , publisher =. 2021 , copyright =. doi:10.48550/ARXIV.2104.08691 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2104.08691 2021

  6. [6]

    AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning , publisher =

    Zhang, Qingru and Chen, Minshuo and Bukharin, Alexander and Karampatziakis, Nikos and He, Pengcheng and Cheng, Yu and Chen, Weizhu and Zhao, Tuo , keywords =. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning , publisher =. 2023 , copyright =. doi:10.48550/ARXIV.2303.10512 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2303.10512 2023

  7. [7]

    QLoRA: Efficient Finetuning of Quantized LLMs , publisher =

    Dettmers, Tim and Pagnoni, Artidoro and Holtzman, Ari and Zettlemoyer, Luke , keywords =. QLoRA: Efficient Finetuning of Quantized LLMs , publisher =. 2023 , copyright =. doi:10.48550/ARXIV.2305.14314 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2305.14314 2023

  8. [8]

    LoTR: Low Tensor Rank Weight Adaptation , publisher =

    Bershatsky, Daniel and Cherniuk, Daria and Daulbaev, Talgat and Mikhalev, Aleksandr and Oseledets, Ivan , keywords =. LoTR: Low Tensor Rank Weight Adaptation , publisher =. 2024 , copyright =. doi:10.48550/ARXIV.2402.01376 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.01376 2024

  9. [9]

    DoRA: Weight-Decomposed Low-Rank Adaptation , publisher =

    Liu, Shih-Yang and Wang, Chien-Yi and Yin, Hongxu and Molchanov, Pavlo and Wang, Yu-Chiang Frank and Cheng, Kwang-Ting and Chen, Min-Hung , keywords =. DoRA: Weight-Decomposed Low-Rank Adaptation , publisher =. 2024 , copyright =. doi:10.48550/ARXIV.2402.09353 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2402.09353 2024

  10. [10]

    MCNC: Manifold-Constrained Reparameterization for Neural Compression

    Thrash, Chayne and Abbasi, Ali and Andreas, Reed and Nooralinejad, Parsa and Koohpayegani, Soroush Abbasi and Pirsiavash, Hamed and Kolouri, Soheil , keywords =. MCNC: Manifold-Constrained Reparameterization for Neural Compression , publisher =. 2024 , copyright =. doi:10.48550/ARXIV.2406.19301 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2406.19301 2024

  11. [11]

    PRANC: Pseudo RAndom Networks for Compacting deep models

    Nooralinejad, Parsa and Abbasi, Ali and Koohpayegani, Soroush Abbasi and Meibodi, Kossar Pourahmadi and Khan, Rana Muhammad Shahroz and Kolouri, Soheil and Pirsiavash, Hamed , keywords =. PRANC: Pseudo RAndom Networks for Compacting deep models , publisher =. 2022 , copyright =. doi:10.48550/ARXIV.2206.08464 , url =

    work page internal anchor Pith review Pith/arXiv arXiv doi:10.48550/arxiv.2206.08464 2022

  12. [12]

    NOLA: Compressing LoRA using Linear Combination of Random Basis , publisher =

    Koohpayegani, Soroush Abbasi and Navaneet, KL and Nooralinejad, Parsa and Kolouri, Soheil and Pirsiavash, Hamed , keywords =. NOLA: Compressing LoRA using Linear Combination of Random Basis , publisher =. 2023 , copyright =. doi:10.48550/ARXIV.2310.02556 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2310.02556 2023

  13. [13]

    Sitzmann, Vincent and Martel, Julien N. P. and Bergman, Alexander W. and Lindell, David B. and Wetzstein, Gordon , keywords =. Implicit Neural Representations with Periodic Activation Functions , publisher =. 2020 , copyright =. doi:10.48550/ARXIV.2006.09661 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2006.09661 2020

  14. [14]

    and Tegmark, Max , keywords =

    Liu, Ziming and Wang, Yixuan and Vaidya, Sachin and Ruehle, Fabian and Halverson, James and Soljačić, Marin and Hou, Thomas Y. and Tegmark, Max , keywords =. KAN: Kolmogorov-Arnold Networks , publisher =. 2024 , copyright =. doi:10.48550/ARXIV.2404.19756 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2404.19756 2024

  15. [15]

    ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximations , publisher =

    Tang, Shanshan and Li, Bo and Yu, Haijun , keywords =. ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximations , publisher =. 2019 , copyright =. doi:10.48550/ARXIV.1911.05467 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1911.05467 2019

  16. [16]

    Fourier Neural Operator for Parametric Partial Differential Equations , publisher =

    Li, Zongyi and Kovachki, Nikola and Azizzadenesheli, Kamyar and Liu, Burigede and Bhattacharya, Kaushik and Stuart, Andrew and Anandkumar, Anima , keywords =. Fourier Neural Operator for Parametric Partial Differential Equations , publisher =. 2020 , copyright =. doi:10.48550/ARXIV.2010.08895 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.2010.08895 2020

  17. [17]

    Deep Residual Learning for Image Recognition , publisher =

    He, Kaiming and Zhang, Xiangyu and Ren, Shaoqing and Sun, Jian , keywords =. Deep Residual Learning for Image Recognition , publisher =. 2015 , copyright =. doi:10.48550/ARXIV.1512.03385 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1512.03385 2015

  18. [18]

    DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , publisher =

    Sanh, Victor and Debut, Lysandre and Chaumond, Julien and Wolf, Thomas , keywords =. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter , publisher =. 2019 , copyright =. doi:10.48550/ARXIV.1910.01108 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1910.01108 2019

  19. [19]

    and Ng, Andrew and Potts, Christopher

    Socher, Richard and Perelygin, Alex and Wu, Jean and Chuang, Jason and Manning, Christopher D. and Ng, Andrew and Potts, Christopher. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing. 2013

    2013

  20. [20]

    Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1 , pages =

    Zhang, Xiang and Zhao, Junbo and LeCun, Yann , title =. Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1 , pages =. 2015 , publisher =

    2015

  21. [21]

    Proceedings of the 32nd International Conference on Machine Learning - Volume 37 , pages =

    Ioffe, Sergey and Szegedy, Christian , title =. Proceedings of the 32nd International Conference on Machine Learning - Volume 37 , pages =. 2015 , publisher =

    2015

  22. [22]

    and Ba, Jimmy , keywords =

    Kingma, Diederik P. and Ba, Jimmy , keywords =. Adam: A Method for Stochastic Optimization , publisher =. 2014 , copyright =. doi:10.48550/ARXIV.1412.6980 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1412.6980 2014

  23. [23]

    Learning Multiple Layers of Features from Tiny Images , url =

    Krizhevsky, Alex , biburl =. Learning Multiple Layers of Features from Tiny Images , url =

  24. [24]

    and Kaiser,

    Vaswani, Ashish and Shazeer, Noam and Parmar, Niki and Uszkoreit, Jakob and Jones, Llion and Gomez, Aidan N. and Kaiser,. Attention is all you need , year =. Proceedings of the 31st International Conference on Neural Information Processing Systems , pages =

  25. [25]

    , title =

    Nair, Vinod and Hinton, Geoffrey E. , title =. Proceedings of the 27th International Conference on International Conference on Machine Learning , pages =. 2010 , isbn =

    2010

  26. [26]

    and Srivastava, Nitish and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan R

    Hinton, Geoffrey E. and Srivastava, Nitish and Krizhevsky, Alex and Sutskever, Ilya and Salakhutdinov, Ruslan R. , keywords =. Improving neural networks by preventing co-adaptation of feature detectors , publisher =. 2012 , copyright =. doi:10.48550/ARXIV.1207.0580 , url =

    Pith/arXiv arXiv doi:10.48550/arxiv.1207.0580 2012

  27. [27]

    2019 , publisher=

    Approximation Theory and Approximation Practice, Extended Edition , author=. 2019 , publisher=

    2019

  28. [28]

    , journal=

    Kolmogorov, Andrey N. , journal=. 1936 , publisher=

    1936

  29. [29]

    Bershatsky, D.; Cherniuk, D.; Daulbaev, T.; Mikhalev, A.; and Oseledets, I. 2024. LoTR: Low Tensor Rank Weight Adaptation

    2024

  30. [30]

    Dettmers, T.; Pagnoni, A.; Holtzman, A.; and Zettlemoyer, L. 2023. QLoRA: Efficient Finetuning of Quantized LLMs

    2023

  31. [31]

    He, K.; Zhang, X.; Ren, S.; and Sun, J. 2015. Deep Residual Learning for Image Recognition

    2015

  32. [32]

    E.; Srivastava, N.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R

    Hinton, G. E.; Srivastava, N.; Krizhevsky, A.; Sutskever, I.; and Salakhutdinov, R. R. 2012. Improving neural networks by preventing co-adaptation of feature detectors

    2012

  33. [33]

    Houlsby, N.; Giurgiu, A.; Jastrzebski, S.; Morrone, B.; de Laroussilhe, Q.; Gesmundo, A.; Attariyan, M.; and Gelly, S. 2019. Parameter-Efficient Transfer Learning for NLP

    2019

  34. [34]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2021. LoRA: Low-Rank Adaptation of Large Language Models

    2021

  35. [35]

    Ioffe, S.; and Szegedy, C. 2015. Batch normalization: accelerating deep network training by reducing internal covariate shift. In Proceedings of the 32nd International Conference on Machine Learning - Volume 37, ICML'15, 448–456. JMLR.org

    2015

  36. [36]

    Islam, T. 2026. TLoRA: Tri-Matrix Low-Rank Adaptation of Large Language Models, 208–223. Springer Nature Singapore. ISBN 9789819570782

    2026

  37. [37]

    P.; and Ba, J

    Kingma, D. P.; and Ba, J. 2014. Adam: A Method for Stochastic Optimization

    2014

  38. [38]

    U ber die beste Ann \

    Kolmogorov, A. N. 1936. \"U ber die beste Ann \"a herung von Funktionen einer gegebenen Funktionenklasse. Annals of Mathematics, 37(1): 107--110

    1936

  39. [39]

    A.; Navaneet, K.; Nooralinejad, P.; Kolouri, S.; and Pirsiavash, H

    Koohpayegani, S. A.; Navaneet, K.; Nooralinejad, P.; Kolouri, S.; and Pirsiavash, H. 2023. NOLA: Compressing LoRA using Linear Combination of Random Basis

    2023

  40. [40]

    Krizhevsky, A. 2009. Learning Multiple Layers of Features from Tiny Images. 32--33

    2009

  41. [41]

    Lester, B.; Al-Rfou, R.; and Constant, N. 2021. The Power of Scale for Parameter-Efficient Prompt Tuning

    2021

  42. [42]

    Li, Z.; Kovachki, N.; Azizzadenesheli, K.; Liu, B.; Bhattacharya, K.; Stuart, A.; and Anandkumar, A. 2020. Fourier Neural Operator for Parametric Partial Differential Equations

    2020

  43. [43]

    Li, Z.; Sajadmanesh, S.; Li, J.; and Lyu, L. 2025. StelLA: Subspace Learning in Low-rank Adaptation using Stiefel Manifold

    2025

  44. [44]

    F.; Cheng, K.-T.; and Chen, M.-H

    Liu, S.-Y.; Wang, C.-Y.; Yin, H.; Molchanov, P.; Wang, Y.-C. F.; Cheng, K.-T.; and Chen, M.-H. 2024 a . DoRA: Weight-Decomposed Low-Rank Adaptation

    2024

  45. [45]

    Y.; and Tegmark, M

    Liu, Z.; Wang, Y.; Vaidya, S.; Ruehle, F.; Halverson, J.; Soljačić, M.; Hou, T. Y.; and Tegmark, M. 2024 b . KAN: Kolmogorov-Arnold Networks

    2024

  46. [46]

    Nair, V.; and Hinton, G. E. 2010. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th International Conference on International Conference on Machine Learning, ICML'10, 807–814. Madison, WI, USA: Omnipress. ISBN 9781605589077

    2010

  47. [47]

    A.; Meibodi, K

    Nooralinejad, P.; Abbasi, A.; Koohpayegani, S. A.; Meibodi, K. P.; Khan, R. M. S.; Kolouri, S.; and Pirsiavash, H. 2022. PRANC: Pseudo RAndom Networks for Compacting deep models

    2022

  48. [48]

    Sanh, V.; Debut, L.; Chaumond, J.; and Wolf, T. 2019. DistilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter

    2019

  49. [49]

    Sitzmann, V.; Martel, J. N. P.; Bergman, A. W.; Lindell, D. B.; and Wetzstein, G. 2020. Implicit Neural Representations with Periodic Activation Functions

    2020

  50. [50]

    D.; Ng, A.; and Potts, C

    Socher, R.; Perelygin, A.; Wu, J.; Chuang, J.; Manning, C. D.; Ng, A.; and Potts, C. 2013. Recursive Deep Models for Semantic Compositionality Over a Sentiment Treebank. In Yarowsky, D.; Baldwin, T.; Korhonen, A.; Livescu, K.; and Bethard, S., eds., Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, 1631--1642. Seattle...

    2013

  51. [51]

    Tang, S.; Li, B.; and Yu, H. 2019. ChebNet: Efficient and Stable Constructions of Deep Neural Networks with Rectified Power Units via Chebyshev Approximations

    2019

  52. [52]

    A.; Pirsiavash, H.; and Kolouri, S

    Thrash, C.; Abbasi, A.; Andreas, R.; Nooralinejad, P.; Koohpayegani, S. A.; Pirsiavash, H.; and Kolouri, S. 2024. MCNC: Manifold-Constrained Reparameterization for Neural Compression

    2024

  53. [53]

    Trefethen, L. 2019. Approximation Theory and Approximation Practice, Extended Edition. Other Titles in Applied Mathematics. SIAM, Society for Industrial and Applied Mathematics. ISBN 9781611975949

    2019

  54. [54]

    Zhang, Q.; Chen, M.; Bukharin, A.; Karampatziakis, N.; He, P.; Cheng, Y.; Chen, W.; and Zhao, T. 2023. AdaLoRA: Adaptive Budget Allocation for Parameter-Efficient Fine-Tuning

    2023

  55. [55]

    Zhang, X.; Zhao, J.; and LeCun, Y. 2015. Character-level convolutional networks for text classification. In Proceedings of the 29th International Conference on Neural Information Processing Systems - Volume 1, NIPS'15, 649–657. Cambridge, MA, USA: MIT Press

    2015

This paper was first reviewed by deepseek-v4-flash on August 1, 2026.