Pith. sign in

REVIEW 23 cited by

KronA: Parameter Efficient Tuning with Kronecker Adapter

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2212.10650 v1 pith:F2XV7NX4 submitted 2022-12-20 cs.CL

Ali Edalati , Marzieh Tahaei , Ivan Kobyzev , Vahid Partovi Nia , James J. Clark , Mehdi Rezagholizadeh This is my paper · ORCID
classification cs.CL
keywords fine-tuningmethodsefficientkroneckerlow-rankmodelplmsadapter
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Fine-tuning a Pre-trained Language Model (PLM) on a specific downstream task has been a well-known paradigm in Natural Language Processing. However, with the ever-growing size of PLMs, training the entire model on several downstream tasks becomes very expensive and resource-hungry. Recently, different Parameter Efficient Tuning (PET) techniques are proposed to improve the efficiency of fine-tuning PLMs. One popular category of PET methods is the low-rank adaptation methods which insert learnable truncated SVD modules into the original model either sequentially or in parallel. However, low-rank decomposition suffers from limited representation power. In this work, we address this problem using the Kronecker product instead of the low-rank representation. We introduce KronA, a Kronecker product-based adapter module for efficient fine-tuning of Transformer-based PLMs. We apply the proposed methods for fine-tuning T5 on the GLUE benchmark to show that incorporating the Kronecker-based modules can outperform state-of-the-art PET methods.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 23 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. MoKA: Mixture of Kronecker Adapters

    cs.LG 2025-08 conditional novelty 6.0 of 10

    MoKA models weight updates as a gated mixture of Kronecker products and reports higher accuracy with up to 27x fewer parameters than QLoRA on 4-bit LLaMA models.

  2. Time Series Foundation Models for Multivariate Financial Time Series Forecasting

    q-fin.GN 2025-07 reject novelty 6.0 of 10

    Pretrained TTM shows large transfer and sample-efficiency gains in three financial forecasting tasks relative to training from scratch, but methodological flaws including possible look-ahead bias weaken the quantitati...

  3. Automatic Rank Determination for Low-Rank Adaptation via Submodular Function Maximization

    cs.LG 2025-07 conditional novelty 6.0 of 10

    SubLoRA projects the Hessian of the fine-tuning loss onto a submodular quadratic objective and uses greedy selection to automatically allocate LoRA ranks under a budget.

  4. Noise Consistency Regularization for Improved Subject-Driven Image Synthesis

    cs.GR 2025-06 conditional novelty 6.0 of 10

    Adding consistency-to-pretrained and multiplicative-noise consistency losses to fine-tuning improves subject identity and background diversity over DreamBooth on a 30-subject benchmark.

  5. SingLoRA: Low Rank Adaptation Using a Single Matrix

    cs.AI 2025-07 conditional novelty 5.0 of 10

    SingLoRA replaces LoRA's two matrices A and B with one matrix A and the symmetric update AA^T, cutting adapter parameters roughly in half while claiming more stable fine-tuning.

  6. EfficientLLM: Efficiency in Large Language Models

    cs.CL 2025-05 conditional novelty 5.0 of 10

    A large-scale benchmark of LLM efficiency techniques finds that every method trades off one resource for another, with the best choice depending on model scale, task, and hardware.

  7. SEAL: Entangled White-box Watermarks on Low-Rank Adaptation

    cs.AI 2025-01 reject novelty 5.0 of 10

    SEAL embeds a secret matrix between LoRA's low-rank factors, then decomposes and hides it in the released weights, but its ownership verification can be gamed by claiming the identity matrix as the passport.

  8. SuperCode: Sustainability PER AI-driven CO-DEsign

    astro-ph.IM 2024-12 unverdicted novelty 5.0 of 10

    The paper proposes an AI-driven hardware-software-science co-design methodology for radio astronomy, using sustainability as the key performance indicator, with no empirical results yet.

  9. Parameter-Efficient Fine-Tuning of 3D DDPM for MRI Image Generation Using Tensor Networks

    eess.IV 2025-07 conditional novelty 4.0 of 10

    TenVOO represents 3D convolution weight updates as tensor networks, fine-tuning a brain MRI DDPM with only 0.3% of full trainable parameters while achieving competitive or better structural similarity on ADNI, PPMI, a...

  10. Look Within or Look Beyond? A Theoretical Comparison Between Parameter-Efficient and Full Fine-Tuning

    cs.LG 2025-05 reject novelty 4.0 of 10

    The paper claims PEFT is a strict, less robust, lower-capacity subset of full fine-tuning, but the mathematical proofs contain load-bearing errors and the experiments, while suggestive, cannot repair them.

  11. $\mu$-MoE: Test-Time Pruning as Micro-Grained Mixture-of-Experts

    cs.LG 2025-05 conditional novelty 4.0 of 10

    Test-time Wanda pruning, reframed as a mixture of micro-experts, adapts the sparse weight mask to each prompt and improves perplexity and VQA accuracy over static pruning baselines.

  12. Parameter-Efficient Fine-Tuning of Multispectral Foundation Models for Hyperspectral Image Classification

    cs.CV 2025-05 conditional novelty 4.0 of 10

    KronA+ fine-tunes SpectralGPT for hyperspectral image classification using only 0.056% trainable parameters and reaches accuracy close to full fine-tuning on five public datasets.

  13. Dual Decomposition of Weights and Singular Value Low Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    DuDe combines DoRA's magnitude-direction decomposition with PiSSA's SVD-based initialization, reporting consistent but modest accuracy gains over LoRA, DoRA, and PiSSA on commonsense reasoning, GPQA, MMLU, and GSM8K.

  14. A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

    cs.SE 2025-04 conditional novelty 4.0 of 10

    This review synthesizes 28 studies to classify PEFT use in software engineering, finding LoRA and adapters dominant and PEFT often competitive with full fine-tuning.

  15. KARST: Multi-Kernel Kronecker Adaptation with Re-Scaling Transmission for Visual Classification

    cs.CV 2025-02 conditional novelty 4.0 of 10

    A sum of low-rank Kronecker adapters plus channel-wise re-scaling gives small average accuracy gains over prior PEFT methods on visual classification benchmarks.

  16. EDoRA: Efficient Weight-Decomposed Low-Rank Adaptation via Singular Value Decomposition

    cs.LG 2025-01 conditional novelty 4.0 of 10

    EDoRA combines DoRA's magnitude-direction decomposition with LoRA-XS's frozen SVD-based sandwich parameterization, cutting trainable parameters by roughly 30x versus LoRA and DoRA on GLUE with RoBERTa-base.

  17. A Resource-Efficient Training Framework for Remote Sensing Text--Image Retrieval

    cs.CV 2025-01 conditional novelty 4.0 of 10

    CMER reports 2 to 5 percent higher retrieval accuracy and 49 percent lower memory use using a Focus-Adapter, scene-label prompt augmentation, and negative sample recycling.

  18. Fast randomized Kronecker tensor decomposition: algorithms and error analysis

    math.NA 2024-12 reject novelty 4.0 of 10

    Randomized SVD is substituted for deterministic SVD inside the TTr1SVD algorithm to obtain a faster Kronecker tensor decomposition, with an unproven recursive error bound.

  19. Learning Text Styles: A Study on Transfer, Attribution, and Verification

    cs.CL 2025-07 conditional novelty 3.0 of 10

    A thesis compiles published work claiming that lightweight adapters, contrastive disentanglement, and instruction tuning improve text style transfer, authorship attribution, and authorship verification.

  20. Singular Value Decomposition on Kronecker Adaptation for Large Language Model

    cs.LG 2025-06 reject novelty 3.0 of 10

    SoKA combines Kronecker-product SVD initialization with spectrum-based rank selection, using 25% fewer parameters than PiSSA on LLaMA2-7B while trailing PiSSA by less than one point on three benchmarks.

  21. Progtuning: Progressive Fine-tuning Framework for Transformer-based Language Models

    cs.CL 2025-06 reject novelty 2.0 of 10

    A progressive scheduling trick that updates only the last remaining blocks in later epochs reduces parameter-update counts by about 25% with roughly unchanged GLUE and SQuAD scores.

  22. Parameter-Efficient Fine-Tuning for Foundation Models

    cs.CL 2025-01 conditional novelty 2.0 of 10

    A survey that categorizes and summarizes parameter-efficient fine-tuning methods across large language, vision, and multimodal models.

  23. Survey of different Large Language Model Architectures: Trends, Benchmarks, and Challenges

    cs.LG 2024-12 conditional

    A broad but error-prone survey of LLM and MLLM architectures, training methods, benchmarks, and challenges.

Pith tools