Pith. sign in

REVIEW 8 cited by

CorDA: Context-Oriented Decomposition Adaptation of Large Language Models for Task-Aware Parameter-Efficient Fine-tuning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2406.05223 v3 pith:K7UB6H6M submitted 2024-06-07 cs.LG cs.AI

Yibo Yang , Xiaojie Li , Zhongzhu Zhou , Shuaiwen Leon Song , Jianlong Wu , Liqiang Nie , Bernard Ghanem This is my paper · ORCID
classification cs.LGcs.AI
keywords decompositionadaptationcontextfine-tuningknowledgesamplestaskcomponents
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
abstract

Current parameter-efficient fine-tuning (PEFT) methods build adapters widely agnostic of the context of downstream task to learn, or the context of important knowledge to maintain. As a result, there is often a performance gap compared to full-parameter fine-tuning, and meanwhile the fine-tuned model suffers from catastrophic forgetting of the pre-trained world knowledge. In this paper, we propose CorDA, a Context-oriented Decomposition Adaptation method that builds learnable task-aware adapters from weight decomposition oriented by the context of downstream task or the world knowledge to maintain. Concretely, we collect a few data samples, and perform singular value decomposition for each linear layer of a pre-trained LLM multiplied by the covariance matrix of the input activation using these samples. The inverse of the covariance matrix is multiplied with the decomposed components to reconstruct the original weights. By doing so, the context of the representative samples is captured through deciding the factorizing orientation. Our method enables two options, the knowledge-preserved adaptation and the instruction-previewed adaptation. For the former, we use question-answering samples to obtain the covariance matrices, and use the decomposed components with the smallest $r$ singular values to initialize a learnable adapter, with the others frozen such that the world knowledge is better preserved. For the latter, we use the instruction data from the fine-tuning task, such as math or coding, to orientate the decomposition and train the largest $r$ components that most correspond to the task to learn. We conduct extensive experiments on Math, Code, and Instruction Following tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 8 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. LatentLLM: Attention-Aware Joint Tensor Compression

    cs.LG 2025-05 conditional novelty 6.0 of 10

    LatentLLM compresses pretrained LLMs and multimodal models with attention-aware joint low-rank tensor decomposition, outperforming SVD-based baselines on OPT perplexity and LLaVA ScienceQA.

  2. Continuous Knowledge-Preserving Decomposition with Adaptive Layer Selection for Few-Shot Class-Incremental Learning

    cs.CV 2025-01 reject novelty 6.0 of 10

    CKPD-FSCIL uses covariance-guided weight decomposition and adaptive layer selection to learn new classes incrementally without changing the model's architecture or inference cost.

  3. EAGLE: Enhanced Visual Grounding Minimizes Hallucinations in Instructional Multimodal Models

    cs.CV 2025-01 conditional novelty 6.0 of 10

    Tuning a CLIP-style visual encoder with masked-pooling contrastive losses on instance segmentation data reduces hallucinations in downstream instruction-tuned vision-language models when the encoder is swapped in with...

  4. AWARe: Mitigating Catastrophic Forgetting via Activation-Weighted Adaptive REtention

    cs.CL 2026-08 conditional novelty 5.0 of 10

    AWARe freezes the highest-activation neurons in selected projection layers during fine-tuning, reducing catastrophic forgetting in multimodal LLMs.

  5. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  6. CoLA: Collaborative Low-Rank Adaptation

    cs.CL 2025-05 conditional novelty 4.0 of 10

    CoLA generalizes LoRA to multiple A and B matrices with a principal-component initialization and reports gains of roughly 2-4 accuracy points over PiSSA on low-sample fine-tuning benchmarks.

  7. A Systematic Literature Review of Parameter-Efficient Fine-Tuning for Large Code Models

    cs.SE 2025-04 conditional novelty 4.0 of 10

    This review synthesizes 28 studies to classify PEFT use in software engineering, finding LoRA and adapters dominant and PEFT often competitive with full fine-tuning.

  8. Spectral-Aware Low-Rank Adaptation for Speaker Verification

    eess.AS 2025-01 conditional novelty 4.0 of 10

    SpectralFT fine-tunes only the top singular vectors of pre-trained Transformer weights for speaker verification, achieving modest EER reductions over LoRA on two benchmarks.

Pith tools