Pith. sign in

REVIEW 5 cited by

LoRI: Reducing Cross-Task Interference in Multi-Task Low-Rank Adaptation

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2504.07448 v2 pith:7II4CS2Z submitted 2025-04-10 cs.LG cs.AIcs.CL

Juzheng Zhang , Jiacheng You , Ashwinee Panda , Tom Goldstein This is my paper
classification cs.LGcs.AIcs.CL
keywords loriinterferenceadaptercross-taskloramulti-taskadaptationcode
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
abstract

Low-Rank Adaptation (LoRA) has emerged as a popular parameter-efficient fine-tuning (PEFT) method for Large Language Models (LLMs), yet it still incurs notable overhead and suffers from parameter interference in multi-task scenarios. We propose LoRA with Reduced Interference (LoRI), a simple yet effective approach that freezes the projection matrices $A$ as random projections and sparsifies the matrices $B$ using task-specific masks. This design substantially reduces the number of trainable parameters while maintaining strong task performance. Moreover, LoRI minimizes cross-task interference in adapter merging by leveraging the orthogonality between adapter subspaces, and supports continual learning by using sparsity to mitigate catastrophic forgetting. Extensive experiments across natural language understanding, mathematical reasoning, code generation, and safety alignment tasks demonstrate that LoRI outperforms full fine-tuning and existing PEFT methods, while using up to 95% fewer trainable parameters than LoRA. In multi-task experiments, LoRI enables effective adapter merging and continual learning with reduced cross-task interference. Code is available at: https://github.com/juzhengz/LoRI

Discussion (0). Sign in to comment.

Forward citations

Cited by 5 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. REAL-OW: Rehearsal-free Open World Object Detection with Low-Rank Adaptation and Dual-Stage Objectness Modeling

    cs.CV 2026-07 conditional novelty 6.0 of 10

    A rehearsal-free open-world detector using collaborative LoRA adapters and dual-stage objectness modeling outperforms exemplar-replay OWOD methods on standard benchmarks.

  2. TARA: Token-Aware LoRA for Composable Personalization in Diffusion Models

    cs.CV 2025-08 conditional novelty 6.0 of 10

    TARA adds token-focused masking and a token alignment loss to LoRA adapters, allowing several independently trained personalized adapters to be composed with less identity loss and feature leakage.

  3. Decouple and Orthogonalize: A Data-Free Framework for LoRA Merging

    cs.CV 2025-05 conditional novelty 6.0 of 10

    A data-free LoRA merging framework that decouples weight magnitude from direction and orthogonalizes directions to reduce task interference, outperforming existing merging methods across vision, language and multimoda...

  4. When Less is More: 8-bit Quantization Improves Continual Learning in Large Language Models

    cs.LG 2025-12 conditional novelty 5.0 of 10

    Quantized (INT8/INT4) LLMs can outperform FP16 in later-task forward accuracy and retention during continual learning, though single-seed runs leave the effect unquantified.

  5. AFLoRA: Adaptive Federated Fine-Tuning of Large Language Models with Resource-Aware Low-Rank Adaption

    cs.LG 2025-05 conditional novelty 4.0 of 10

    AFLoRA prunes low-rank adapter dimensions per client, trains only the client-specific part locally and the shared part on the server, and aggregates heterogeneous updates by zero-padding and rank-aware weighting.

Pith tools