Pith. sign in

REVIEW 17 cited by

Orthogonal Subspace Learning for Language Model Continual Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.14152 v1 pith:XAQSR45Q submitted 2023-10-22 cs.CL cs.LG

Xiao Wang , Tianze Chen , Qiming Ge , Han Xia , Rong Bao , Rui Zheng , Qi Zhang , Tao Gui
show 1 more author
Xuanjing Huang
This is my paper · ORCID
classification cs.CLcs.LG
keywords learninglanguagetaskscontinualmethodorthogonalcatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal

Signed reviews

No signed human review yet.

0 comments
read the original abstract

Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 17 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

  2. In-Context Collapse in Vision-Language Models and How to Mitigate it?

    cs.CV 2026-08 conditional novelty 6.0 of 10

    Many-shot in-context learning in vision-language models can collapse accuracy as demonstrations accumulate, and the failure is causally localized to the vision-language integration pathway, where a small adapter repairs it.

  3. CLoRA: Parameter-Efficient Continual Learning with Low-Rank Adaptation

    cs.LG 2025-07 conditional novelty 6.0 of 10

    A single LoRA adapter shared across tasks, combined with knowledge distillation, enables resource-efficient class-incremental semantic segmentation with accuracy comparable to full fine-tuning.

  4. One Rank at a Time: Cascading Error Dynamics in Sequential Learning

    cs.LG 2025-05 conditional novelty 6.0 of 10

    Errors from each rank-1 step in sequential low-rank learning compound through factors that grow when singular values are close, so early steps deserve more compute.

  5. Demystifying Catastrophic Forgetting in Two-Stage Incremental Object Detector

    cs.CV 2025-02 conditional novelty 6.0 of 10

    Forgetting in incremental Faster R-CNN comes mainly from the RoI classification head; replaying coarse and fine RoI feature prototypes plus null-space gradient projection reduces forgetting and raises old-class mAP.

  6. Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection

    cs.CV 2026-07 reject novelty 5.0 of 10

    OKR beats exemplar-free domain-incremental detectors by 5.6–6.5 mAP using per-domain LoRA branches plus gradient orthogonality and prototype alignment.

  7. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  8. SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

  9. Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    An LLM unlearning method that projects hidden states so harmful information is irreversibly removed while useful knowledge is preserved.

  10. GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.

  11. SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting

    cs.LG 2025-05 reject novelty 5.0 of 10

    SplitLoRA picks the LoRA update subspace size from previous-task gradient singular values using a hyperparameter alpha, and freezes the projection to keep updates in that subspace.

  12. DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning

    cs.LG 2024-11 reject novelty 5.0 of 10

    DESIRE combines LoRA merging with feature-level entropy minimization and pseudo-feature classifier replay to improve rehearsal-free class-incremental learning, but it fits merging coefficients on unlabeled test data.

  13. Multi-Task Model Merging via Adaptive Weight Disentanglement

    cs.LG 2024-11 conditional novelty 5.0 of 10

    AWD improves multi-task model merging by optimizing task vectors toward orthogonality through a shared redundant-vector subtraction.

  14. Multi LoRA Meets Vision: Merging multiple adapters to create a multi task model

    cs.CV 2024-11 conditional novelty 5.0 of 10

    Merging LoRA adapters across six vision tasks produces usable multitask models, with performance depending strongly on dataset similarity.

  15. RegCL: Continual Adaptation of Segment Anything Model via Model Merging

    cs.LG 2025-07 conditional novelty 4.0 of 10

    RegCL merges LoRA adapters for SAM continually via an incremental form of RegMean, achieving higher retention than sequential fine-tuning across five segmentation datasets.

  16. Complementary Subspace Low-Rank Adaptation of Vision-Language Models for Few-Shot Classification

    cs.CV 2025-01 conditional novelty 4.0 of 10

    Comp-LoRA confines LoRA updates to the SVD residual subspace of CLIP weights, yielding small but consistent improvements over direct LoRA in few-shot classification and in preserving zero-shot ability.

  17. Polysemy of Synthetic Neurons Towards a New Type of Explanatory Categorical Vector Spaces

    cs.CL 2025-04 reject novelty 3.0 of 10

    A GPT2-XL analysis reports that a neuron's highest-activation tokens are also the ones most similar to multiple categorical subclusters, offered as evidence for an intra-neuronal vector-space view of polysemy.

Pith tools