Pith. sign in

REVIEW 20 cited by

Orthogonal Subspace Learning for Language Model Continual Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2310.14152 v1 pith:XAQSR45Q submitted 2023-10-22 cs.CL cs.LG

Xiao Wang , Tianze Chen , Qiming Ge , Han Xia , Rong Bao , Rui Zheng , Qi Zhang , Tao Gui
show 1 more author
Xuanjing Huang
This is my paper
classification cs.CLcs.LG
keywords learninglanguagetaskscontinualmethodorthogonalcatastrophicforgetting
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Benefiting from massive corpora and advanced hardware, large language models (LLMs) exhibit remarkable capabilities in language understanding and generation. However, their performance degrades in scenarios where multiple tasks are encountered sequentially, also known as catastrophic forgetting. In this paper, we propose orthogonal low-rank adaptation (O-LoRA), a simple and efficient approach for continual learning in language models, effectively mitigating catastrophic forgetting while learning new tasks. Specifically, O-LoRA learns tasks in different (low-rank) vector subspaces that are kept orthogonal to each other in order to minimize interference. Our method induces only marginal additional parameter costs and requires no user data storage for replay. Experimental results on continual learning benchmarks show that our method outperforms state-of-the-art methods. Furthermore, compared to previous approaches, our method excels in preserving the generalization ability of LLMs on unseen tasks.

Discussion (0). Sign in to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Hidden Forgetting in Continual Multimodal Learning: When Accuracy Survives but Grounding Fails

    cs.AI 2026-07 unverdicted novelty 7.0 of 10

    RCL preserves evidence-reliance in continual multimodal learning to reduce hidden forgetting beyond standard accuracy metrics.

  2. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 unverdicted novelty 7.0 of 10

    The paper offers a comprehensive survey and proposes a new taxonomy for continual learning strategies in VLMs and MLLMs to combat catastrophic forgetting beyond traditional methods.

  3. TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    TFGN is an architectural overlay for transformers enabling task-free, replay-free continual pre-training across heterogeneous domains at LLM scale with near-zero backward transfer and high gradient orthogonality.

  4. Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Muon-OGD introduces a spectral-norm constrained orthogonal projection method solved via dual iterations and Newton-Schulz approximations to improve stability-plasticity trade-off in sequential LLM adaptation.

  5. Rotation-Preserving Supervised Fine-Tuning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    RPSFT improves the in-domain versus out-of-domain performance trade-off during LLM supervised fine-tuning by penalizing rotations in pretrained singular subspaces as a proxy for loss-sensitive directions.

  6. Compress Then Adapt? No, Do It Together via Task-aware Union of Subspaces

    cs.AI 2026-05 unverdicted novelty 6.0 of 10

    JACTUS unifies low-rank compression and task adaptation via a task-aware union of subspaces and global rank allocation by marginal gain, outperforming 100% PEFT methods like DoRA on ViT-Base (89.2% avg) and Llama2-7B ...

  7. Forget What's Sensitive, Remember What Matters: Token-Level Differential Privacy in Memory Sculpting for Continual Learning

    cs.AI 2025-09 unverdicted novelty 6.0 of 10

    PeCL applies token-level dynamic differential privacy and privacy-guided memory sculpting to achieve superior privacy-utility balance in continual learning.

  8. Little by Little: Continual Learning via Incremental Mixture of Rank-1 Associative Memory Experts

    cs.LG 2025-06 unverdicted novelty 6.0 of 10

    MoRAM frames continual learning as incremental addition of rank-1 adapters viewed as self-activating key-value associative memory units in a mixture-of-experts setup.

  9. Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection

    cs.CV 2026-07 reject novelty 5.0 of 10

    OKR beats exemplar-free domain-incremental detectors by 5.6–6.5 mAP using per-domain LoRA branches plus gradient orthogonality and prototype alignment.

  10. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  11. CADRE: Stable, Parameter Efficient Adaptation of Medical Vision Language Models with Bounded Forgetting and Prior Drift

    cs.AI 2026-06 unverdicted novelty 5.0 of 10

    CADRE combines LoRA with self-scaling similarity-aware EWC and an anchor-to-prior penalty to achieve lower forgetting and positive backward transfer when adapting a medical VLM across dissimilar imaging modalities whi...

  12. Mask the Target: A Plug-and-Play Regularizer Against LoRA Forgetting

    cs.CL 2026-05 unverdicted novelty 5.0 of 10

    A plug-and-play KL regularizer that masks the target token and renormalizes probabilities to improve the learning-forgetting trade-off in LoRA adaptation of LLMs.

  13. CP-MoE: Consistency-Preserving Mixture-of-Experts for Continual Learning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    CP-MoE uses a transient expert, consistency-preserving routing bias, and guided regularization to reduce catastrophic forgetting in MoE-based LLMs and VLMs while preserving cross-task transfer, reporting SOTA on Super...

  14. Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Muon-OGD integrates Muon-style spectral-norm geometry with orthogonal gradient constraints to improve the stability-plasticity trade-off during sequential LLM adaptation.

  15. SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

  16. Reliable Unlearning Harmful Information in LLMs with Metamorphosis Representation Projection

    cs.LG 2025-08 unverdicted novelty 5.0 of 10

    An LLM unlearning method that projects hidden states so harmful information is irreversibly removed while useful knowledge is preserved.

  17. GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay

    cs.CL 2025-08 conditional novelty 5.0 of 10

    A fixed 1,000-sample replay set from pretraining text, trained with a threshold-based margin loss on last-layer hidden states, reduces forgetting across 15 sequential finetuning tasks in Llama-3.1-8B.

  18. Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting

    cs.CV 2025-08 conditional novelty 5.0 of 10

    A taxonomy survey of continual learning for vision-language models, grouping methods into multi-modal replay, cross-modal regularization, and parameter-efficient adaptation, with a review of benchmarks and metrics.

  19. Preserving Knowledge in Large Language Model with Model-Agnostic Self-Decompression

    cs.CL 2024-06 unverdicted novelty 5.0 of 10

    Introduces Tree Generation (TG-SFT) to generate synthetic instruction-tuning data from LLMs, reducing catastrophic forgetting when fine-tuning MLLMs on domain-specific or multimodal data.

  20. Parameter-Efficient Fine-Tuning for Large Models: A Comprehensive Survey

    cs.LG 2024-03 accept novelty 4.0 of 10

    A comprehensive survey of PEFT algorithms for large models, covering their performance, overhead, applications, and real-world system implementations.

Pith tools