Pith. sign in

REVIEW 20 cited by

Gradient Projection Memory for Continual Learning

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2103.09762 v1 pith:GZJPF47W submitted 2021-03-17 cs.LG cs.CV

Gobinda Saha , Isha Garg , Kaushik Roy This is my paper
classification cs.LGcs.CV
keywords gradienttaskslearningmemorynetworkpastapproachesartificial
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

The ability to learn continually without forgetting the past tasks is a desired attribute for artificial learning systems. Existing approaches to enable such learning in artificial neural networks usually rely on network growth, importance based weight update or replay of old data from the memory. In contrast, we propose a novel approach where a neural network learns new tasks by taking gradient steps in the orthogonal direction to the gradient subspaces deemed important for the past tasks. We find the bases of these subspaces by analyzing network representations (activations) after learning each task with Singular Value Decomposition (SVD) in a single shot manner and store them in the memory as Gradient Projection Memory (GPM). With qualitative and quantitative analyses, we show that such orthogonal gradient descent induces minimum to no interference with the past tasks, thereby mitigates forgetting. We evaluate our algorithm on diverse image classification datasets with short and long sequences of tasks and report better or on-par performance compared to the state-of-the-art approaches.

Discussion (0). Sign in to comment.

Forward citations

Cited by 20 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Recursive Binding on a Budget: Subspace Carving in Order-p Tensor Memories

    cs.LG 2026-06 unverdicted novelty 7.0 of 10

    Orthogonal Subspace Carving decouples tensor order from recursion depth by null-space projections, enabling deep symbolic binding in constant-size memories and framing TPR as a Clifford algebra case.

  2. Interference-Aware Multi-Task Unlearning

    cs.AI 2026-05 unverdicted novelty 7.0 of 10

    Introduces interference-aware multi-task unlearning with task-aware gradient projection and instance-level gradient orthogonalization, reducing interference scores by 30.3% and 52.9% on vision benchmarks.

  3. Continual Learning of Domain-Invariant Representations

    cs.LG 2026-05 unverdicted novelty 7.0 of 10

    Introduces replay-based continual learning with sequential invariance alignment to learn domain-invariant representations, outperforming baselines on generalization to unseen domains across six datasets in vision, med...

  4. SLE-FNO: Single-Layer Extensions for Task-Agnostic Continual Learning in Fourier Neural Operators

    cs.LG 2026-03 unverdicted novelty 7.0 of 10

    SLE-FNO achieves zero forgetting and strong plasticity-stability balance in continual learning for FNO surrogate models of pulsatile blood flow by adding minimal single-layer extensions across four out-of-distribution tasks.

  5. TriShield: Zero-Utility-Loss Defense Against Privacy Backdoors in Federated Language Model Fine-Tuning via Orthogonal Gradient Projection and Optimizer State Entanglement

    cs.LG 2026-07 conditional novelty 6.0 of 10

    TriShield combines artifact detection, Adam momentum pre-entanglement, and SVD task-subspace projection to drive NeuroImprint reconstruction to 0% with claimed near-zero utility loss.

  6. SUM: Unified Geometric Surgery on Spatio-Temporal Adaptation Vectors for Federated Class Incremental Learning

    cs.LG 2026-07 conditional novelty 6.0 of 10

    SUM projects client and task adaptation vectors to remove directional interference during server aggregation, improving federated class-incremental learning accuracy without client-side changes.

  7. FreqOrtho-SR: Frequency-Guided Orthogonal Expert Learning for Real-World Image Super-Resolution

    cs.CV 2026-06 unverdicted novelty 6.0 of 10

    FreqOrtho-SR combines FFT-routed MoE adapters with SVD-based orthogonal projection of semantic gradients to improve fidelity-perception trade-off in single-step real-world super-resolution.

  8. Task-Differentiated Atomic Skill Expansion and Routing for Continual Learning Across Highly Heterogeneous Tasks

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    TASER dynamically expands and orthogonality-constrains atomic skills then routes them with task-conditioned gating, outperforming baselines on the new 19-task HeteroCLBench benchmark for heterogeneous continual learning.

  9. PACT: Preserving Anchored Cores in Task-vectors for Model Merging

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    PACT preserves load-bearing wall dimensions from pre-trained weights inside task vectors to reduce conflicts and improve merged model performance.

  10. Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

    cs.LG 2026-05 unverdicted novelty 6.0 of 10

    Muon-OGD introduces a spectral-norm constrained orthogonal projection method solved via dual iterations and Newton-Schulz approximations to improve stability-plasticity trade-off in sequential LLM adaptation.

  11. The Devil Is in Gradient Entanglement: Energy-Aware Gradient Coordinator for Robust Generalized Category Discovery

    cs.LG 2026-03 unverdicted novelty 6.0 of 10

    EAGC mitigates gradient entanglement in GCD by anchoring supervised gradients and adaptively projecting unlabeled ones, boosting existing methods to new state-of-the-art performance.

  12. PLATE: Plasticity-Tunable Efficient Adapters for Geometry-Aware Continual Learning

    cs.LG 2026-02 conditional novelty 6.0 of 10

    PLATE constructs frozen weight-derived bases B and Q and trains only a small core A, reducing catastrophic forgetting during data-free continual fine-tuning.

  13. Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Frozen-embedding GMMs route compact SVD-subspace LoRA adapters for task-agnostic continual learning with SOTA average performance and near-zero forgetting.

  14. TriShield: Zero-Utility-Loss Defense Against Privacy Backdoors in Federated Language Model Fine-Tuning via Orthogonal Gradient Projection and Optimizer State Entanglement

    cs.LG 2026-07 reject novelty 5.0 of 10

    TriShield claims to eliminate NeuroImprint-style privacy backdoors in federated fine-tuning, but its full-scale results are projections and its key theorems assume the very conditions they need to establish.

  15. Orthogonal Knowledge Refreshing for Domain-Incremental Object Detection

    cs.CV 2026-07 reject novelty 5.0 of 10

    OKR beats exemplar-free domain-incremental detectors by 5.6–6.5 mAP using per-domain LoRA branches plus gradient orthogonality and prototype alignment.

  16. Invariant Reasoning Directions in Latent Trajectories of Language Models

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    TILR identifies low-rank invariant subspaces from contrastive latent trajectory differences in LLMs and constrains interventions to them, improving paraphrase consistency by ~10% and reducing variance by up to 50%.

  17. LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning

    cs.LG 2026-06 unverdicted novelty 5.0 of 10

    The paper reformulates industrial continual learning for LLMs as a closed-loop ecosystem problem, identifies three core challenges, and organizes solutions around five lifecycle design principles.

  18. Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    Muon-OGD integrates Muon-style spectral-norm geometry with orthogonal gradient constraints to improve the stability-plasticity trade-off during sequential LLM adaptation.

  19. SelfAug: Mitigating Catastrophic Forgetting in Retrieval-Augmented Generation via Distribution Self-Alignment

    cs.CL 2025-09 conditional novelty 5.0 of 10

    Adding a KL penalty between fine-tuned and original model logits on input tokens during RAG fine-tuning reduces catastrophic forgetting while preserving downstream performance.

  20. Seeing Clearly, Forgetting Deeply: Revisiting Fine-Tuned Video Generators for Driving Simulation

    cs.CV 2025-08 conditional novelty 5.0 of 10

    Fine-tuning video generators on driving data can improve visual fidelity while degrading how accurately the model predicts the movement of cars and pedestrians.

Pith tools