Pith. sign in

REVIEW 9 cited by

Spurious Forgetting in Continual Learning of Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2501.13453 v1 pith:JG65UMVE submitted 2025-01-23 cs.LG

Junhao Zheng , Xidi Cai , Shengjie Qiu , Qianli Ma This is my paper
classification cs.LG
keywords continuallearningtaskalignmentknowledgemodelmodelsperformance
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

Recent advancements in large language models (LLMs) reveal a perplexing phenomenon in continual learning: despite extensive training, models experience significant performance declines, raising questions about task alignment and underlying knowledge retention. This study first explores the concept of "spurious forgetting", proposing that such performance drops often reflect a decline in task alignment rather than true knowledge loss. Through controlled experiments with a synthesized dataset, we investigate the dynamics of model performance during the initial training phases of new tasks, discovering that early optimization steps can disrupt previously established task alignments. Our theoretical analysis connects these shifts to orthogonal updates in model weights, providing a robust framework for understanding this behavior. Ultimately, we introduce a Freezing strategy that fix the bottom layers of the model, leading to substantial improvements in four continual learning scenarios. Our findings underscore the critical distinction between task alignment and knowledge retention, paving the way for more effective strategies in continual learning.

Discussion (0). Sign in to comment.

Forward citations

Cited by 9 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Rethinking Transfer in Continual Learning: A Replay-Based Realisation

    cs.LG 2026-07 conditional novelty 7.0 of 10

    In continual learning, forward transfer requires target headroom, a persistent carrier, and a compatible source; routing replay by gradient signatures improves accuracy and stability over uniform replay.

  2. Curriculum Reinforcement Learning Can Incentivize Reasoning Capacity in LLMs Beyond the Base Model

    cs.LG 2026-06 unverdicted novelty 6.0 of 10

    Boundary-aware Curriculum RL raises average pass@256 by 9.8 points over base models and 10.3 points over vanilla RLVR on Qwen, Llama, and DeepSeek families.

  3. EpiEvolve: Self-Evolving Agents for Streaming Pandemic Forecasting under Regime Shifts

    cs.AI 2026-06 unverdicted novelty 6.0 of 10

    EpiEvolve achieves 0.629 accuracy in streaming COVID-19 forecasting by using episodic memory, reflection on delayed labels, and regime-aware retrieval, outperforming static LLMs (0.561) and CDC ensembles (0.325) while...

  4. Continual Safety Alignment via Gradient-Based Sample Selection

    cs.LG 2026-04 unverdicted novelty 6.0 of 10

    Gradient-based selection that drops high-gradient samples during continual fine-tuning preserves safety alignment in LLMs better than standard fine-tuning while keeping task performance competitive.

  5. UItron: Foundational GUI Agent with Advanced Perception and Planning

    cs.CV 2025-08 conditional novelty 6.0 of 10

    UItron, trained with multi-stage SFT and curriculum RL on a new 1M-step Chinese app dataset, beats prior GUI agents on perception, grounding, planning, and especially Chinese mobile app scenarios.

  6. Attribution-Guided Continual Learning for Large Language Models

    cs.LG 2026-05 unverdicted novelty 5.0 of 10

    An attribution-based continual learning framework for LLMs modulates per-parameter gradients using task-specific importance scores to reduce forgetting of prior tasks.

  7. Attribution-Guided Continual Learning for Large Language Models

    cs.LG 2026-05 conditional novelty 5.0 of 10

    LRP-derived element-wise parameter importance scores gate gradients so parameters critical to earlier tasks receive smaller updates during continual LLM fine-tuning.

  8. BRAIN: Bias-Mitigation Continual Learning Approach to Vision-Brain Understanding

    cs.CV 2025-08 unverdicted novelty 5.0 of 10

    BRAIN uses bias-mitigation continual learning with a new de-bias contrastive loss and angular forgetting mitigation to achieve SOTA performance on vision-brain understanding benchmarks despite brain signal inconsisten...

  9. Mixture-of-Translators: Translating KV Caches Across Heterogeneous Large Language Models

    cs.CL 2026-07 conditional novelty 4.0 of 10

    MoT translates KV caches between LLMs using token-level translator mixtures and a context-correction loss; QA results are strong for same-family pairs but do not demonstrate true cross-architecture reuse.

Pith tools