Pith. sign in

REVIEW 4 cited by

Progressive Prompts: Continual Learning for Language Models

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2301.12314 v1 pith:UEI3PAXJ submitted 2023-01-29 cs.CL cs.AIcs.LG

Anastasia Razdaibiedina , Yuning Mao , Rui Hou , Madian Khabsa , Mike Lewis , Amjad Almahairi This is my paper
classification cs.CLcs.AIcs.LG
keywords promptscontinuallearningprogressiveapproachlanguagemethodmethods
verification ladder T0 review T1 audit T2 compute T3 formal
0 comments
read the original abstract

We introduce Progressive Prompts - a simple and efficient approach for continual learning in language models. Our method allows forward transfer and resists catastrophic forgetting, without relying on data replay or a large number of task-specific parameters. Progressive Prompts learns a new soft prompt for each task and sequentially concatenates it with the previously learned prompts, while keeping the base model frozen. Experiments on standard continual learning benchmarks show that our approach outperforms state-of-the-art methods, with an improvement >20% in average test accuracy over the previous best-preforming method on T5 model. We also explore a more challenging continual learning setup with longer sequences of tasks and show that Progressive Prompts significantly outperforms prior methods.

Discussion (0). Sign in to comment.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Latent-LoRA: Compact Latent-Space Adapters with Gradient-Free Routing for Continual Learning

    cs.LG 2026-07 conditional novelty 5.5 of 10

    Frozen-embedding GMMs route compact SVD-subspace LoRA adapters for task-agnostic continual learning with SOTA average performance and near-zero forgetting.

  2. SOS-LoRA: Static Orthogonal-Subspace Low-Rank Adaptation with Fixed Multi-Scale Scaling

    cs.LG 2026-06 conditional novelty 5.0 of 10

    A LoRA update split into several fixed, differently-scaled low-rank experts with orthogonal input directions improves fine-tuning accuracy at the same parameter count.

  3. Enhancing Memory Recall in LLMs with Gauss-Tin: A Hybrid Instructional and Gaussian Replay Approach

    cs.LG 2025-08 reject novelty 4.0 of 10

    Gauss-Tin, a replay method using a Gaussian mixture model with prompt-guided exemplar selection, reports positive backward transfer on the Natural Instructions benchmark versus sequential fine-tuning.

  4. Continual Learning for Generative AI: From LLMs to MLLMs and Beyond

    cs.LG 2025-06 conditional novelty 4.0 of 10

    A survey that categorizes continual learning methods for generative models into architecture-based, regularization-based, and replay-based paradigms across four model families.

Pith tools