REVIEW 6 cited by
LAMOL: LAnguage MOdeling for Lifelong Language Learning
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
LAMOL: LAnguage MOdeling for Lifelong Language Learning
read the original abstract
Most research on lifelong learning applies to images or games, but not language. We present LAMOL, a simple yet effective method for lifelong language learning (LLL) based on language modeling. LAMOL replays pseudo-samples of previous tasks while requiring no extra memory or model capacity. Specifically, LAMOL is a language model that simultaneously learns to solve the tasks and generate training samples. When the model is trained for a new task, it generates pseudo-samples of previous tasks for training alongside data for the new task. The results show that LAMOL prevents catastrophic forgetting without any sign of intransigence and can perform five very different language tasks sequentially with only one model. Overall, LAMOL outperforms previous methods by a considerable margin and is only 2-3% worse than multitasking, which is usually considered the LLL upper bound. The source code is available at https://github.com/jojotenya/LAMOL.
Forward citations
Cited by 6 Pith papers
-
ReCoLoRA: Spectrum-Aware Recursive Consolidation for Continual LLM Fine-Tuning
Spectrum-initialized LoRA with elbow ranks and recursive SVD consolidation of the effective weight beats rank-swept PEFT baselines on three of four 7–8B models in continual GLUE fine-tuning.
-
Task-Differentiated Atomic Skill Expansion and Routing for Continual Learning Across Highly Heterogeneous Tasks
TASER dynamically expands and orthogonality-constrains atomic skills then routes them with task-conditioned gating, outperforming baselines on the new 19-task HeteroCLBench benchmark for heterogeneous continual learning.
-
Robust Policy Optimization to Prevent Catastrophic Forgetting
FRPO applies a max-min robust optimization over KL-bounded policy neighborhoods during RLHF to reduce catastrophic forgetting of safety and accuracy under subsequent SFT or RL fine-tuning.
-
Attribution-Guided Continual Learning for Large Language Models
An attribution-based continual learning framework for LLMs modulates per-parameter gradients using task-specific importance scores to reduce forgetting of prior tasks.
-
Attribution-Guided Continual Learning for Large Language Models
LRP-derived element-wise parameter importance scores gate gradients so parameters critical to earlier tasks receive smaller updates during continual LLM fine-tuning.
-
OrthoPhys: Physically Plausible Video Generation with Orthogonal-View Geometry Guidance
Generating synchronized four-view orthogonal foreground videos with geometry-enhanced attention, then using them as rigid guidance, improves physical realism in video generation over direct 2D methods.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.