REVIEW 25 cited by
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models
read the original abstract
Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. However, the continual learning aspect of these aligned LLMs has been largely overlooked. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs, owing to both their simplicity and the models' potential exposure during instruction tuning. In this paper, we introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs. TRACE consists of 8 distinct datasets spanning challenging tasks including domain-specific tasks, multilingual capabilities, code generation, and mathematical reasoning. All datasets are standardized into a unified format, allowing for effortless automatic evaluation of LLMs. Our experiments show that after training on TRACE, aligned LLMs exhibit significant declines in both general ability and instruction-following capabilities. For example, the accuracy of llama2-chat 13B on gsm8k dataset declined precipitously from 28.8\% to 2\% after training on our datasets. This highlights the challenge of finding a suitable tradeoff between achieving performance on specific tasks while preserving the original prowess of LLMs. Empirical findings suggest that tasks inherently equipped with reasoning paths contribute significantly to preserving certain capabilities of LLMs against potential declines. Motivated by this, we introduce the Reasoning-augmented Continual Learning (RCL) approach. RCL integrates task-specific cues with meta-rationales, effectively reducing catastrophic forgetting in LLMs while expediting convergence on novel tasks.
Forward citations
Cited by 25 Pith papers
-
Rethinking Transfer in Continual Learning: A Replay-Based Realisation
In continual learning, forward transfer requires target headroom, a persistent carrier, and a compatible source; routing replay by gradient signatures improves accuracy and stability over uniform replay.
-
CEO-Bench: Can Agents Play the Long Game?
Only two of ten advanced AI agents finish a 500-day simulated CEO challenge above the starting cash, and none surpass a hand-tuned rule-based baseline.
-
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
ProcBench provides a standardized process-level evaluation framework for LLM coding agents using an ontology of 11 defect types across 4 categories and control preservation scores.
-
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
ProcCtrlBench introduces an ontology of 11 defect types across 4 categories plus control preservation metrics to evaluate LLM coding agent trajectories on 200 cases from AndroidBench, TerminalBench, and SWE-bench-Verified.
-
The Parts Are Greater Than the Sum: Automated Task Sequencing for Efficient Training of Multi-Policy LLMs
Automatically grouping and sequencing tasks into multiple QLoRA adapters improves continual fine-tuning performance over a single shared adapter at matched trainable capacity.
-
Task-Differentiated Atomic Skill Expansion and Routing for Continual Learning Across Highly Heterogeneous Tasks
TASER dynamically expands and orthogonality-constrains atomic skills then routes them with task-conditioned gating, outperforming baselines on the new 19-task HeteroCLBench benchmark for heterogeneous continual learning.
-
CEO-Bench: Can Agents Play the Long Game?
CEO-Bench evaluates AI agents on managing a startup over 500 days, showing that even top models like Claude Opus 4.8 and GPT-5.5 barely maintain starting capital and fail to turn consistent profits.
-
RECAP: Regression Evaluation for Continual Adaptation of Prompts
RECAP benchmark finds that six prompt optimization methods show no significant performance gains under proactive continual adaptation to evolving constraints across four LLMs.
-
Dynamic Proxy-Mixing: Transferring Replay Controllers from Small to Large Models for Continual Instruction Tuning
PROXYMIX learns a dynamic replay controller on a small proxy model and transfers it to a large target model, improving accuracy by 3.4 points and reducing forgetting by 3.5 points on LLaMA-3-8B continual tuning sequences.
-
ProcCtrlBench: Evaluating Process-Level Defects and Control Preservation in LLM Coding Agents
ProcBench is a benchmark framework that organizes process defects into an ontology, standardizes agent trajectories, and uses calibrated risk-based scorecards to evaluate LLM coding agents beyond final outcomes.
-
Is One Score Enough? Rethinking the Evaluation of Sequentially Evolving LLM Memory
SeqMem-Eval reveals that high final accuracy in sequential LLM memory tasks often coexists with substantial forgetting and negative transfer, exposing stability-adaptability trade-offs hidden by standard aggregate metrics.
-
TFGN: Task-Free, Replay-Free Continual Pre-Training Without Catastrophic Forgetting at LLM Scale
TFGN is an architectural overlay for transformers enabling task-free, replay-free continual pre-training across heterogeneous domains at LLM scale with near-zero backward transfer and high gradient orthogonality.
-
Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning
Muon-OGD introduces a spectral-norm constrained orthogonal projection method solved via dual iterations and Newton-Schulz approximations to improve stability-plasticity trade-off in sequential LLM adaptation.
-
CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning
CRAFT is a continual learning method for LLMs that applies low-rank interventions on hidden states, unified by KL divergence for routing similar tasks, regularizing against forgetting, and merging updates, showing red...
-
LLM Evolution as an Industry-Scale Ecosystem: A Lifecycle Perspective on Continual Learning
The paper reformulates industrial continual learning for LLMs as a closed-loop ecosystem problem, identifies three core challenges, and organizes solutions around five lifecycle design principles.
-
CRMA: A Spectrally-Bounded Backbone for Modular Continual Fine-Tuning of LLMs
CRMA adds a spectrally bounded residual adapter backbone to modular continual fine-tuning of LLMs, achieving near-zero loss drift and positive backward transfer on Mistral-7B across domains.
-
TRACE: Discovering Task-Specific Parameter via Adaptation-Aware Probing for Continual Fine-Tuning
TRACE uses a brief adaptation probe to isolate task-specific parameters and updates only those during continual LLM fine-tuning to reduce catastrophic forgetting.
-
Trajectory Supervision for Continual Tool-Use Learning in LLMs
Retaining tool-use trajectories during sequential fine-tuning on API domains improves next-call prediction accuracy by 17.7 points over stripped-history training.
-
Muon-OGD: Muon-based Spectral Orthogonal Gradient Projection for LLM Continual Learning
Muon-OGD integrates Muon-style spectral-norm geometry with orthogonal gradient constraints to improve the stability-plasticity trade-off during sequential LLM adaptation.
-
CRAFT: Forgetting-Aware Intervention-Based Adaptation for Continual Learning
CRAFT is a continual learning method for LLMs that learns low-rank interventions on hidden representations, using a unified KL-divergence objective to handle task routing by output divergence, forgetting control via p...
-
LifeAlign: Lifelong Alignment for Large Language Models with Memory-Augmented Focalized Preference Optimization
LifeAlign uses focalized preference optimization and short-to-long memory consolidation via dimensionality reduction to let LLMs align with new preferences while retaining prior knowledge.
-
RPO-PDT: Demonstrating Role-Play-Based Knowledge Adaptation for Student Support Dialogue (Demonstration System)
RPO-PDT demonstrates a role-play-based, retrieval-grounded system for adaptive, policy-constrained student support dialogue with reverse-roleplay for strategy memory.
-
TailLoR: Protecting Principal Components in Parameter-Efficient Continual Learning
TailLoR applies low-rank updates to the singular value matrix of pre-trained weights while using a soft spectral penalty to protect dominant singular directions during continual learning.
-
RIZZ: Routing Interactions to Near Zero-Interference Zones for Continual Adaptation of Black-Box Agents
RIZZ is a continual adaptation framework for black-box LLM agents that uses dynamically spawned memory branches, context-aware routing, verifier-gated updates, and prompt compilation to control interference across non...
-
A Survey of Self-Evolving Agents: What, When, How, and Where to Evolve on the Path to Artificial Super Intelligence
The paper delivers the first systematic review of self-evolving agents, structured around what components evolve, when adaptation occurs, and how it is implemented.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.