CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
hub
SkillLearnBench: Benchmarking Continual Learning Methods for Agent Skill Generation on Real-World Tasks
10 Pith papers cite this work. Polarity classification is still indexing.
10
Pith papers citing it
abstract
Skills have become the de facto way to enable LLM agents to perform complex real-world tasks with customized instructions, workflows, and tools, but how to learn them automatically and effectively remains unclear. We introduce SkillLearnBench, the first benchmark for evaluating continual skill learning methods, comprising 20 verified, skill-dependent tasks across 15 sub-domains derived from a real-world skill taxonomy , evaluated at three levels: skill quality, execution trajectory, and task outcome. Using this benchmark, we evaluate recent continual learning techniques, those leveraging one-shot, self/teacher feedback, and skill creator to generate skills from agent experiences. We find that all continual learning methods improve over the no-skill baseline, yet consistent gains remain elusive: no method leads across all tasks and LLMs, and scaling to stronger LLMs does not reliably help. Continual learning improves tasks with clear, reusable workflows but struggles on open-ended tasks, and using stronger LLM backbones does not consistently produce better skills. Our analysis also revealed that multiple iterations in continual learning facilitate genuine improvement via external feedback, whereas self-feedback alone induces recursive drift. Our data and code are open-source at https://github.com/cxcscmu/SkillLearnBench to enable further studies of automatic skill generation and continual learning techniques.
hub tools
citation-role summary
baseline 1
citation-polarity summary
years
2026 10roles
baseline 1polarities
baseline 1representative citing papers
Framework estimates context-dependent marginal utility of candidate skills via reward gaps in matched base vs. skill-augmented rollouts to filter skills and co-train policy as generator.
SkillOps maintains LLM skill libraries via Skill Contracts and ecosystem graphs, raising ALFWorld task success to 79.5% as a standalone agent and improving retrieval baselines by up to 2.9 points with near-zero library-time LLM cost.
SkillCoach introduces self-evolving rubrics derived from rollouts to evaluate and supervise four process dimensions of agentic skill-use separately from outcome success.
Contrastive success/failure evidence, replay-based patch validation, and topology-aware routing improve training-free skill self-evolution for LLM agents.
Empirical study finds Progressive Disclosure raises distinct resources touched (1.18 to 3.85) and uptake events (1.33 to 3.92) per trajectory, adds 17 passing trials out of 410 (+4.1%), with gains task-dependent.
Skill coverage measures which natural-language skill constraints an LLM agent trajectory exercises and passes, revealing low coverage on SkillsBench and enabling a 16% recovery of failed tasks via targeted skill emphasis.
Cost-aware skill rewriting that preserves task-relevant operational anchors reduces LLM-agent total token cost by ~7% on a 20-task held-out panel and ~15% across agent stacks while maintaining verifier quality.
A skill-lifecycle agent (create, memory, manage, evaluate, refine) beats Hermes, Codex, and Claude Code on SkillsBench/SkillLearnBench and transfers skills better.
Across 56 tasks, 9 model configurations, and 10,584 runs, LLM-generated skill files provided no reliable performance improvement over task-only prompting for data-science workflows.
citing papers explorer
-
Continual Learning Bench: Evaluating Frontier AI Systems in Real-World Stateful Environments
CL-Bench is the first expert-validated benchmark for continual learning in frontier LLMs across six real-world domains, showing limited gains and that naive in-context learning outperforms dedicated memory systems.
-
Co-Evolving Skill Generation and Policy Optimization
Framework estimates context-dependent marginal utility of candidate skills via reward gaps in matched base vs. skill-augmented rollouts to filter skills and co-train policy as generator.
-
SkillOps: Managing LLM Agent Skill Libraries as Self-Maintaining Software Ecosystems
SkillOps maintains LLM skill libraries via Skill Contracts and ecosystem graphs, raising ALFWorld task success to 79.5% as a standalone agent and improving retrieval baselines by up to 2.9 points with near-zero library-time LLM cost.
-
SkillCoach: Self-Evolving Rubrics for Evaluating and Enhancing Agentic Skill-Use
SkillCoach introduces self-evolving rubrics derived from rollouts to evaluate and supervise four process dimensions of agentic skill-use separately from outcome success.
-
SkillCAT: Contrastive, Assessment-Augmented and Topology-AwareSkill Self-Evolution for LLM Agents
Contrastive success/failure evidence, replay-based patch validation, and topology-aware routing improve training-free skill self-evolution for LLM agents.
-
SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior
Empirical study finds Progressive Disclosure raises distinct resources touched (1.18 to 3.85) and uptake events (1.33 to 3.92) per trajectory, adds 17 passing trials out of 410 (+4.1%), with gains task-dependent.
-
Skill Coverage: A Test Adequacy Metric for Agent Skills
Skill coverage measures which natural-language skill constraints an LLM agent trajectory exercises and passes, revealing low coverage on SkillsBench and enabling a 16% recovery of failed tasks via targeted skill emphasis.
-
What Should a Skill Remember? Quality--Cost Trade-offs in Cost-Aware Skill Rewriting for Language Model Agents
Cost-aware skill rewriting that preserves task-relevant operational anchors reduces LLM-agent total token cost by ~7% on a 20-task held-out panel and ~15% across agent stacks while maintaining verifier quality.
-
MUSE-Autoskill: Self-Evolving Agents via Skill Creation, Memory, Management, and Evaluation
A skill-lifecycle agent (create, memory, manage, evaluate, refine) beats Hermes, Codex, and Claude Code on SkillsBench/SkillLearnBench and transfers skills better.
-
Do LLM-Generated Skills Make Better AI Data Scientists? A Component Ablation Across Data-Science Workflows
Across 56 tasks, 9 model configurations, and 10,584 runs, LLM-generated skill files provided no reliable performance improvement over task-only prompting for data-science workflows.