Pith. sign in

REVIEW 3 major objections 5 minor 35 references

Solving tasks sequentially helps LLM agents by 16.9% on average, yet a stored skill library performs no better than pure in-context learning—the gains come from context and feedback, not reusable skills.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Across five 100-task agent streams, sequential experience improves normalized reward by 16.9% in 14 of 15 model-domain combinations, but explicit skill maintenance matches pure in-context learning (0.602 vs 0.605) and weaker models build larger, less reusable skill pools.

T0 review reviewed 2026-08-05 challenge →

load-bearing objection A reusable 500-task benchmark and a clean three-way comparison, but the headline ICL-vs-skills parity rests on one model, three domains, and single runs, so treat the claim as conditional on the harness. the 3 major comments →

arxiv 2608.03874 v1 pith:IGVUSGZS submitted 2026-08-04 cs.AI cs.CLcs.LG

ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?

Tianyi Guan , Yiding Wang , Haotong Yang , Siyuan Cao , Shirui Liu , Yi Hu , Jiaqi Li , Muhan Zhang This is my paper
classification cs.AI cs.CLcs.LG
keywords continual skill learningin-context learningLLM agentsskill librariesbenchmark constructiontask sequencingskill consolidationagent evaluation
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether LLM agents truly evolve reusable skills while solving a stream of tasks, the premise behind agent platforms that ship external skill libraries. It builds ContinualSkillBench—five 100-task streams in Law, Finance, Healthcare, Math, and Office, ordered so later tasks reuse skills exercised earlier—and measures agents that work through the stream against agents that solve each task from scratch. Sequential execution wins: normalized reward improves in 14 of 15 model–domain combinations, a 16.9% relative aggregate gain. But the paper's sharper finding is that a pure in-context agent, which keeps only the conversation and feedback and never builds a skill library, matches the skill-maintaining agent (0.605 vs 0.602 normalized reward). The upshot: current agents adapt to context and feedback well, but consolidate experience into robust, transferable skills poorly—stored skills help mainly where outputs or procedures must be exact, and weaker models accumulate larger, more fragmented libraries.

Core claim

On its own terms, the paper claims that current in-context skill evolution mechanisms let agents adapt sequentially but do not yet consolidate experience into robust, transferable skills. The evidence: sequential execution beats independent execution in 14 of 15 model–domain combinations, with an aggregate 16.9% relative gain in normalized reward; but a pure in-context agent that keeps the task order and feedback while dropping the skill library matches explicit skill maintenance (0.605 vs 0.602 normalized reward across Law, Finance, and Healthcare), so most of the sequential gain is contextual adaptation. Explicit skills retain selective value where output or execution formats are rigid—exa

What carries the argument

The load-bearing object is ContinualSkillBench itself: five streams of 100 interconnected subtasks, assembled from a three-tier pool of source benchmarks and ordered by LLM-judged pairwise skill-transfer dependencies under a difficulty constraint, then validated structurally—69.5% of tasks reuse at least one core skill from earlier in their stream, and the curated order beats random permutations on local skill coverage. Evaluation runs each task through a three-turn protocol: instruction with the current skill index, execution, and feedback-driven reflection in which the agent may create or edit skills through create-skill and modify-skill meta-skills. The critical control is the pure in-con

Load-bearing premise

The benchmark's skill-maintenance harness—the three-turn protocol, the create-skill/modify-skill meta-skills, and a prompt that pushes agents to write maximally reusable, task-agnostic skills—is treated as the representative way to do explicit skill maintenance; if a stronger retriever or a prompt that allowed task-specific procedures changed the balance against pure in-context learning, the central conclusion about skill evolution would not generalize beyond this harness.

What would settle it

Run the same five 100-task streams under two modified maintenance regimes—(a) automatic semantic retrieval over the skill library instead of the agent browsing its repository in the prompt, and (b) a maintenance prompt that permits task-specific procedures and notes which task produced each skill—and compare against the same-sequence, no-skill in-context baseline. If either regime makes skill maintenance beat in-context learning by a wide margin across all five domains, the paper's parity finding and its conclusion that agents fail to consolidate experience would be specific to the original ha

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Sequential experience helps: running tasks in the engineered order raises normalized reward in 14 of 15 model–domain combinations, a 16.9% relative aggregate gain over independent execution.
  • Skill libraries are not the driver: on Law, Finance, and Healthcare, an agent that may create and edit skills scores 0.602 normalized reward versus 0.605 for an agent that simply keeps the conversation and feedback.
  • Stored skills still earn their keep where correctness is rigid: explicit skill maintenance beats in-context learning on exact-match and programmatic tasks, doubling Healthcare programmatic reward from 0.250 to 0.500.
  • A retrieval-augmented memory baseline behaves like in-context learning—boosting rubric scores without improving exact-match—which the paper reads as further evidence that open-ended gains come from adapting to prior feedback, not from skill abstraction.
  • Consolidation quality tracks model capability: the stronger model keeps 205 skills with average quality 7.94; the weaker keeps 384 skills at 5.68 and reuses them less, so fragmentation grows as capability falls.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Editorial extension: the parity result is measured under one maintenance regime—a prompt that pushes agents toward maximally reusable, task-agnostic skills and a repository the agent browses before each task. A semantically indexed automatic retriever, or a prompt that permits task-specific procedures, could plausibly move the in-context-versus-skill balance; the paper's conclusion about the limit
  • Editorial extension: the benchmark measures reuse within the 100-task stream, not transfer to genuinely new tasks. A held-out probe—tasks of the same core-skill types that never appear in the sequence—would test whether evolved skills transfer at all, which is the load-bearing sense of 'skill'.
  • Editorial extension: the skill-quality and reuse statistics suggest a testable hypothesis the paper does not run: per-skill quality scores should predict both downstream call frequency and per-task gains; if quality does not predict reuse, fragmentation may be a symptom of weak retrieval rather than weak consolidation.
  • Editorial extension: a natural next experiment is to intervene on consolidation directly—deduplicating, merging, or provenance-tagging skills after each reflection turn—and ask whether those interventions raise exact-match and programmatic rewards beyond the in-context baseline.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper introduces ContinualSkillBench, a benchmark for evaluating whether LLM agents can evolve their capabilities through in-context continual skill learning. It constructs five domain-specific streams of 100 tasks each, ordered by difficulty and cross-task skill dependencies, and evaluates three models (GPT-4o, GPT-5.3-Codex, Claude 4.7 Opus) under independent, sequential, pure in-context learning (ICL), and retrieval-augmented conditions. The headline findings are that sequential execution improves normalized reward in 14 of 15 model-domain combinations, that explicit skill maintenance performs comparably to pure ICL (0.602 vs. 0.605 normalized reward over three domains), and that the weaker model (GPT-4o) accumulates larger, more fragmented skill libraries than GPT-5.3-Codex. The paper concludes that current in-context skill evolution mechanisms support adaptation but struggle to consolidate experience into robust, transferable skills.

Significance. If the results hold, ContinualSkillBench is a valuable evaluation resource: it provides a reproducible, human-reviewed task-ordering methodology, structural validation of skill continuity via permutation tests, and a clear operational separation between raw and normalized rewards. I verified the headline arithmetic in Table 1 and Table 2; the reported counts (14/15, 13/15, 16.9%, 16.2%, and the 0.605 vs. 0.602 parity) are internally consistent. The central empirical claim—that much of the sequential improvement comes from retained context and feedback rather than reusable skill abstraction—is important and falsifiable. However, that claim rests on a specific skill-maintenance protocol, a single model, a single run per condition, and an underspecified normalized-reward definition for the ICL comparison, so the evidence is more conditional than the abstract suggests.

major comments (3)
  1. [§3.4, Table 2] The normalized reward is defined only for the Ind/Seq intersection ('tasks where both the Sequential and Independent settings successfully generate valid output files'). The ICL rows in Table 2 and the aggregate 0.605 vs. 0.602 comparison use this metric, but the exact task subset for ICL is unspecified. If ICL normalized reward is computed over a different valid-output set (e.g., tasks where ICL produced valid files, or the ICL/Seq intersection), the parity result is not a like-for-like comparison. Please state the common task subset used for all three rows, or recompute all normalized rewards on one fixed intersection, and show per-domain counts.
  2. [§4.3, Table 2] The headline 'in-context learning performs comparably to explicit skill maintenance' rests on a single run of a single model (GPT-5.3-Codex) over three domains; the aggregate normalized gap is 0.003. No seed/run variance, confidence intervals, or per-task distributions are reported. At this resolution the claim of parity is not quantitatively established. At a minimum, report multiple runs or reframe as 'in one run we observed no consistent aggregate advantage'; ideally add a second model to the ablation, since the abstract generalizes beyond GPT-5.3-Codex.
  3. [§3.5, Appendix D, Fig. 5] The skill-maintenance protocol is a specific, reusability-biased implementation: the prompt tells the agent to 'Maximize Reusability' and 'Avoid task IDs, temporary names, or absolute paths', and Turn 1 exposes skills as a flat listing with no retrieval or scoring. Appendix H shows that even under this prompt, the most useful-looking generated skills (e.g., 'tatqa-metric-alignment', the Claude medical-advice skill) contain benchmark-specific names and absolute paths, suggesting the protocol both suppresses and imperfectly enforces the intended abstraction. The central conclusion that explicit skill maintenance adds little may be an artifact of this harness; a retriever or a prompt allowing task-specific procedures could change the parity. Please report an ablation with a less constrained maintenance prompt and/or semantic retrieval, or limit the conclusion to 'under this harness'.
minor comments (5)
  1. [Fig. 3] The axis labels contain garbled glyphs: '/glyph1197umber of Self-Generated Skill Calls per Domain' and 'Total /glyph1197umber of Skills' should be 'Number'.
  2. [Appendix D, Turn 3 prompt] Typo: 'Y AML front matter' should be 'YAML front matter'.
  3. [§3.4] Please state explicitly whether raw reward counts tasks with no valid output as zero or excludes them, and specify the denominator for normalized reward when applied to ICL. The current definition only covers Ind vs. Seq.
  4. [§4.4, Appendix G] The claim that 'less capable models tend to accumulate larger, more fragmented collections' is based on two models (GPT-4o and GPT-5.3-Codex). This is a two-point comparison; please phrase it as an observation about these two models, not a general tendency.
  5. [References] The NuminaMath reference is formatted inconsistently as 'Jia LI'; other references use title case. Please normalize capitalization.

Circularity Check

0 steps flagged

No significant circularity: the paper's central claims are empirical, external-reward measurements; no fitted parameter or self-citation chain forces the conclusions.

full rationale

The paper's central claims are grounded in direct experimental comparisons against external evaluators, not in any derivation that reduces to its own inputs. The main Sequential-vs-Independent results (Table 1) use task rewards from exact match, F1, numeric, programmatic, and rubric judges on 100 tasks per domain; the ICL-vs-skills ablation (Section 4.3, Table 2) cleanly separates retained-context/feedback effects from explicit skill maintenance by disabling skill creation in the ICL arm, and the resulting parity (0.605 vs 0.602) is an observed outcome, not a fitted one. The benchmark-construction order is generated from GPT-5.4 pairwise transfer judgments and later checked for skill recurrence using Qwen3-32B annotations and a cosine threshold (Section 3.3, Eq. 1); this is a sanity check of a designed benchmark property, not a prediction derived from a theory, and the paper's conclusions do not depend on this self-validation. There are self-citations (e.g., OneMillionBench cites co-author Jiaqi Li, and the Harbor infrastructure citations may involve overlapping authors), but these are used as external data sources or infrastructure and are not load-bearing for the paper's central claims. The main caveat—that the ICL-vs-skill parity is measured under one specific maintenance prompt and flat-list retrieval—is a generalization/correctness risk, not circularity, because no parameter was fit to the target outcome and the comparison remains fair between the two conditions as implemented. The limitations section is transparent about model and harness coverage, and no uniqueness theorem or ansatz is imported from self-authored prior work to force the choices made.

Axiom & Free-Parameter Ledger

1 free parameters · 5 axioms · 1 invented entities

The paper is a measurement instrument, so its axioms are domain assumptions about the validity of its construction and evaluation, not mathematical postulates. The central ones: LLM transfer judgments define genuine skill transfer (Section 3.2, explicitly called a 'working assumption'); the cosine threshold on mpnet embeddings captures skill equivalence (Section 3.3); curriculum levels reflect real difficulty (Section B.2); official and LLM-judge evaluators measure the same construct across conditions (Section 3.4, Appendix G); and the Harbor three-turn protocol is a representative implementation of explicit skill maintenance (Section 3.5, Appendix D). The only hand-set numeric parameter is the cosine threshold, which is varied in sensitivity analysis.

free parameters (1)
  • cosine similarity threshold tau = 0.85
    Hand-chosen threshold for treating two skill descriptions as semantic counterparts in the structural validation (Section 3.3, Eq. 1). Sensitivity analysis across 0.80/0.85/0.90 shows the recurrence conclusion is stable, so this is an analysis threshold rather than a parameter fitted to produce the headline result.
axioms (5)
  • domain assumption LLM pairwise transfer judgments (GPT-5.4) identify genuine transferable-skill relationships between tasks
    Stated as a 'working assumption' in Section 3.2; the entire curriculum ordering (Sections B.1-B.2) is built on these judgments, and the benchmark's ability to offer reuse opportunities depends on them.
  • domain assumption Cosine similarity >= 0.85 between mpnet-encoded skill descriptions captures semantic skill equivalence
    Used in Section 3.3 and Eq. 1 for the structural validation; the threshold is hand-set and sensitivity-checked (Table 6), but the validity of the validation itself rests on this embedding-cosine match.
  • domain assumption The predefined curriculum levels reflect true task difficulty
    The ordering algorithm (Section B.2) removes edges that point from higher to lower curriculum levels and concatenates levels easiest-first; if the difficulty ranking is wrong, downstream tasks may not build on earlier skills as intended.
  • domain assumption Official source-benchmark evaluators and LLM rubric judges provide valid, cross-condition-comparable scores
    Section 3.4 and Appendix G; rubric scores come from LLM judges and skill-quality scores from GPT-4.1-mini, so the Independent/ICL/Sequential comparisons inherit the judges' consistency and potential biases.
  • domain assumption The Harbor three-turn protocol with create-skill and modify-skill meta-skills is a representative implementation of explicit skill maintenance
    Section 3.5 and Appendix D; the central 'skills do not beat ICL' and 'skills are fragmented' findings are measured under this specific retrieval and prompting regime, including the reusability-maximizing instruction in Figure 5.
invented entities (1)
  • create-skill and modify-skill meta-skills no independent evidence
    purpose: Give the agent explicit operations to add or revise entries in its skill library during the reflection turn
    Software components introduced by this framework; their effectiveness is only measured within the benchmark itself, so they carry no evidence outside the paper.

reviewed 2026-08-05 · how reviews work

0 comments
Cite this review

Pith. "Pith review of ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?." pith.science (2026). https://pith.science/paper/IGVUSGZS

@misc{pith2026260803874,
  author       = {Pith},
  title        = {Pith review of: ContinualSkillBench: Can LLM Agents Truly Evolve Their Capabilities?},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/IGVUSGZS}},
  note         = {Machine review of arXiv:2608.03874}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Modern agent frameworks equip large language models with external skill libraries to solve complex tasks. However, it remains unclear whether these systems can effectively evolve their skills and whether the resulting skills improve task-solving capabilities. To bridge this gap, we introduce ContinualSkillBench, a dynamic evaluation framework for in-context continual skill learning. It covers five representative domains, each containing 100 interconnected subtasks ordered by increasing difficulty and opportunities for cross-task skill reuse. Our experiments show that sequential execution generally improves performance, but the gains vary substantially across models and domains. Moreover, in-context learning performs comparably to explicit skill maintenance on average, suggesting that much of the improvement arises from adaptation to prior context and feedback rather than reusable skill abstraction alone. Explicit skills nevertheless provide selective benefits for tasks requiring reusable procedures or precise outputs. We further find that less capable models tend to accumulate larger, more fragmented collections of task-specific skills. These findings show that current in-context skill evolution mechanisms can support continual adaptation, but still struggle to consistently consolidate experience into robust and transferable skills.

Figures

Figures reproduced from arXiv: 2608.03874 by Haotong Yang, Jiaqi Li, Muhan Zhang, Shirui Liu, Siyuan Cao, Tianyi Guan, Yiding Wang, Yi Hu.

Figure 1
Figure 1. Figure 1: Overview of the CONTINUALSKILLBENCH pipeline. Phase 1 collects approximately 30,000 tasks from five domains. Phase 2 uses LLM-assisted filtering, skill-dependency analysis, task ordering, and human review to construct five sequential task streams. Phase 3 evaluates agents through instruction, execution, and reflection, comparing Sequential and Independent execution with their performance difference reporte… view at source ↗
Figure 2
Figure 2. Figure 2: Taxonomy of CONTINUALSKILLBENCH. The inner ring shows the five domains, the middle summarizes three macro capability tracks within each domain, and the outer lists the source datasets and benchmarks. potential skill-transfer dependencies. This creates opportunities for skills acquired or refined in one task to be reused across different tasks later in the sequence. To support progressive learning, we furth… view at source ↗
Figure 3
Figure 3. Figure 3: Skill-library behavior of GPT-4o and GPT-5.3-Codex across five domains. The left panel shows how [PITH_FULL_IMAGE:figures/full_fig_p008_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Difference in mean core-skill coverage between the curated sequences and random task permutations. [PITH_FULL_IMAGE:figures/full_fig_p016_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Prompt used during the reflection and skill [PITH_FULL_IMAGE:figures/full_fig_p016_5.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

35 extracted references · 28 canonical work pages

  1. [1]

    Prerequisite sub-skills: Does Task A require sub-skills (e.g., reading tables, ratio calculation, parsing text) that also appear in Task B? Or vice versa?

    work page

  2. [2]

    arXiv preprint arXiv:2602.03587

    Cl-bench: A benchmark for context learning. arXiv preprint arXiv:2602.03587. edev2000. 2025. amc12-full dataset. Hugging Face. Accessed: 2026-05-26. Emergent Mind. 2026. Aime 2024 mathematical reason- ing benchmark. Emergent Mind Topics. Accessed: 2026-05-26. Yu Fan, Jingwei Ni, Jakob Merane, Yang Tian, Yoan Hermstrüwer, Yinya Huang, Mubashara Akhtar, Eti...

    arXiv 2025

  3. [3]

    a_helps_b

    Transfer direction: Is the transfer primarily A->B (doing A first helps B), B->A (doing B first helps A), or bidirectional? Required output Respond in this exact JSON format: { "a_helps_b": "YES" | "PARTIAL" | "NO", "b_helps_a": "YES" | "PARTIAL" | "NO", "shared_skills": ["skill1", "skill2"], "reason": "one sentence explaining the main skill relationship"...

    work page

  4. [4]

    Avoid task IDs, temporary names, or absolute paths

    Maximize Reusability: Keep skills modular and broadly applicable. Avoid task IDs, temporary names, or absolute paths. Figure 5: Prompt used during the reflection and skill- maintenance turn. The agent is instructed to create or revise valid skills through theCreate SkillandModify Skillmeta-skills. Codex Sequential Agent Prompt: Run Ori- entation Turn 0: S...

    work page

  5. [5]

    Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu

    Officeqa pro: An enterprise benchmark for end-to-end grounded reasoning.Preprint, arXiv:2603.08655. Sharath Chandra Raparthy, Eric Hambro, Robert Kirk, Mikael Henaff, and Roberta Raileanu. 2023. General- ization to new sequential decision making tasks with in-context learning.Preprint, arXiv:2312.03801. Timo Schick, Jane Dwivedi-Yu, Roberto Dessì, Roberta...

    arXiv 2023

  6. [6]

    Jie Zhang, Cezara Petrui, Kristina Nikoli ´c, and Flo- rian Tramèr

    Claw-eval: Towards trustworthy evaluation of autonomous agents.Preprint, arXiv:2604.06132. Jie Zhang, Cezara Petrui, Kristina Nikoli ´c, and Flo- rian Tramèr. 2025. Realmath: A continuous bench- mark for evaluating language models on research- level mathematics.Preprint, arXiv:2505.12575. Yuxuan Zhang, Yubo Wang, Yipeng Zhu, Penghui Du, Junwen Miao, Xuan ...

    Pith/arXiv arXiv 2025

  7. [7]

    Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Li- han Feng, Yuyi Ao, Leonardo F

    Clawbench: Can ai agents complete everyday online tasks?Preprint, arXiv:2604.08523. Shanshan Zhong, Yi Lu, Jingjie Ning, Yibing Wan, Li- han Feng, Yuyi Ao, Leonardo F. R. Ribeiro, Markus Dreyer, Sean Ammirati, and Chenyan Xiong. 2026. Skilllearnbench: Benchmarking continual learning methods for agent skill generation on real-world tasks.Preprint, arXiv:26...

    Pith/arXiv arXiv 2026

  8. [8]

    B Pairwise Dependency Evaluation and Graph-based Ordering B.1 Pairwise Dependency Evaluation For each domain, we sample 200 unordered pairs from the 100 filtered tasks

    for paragraph-level question answering and information retrieval over complex documents, combined withWritingBench(Wu et al., 2025) to evaluate professional content generation, stylistic refinement, and formatting capabilities. B Pairwise Dependency Evaluation and Graph-based Ordering B.1 Pairwise Dependency Evaluation For each domain, we sample 200 unord...

    work page 2025

  9. [10]

    Shared skill practice: Do both tasks exercise the same underlying skill, meaning doing either one gives useful practice for the other?

    work page

  10. [12]

    Positive values indicate that the curated order places semantically related core skills closer together than expected under random ordering

    Task Introduction.The agent receives the subtask instruction together with its current 15 Local core-skill coverage gain over random order (cosine >= 0.85) Previous 1 task Previous 5 tasks Previous 10 tasks Law +3.1 pp* q = 0.028 +3.9 pp* q = 0.021 +3.9 pp* q = 0.021 Finance +5.6 pp** q = 0.006 +5.1 pp** q = 0.003 +3.9 pp* q = 0.015 Healthcare +1.9 pp* q ...

    work page

  11. [13]

    Its actions and out- puts are recorded for evaluation

    Task Execution.The agent performs the task using the available skills, tools, and external resources when necessary. Its actions and out- puts are recorded for evaluation

    work page

  12. [14]

    It is prompted to diagnose fail- ures and determine whether the experience should be consolidated into a reusable skill

    Reflection and Skill Update.After execu- tion, the agent receives evaluator feedback, such as programmatic test results or rubric- level scores. It is prompted to diagnose fail- ures and determine whether the experience should be consolidated into a reusable skill. The agent may create a new skill or revise an existing one using theCreate SkillandMod- ify...

    work page

  13. [15]

    Read SKILL.md of create-skill or modify-skill and follow every workflow step

    work page

  14. [16]

    Every SKILL.md must begin with YAML front matter: --- name: <skill-name> description: <one-line> --- Otherwise it will not load

    work page

  15. [17]

    Abstract feedback into methods, architectural patterns, or universal domain rules

    Extract Core Skills: Do not memorize this task. Abstract feedback into methods, architectural patterns, or universal domain rules

    work page

  16. [19]

    Read the SKILL.md of create-skill or modify-skill in $HOME/.agents/skills/, and follow its workflow exactly

    work page

  17. [20]

    Skills without the header will not be loaded by the framework

    EverySKILL.md you write or modify must begin with Y AML front matter. Skills without the header will not be loaded by the framework

    work page

  18. [21]

    Abstract feedback into a reusable methodology, architec- tural pattern, or universal domain rule

    Extract core skills: do not memorize the solution or hardcode fixes for this specific task. Abstract feedback into a reusable methodology, architec- tural pattern, or universal domain rule

    work page

  19. [22]

    Avoid hardcoding task IDs, temporary vari- able names, or absolute file paths

    Maximize reusability: keep the documented skill modular and broadly applicable to future unseen tasks. Avoid hardcoding task IDs, temporary vari- able names, or absolute file paths. E Example Task Format and Skill-Dependency Chain To make the benchmark format more concrete, we show an example chain from theFinancedomain. We omit irrelevant tables, filings...

    work page arXiv 2018

  20. [23]

    Read the provided financial data files

    work page

  21. [24]

    Calculate the required financial components

    work page

  22. [25]

    Compute the weights for equity and debt in the capital structure

    work page

  23. [26]

    Calculate the final W ACC using the given formula

    work page

  24. [27]

    This chain illustrates how reusable skills can transfer across subtasks

    Handle edge cases. This chain illustrates how reusable skills can transfer across subtasks. Early tasks such as Task 1 and Task 7 require locating values in financial tables and matching rows, columns, periods, and units. Later tasks reuse these operations as subrou- tines for more complex reasoning: Task 35 adds percentage-growth calculation, Task 50 add...

    work page 2020

  25. [28]

    - Requested periods and order (for example:`2018 and 2019 respectively`means output`2018 | 2019`)

    Parse the question target exactly: - Metric noun phrase (for example:`cash and cash equivalents`,`total liquidity`,`current assets`). - Requested periods and order (for example:`2018 and 2019 respectively`means output`2018 | 2019`). - Required granularity (`exact table amount`vs` approximately`phrasing)

    work page 2018

  26. [29]

    - Table subtotal/total rows

    Build candidate values from both sources: - Table row values (line items). - Table subtotal/total rows. - Narrative context values (especially management summary sentences)

    work page

  27. [30]

    - If the question wording is broad (`had`,`liquidity`, `available`) and context explicitly states a combined figure, prefer the context-stated combined amount

    Resolve source selection with this priority: - If the question asks for a line item and only the table has that line item, use the line item. - If the question wording is broad (`had`,`liquidity`, `available`) and context explicitly states a combined figure, prefer the context-stated combined amount. - If context restates table values at different precisi...

    work page

  28. [31]

    - Composition check: line item vs subtotal vs combined categories

    Validate before finalizing: - Unit check: thousands vs millions. - Composition check: line item vs subtotal vs combined categories. - Time-order check: ensure output order matches` respectively`order. ## Common Pitfalls - Returning a component (`cash & cash equivalents`) when the prompt/judge expects a combined liquidity figure (`cash + short-term investm...

    work page

  29. [32]

    Identify task number; determine output path

    work page

  30. [33]

    Translate the prompt mentally if non-English

    work page

  31. [34]

    Draft a structured answer with: -`key_findings`: bullet list of medically relevant facts -`country_specific`(when relevant): differences across major countries (US/CDC, France/HAS, UK/NHS, Germany/STIKO, WHO) -`recommendation`: clear, safe clinical guidance -`citations`: list of authoritative sources (CDC, WHO , HAS, NHS, NICE, UpToDate, peer-reviewed) -`...

    work page

  32. [35]

    key_findings

    WRITE THE FILE. ## Common Topics & Quick References ### Pertussis Booster (whooping cough / coqueluche) - Adults: One Tdap dose; then Td/Tdap booster every 10 years - Pregnancy: Tdap each pregnancy (27-36 weeks US, 16-32 UK, 20-36 France) - Cocoon strategy: parents, grandparents, caregivers of infants <12 mo - Healthcare workers: one dose if not received ...

    work page 2024

  33. [2017]

    Overcoming catastrophic forgetting in neural networks.Proceedings of the National Academy of Sciences, 114(13):3521–3526. Haohang Li, Yupeng Cao, Yangyang Yu, Shashid- har Reddy Javaji, Zhiyang Deng, Yueru He, Yuechen Jiang, Zining Zhu, Koduvayur Subbalakshmi, Guojun Xiong, Jimin Huang, Lingfei Qian, Xueqing Peng, Qianqian Xie, and Jordan W. Suchow. 2024....

    Pith/arXiv arXiv 2024

  34. [2023]

    Preprint, arXiv:2311.12983

    Gaia: a benchmark for general ai assistants. Preprint, arXiv:2311.12983. OpenAI, :, Aaron Hurst, Adam Lerer, Adam P. Goucher, Adam Perelman, Aditya Ramesh, Aidan Clark, AJ Ostrow, Akila Welihinda, Alan Hayes, Alec Radford, Aleksander M ˛ adry, Alex Baker-Whitcomb, Alex Beutel, Alex Borzunov, Alex Carney, Alex Chow, Alex Kirillov, and 401 others. 2024. Gpt...

    Pith/arXiv arXiv 2024

  35. [2026]

    Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, and 1 others

    Skillcraft: Can llm agents learn to use tools skillfully?arXiv preprint arXiv:2603.00718. Shihan Dou, Ming Zhang, Zhangyue Yin, Chenhao Huang, Yujiong Shen, Junzhe Wang, Jiayi Chen, Yuchen Ni, Junjie Ye, Cheng Zhang, and 1 others

    arXiv

This paper was first reviewed by deepseek-v4-flash on August 5, 2026.