The reviewed record of science sign in
Pith

arxiv: 2607.05202 · v1 · pith:QOBYKAH6 · submitted 2026-07-06 · cs.AI

EvoAgentBench: Benchmarking Agent Self-Evolution via Ability Transfer

Xingze Gao , Chuanrui Hu , Hongda Chen , Pengfei Yao , Zhao Wang , Yi Bai , Zhengwei Wu , Yunyun Han
show 4 more authors
Xiaofeng Cong Jie Gui Yafeng Deng Teng Li
This is my paper

Reviewed by Pith2026-07-07 23:39 UTCglm-5.2pith:QOBYKAH6open to challenge →

classification cs.AI
keywords agentevoagentbenchabilityself-evolutionacrossproceduraltesttransfer
0
0 comments X

The pith

Procedural skills transfer across AI agents — if you route them right

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces EvoAgentBench, a benchmark designed to isolate a specific question: can an AI agent extract reusable procedures from past task executions and successfully apply them to new tasks? The central structural object is the Ability Graph — a network where nodes are tasks and edges represent shared reusable procedures (called Abilities) extracted from agent execution traces. The benchmark guarantees that every test task shares at least one Ability with a training task, so any improvement from experience cannot be attributed to memorization or near-duplicate exposure. The authors find that when procedural content is curated by the benchmark pipeline and delivered with correct routing (a diagnostic condition they call Anchor Skill), it produces positive performance gains across all 24 tested combinations of method, domain, and model — even when the models that generated the procedures are completely different from the models being tested. However, when current automatic self-evolution methods must extract and route this content on their own, every method fails in at least one setting, producing negative transfer. The paper's conclusion is that the procedural knowledge itself is transferable; the bottleneck lies in how methods extract, index, and deliver that knowledge at test time.

Core claim

The paper establishes two findings that together form its core. First, reusable procedural content — strategies for searching, debugging, and verification distilled from agent traces — transfers reliably across model families when correctly identified and routed, producing positive gains in all 24 method-domain-setting cells tested. Second, no current automatic self-evolution method (Memento, ReasoningBank, GEPA) sustains positive transfer across all settings; each exhibits negative transfer in at least one cell. Because all methods operate under identical tasks, tools, scoring, and agent configuration, the gap between the diagnostic reference and automatic methods isolates the failure to a

What carries the argument

The Ability Graph is the central object. Nodes are tasks; edges connect tasks that share at least one edge-eligible canonical Ability — a reusable operation extracted from agent execution traces, canonicalized through a multi-stage process of embedding-based blocking, three-LLM adjudication, and expert review. The benchmark enforces an ability-supported yet instance-disjoint split: every test task must share a procedural Ability with at least one training task, but no test task instance appears in training. This design separates genuine procedural transfer from memorization.

If this is right

  • If procedural content transfers across model families when correctly routed, the path to better self-evolving agents lies in improving extraction and routing pipelines rather than in scaling model size — the paper shows scale alone does not predict transfer.
  • The benchmark's diagnostic reference (Anchor Skill) provides an upper bound on transferability for any given task set, giving method developers a concrete target: close the gap between automatic extraction and curator-grade extraction.
  • The finding that cost overhead is decoupled from accuracy direction means the field should report both dimensions jointly; a method that improves accuracy but doubles turn count may be impractical for deployment.
  • The Ability Graph structure could be extended to measure transfer between domains (not just within them), testing whether a debugging procedure from software engineering helps with algorithmic reasoning.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper tests only text-based agentic domains; if the Ability extraction pipeline generalizes to multimodal or embodied agents, the same ability-supported/instance-disjoint split principle could diagnose procedural transfer in robotics or vision-language tasks.
  • The Anchor Skill condition uses curator-side Ability labels for routing, which no automatic method has access to; this suggests that a method combining automatic extraction with learned or structured routing policies might close much of the gap, rather than requiring full curator intervention.
  • The fact that GEPA (a single evolved prompt broadcast to all tasks) nearly matches Anchor on one backbone but fails on others suggests that the value of per-task routing vs. global strategies depends on the interaction between model family and domain — a testable hypothesis for future method design.

Load-bearing premise

The entire benchmark depends on the Ability extraction pipeline — a single LLM extracting raw procedural cards from traces, followed by three other LLMs merging them into canonical units — producing operationally meaningful and genuinely transferable procedures. If the extracted Abilities are superficial descriptions rather than real reusable operations, the benchmark measures label-sharing rather than skill transfer.

What would settle it

Show that an automatic self-evolution method with improved extraction and routing achieves uniformly positive transfer across all cells, which would confirm the paper's claim that the bottleneck is method-side rather than content-side.

Figures

Figures reproduced from arXiv: 2607.05202 by Chuanrui Hu, Hongda Chen, Jie Gui, Pengfei Yao, Teng Li, Xiaofeng Cong, Xingze Gao, Yafeng Deng, Yi Bai, Yunyun Han, Zhao Wang, Zhengwei Wu.

Figure 1
Figure 1. Figure 1: EvoAgentBench construction pipeline. Stage 1: no-skill executions from multiple backbones across four domains yield a trace pool with reusable spans. Stage 2: raw Ability cards are merged into canonical families (Method, Guard, or Workflow) via embedding blocking, LLM adjudication, and expert review. Stage 3: the task–Ability graph is partitioned into communities, and constrained splitting guarantees train… view at source ↗
Figure 2
Figure 2. Figure 2: Ability extraction example from the algorithmic reasoning domain. Top: source task and baseline failure. [PITH_FULL_IMAGE:figures/full_fig_p015_2.png] view at source ↗
read the original abstract

Agent self-evolution in long-horizon LLM systems is largely procedural: useful experience is not merely stored information, but reusable procedures for searching, debugging, and verification. Yet current evaluations do not isolate this form of transfer. Agent benchmarks test single-episode task solving; memory benchmarks target information retention rather than procedural reuse. We introduce EvoAgentBench, a benchmark for agent self-evolution via Ability-guided transfer across four agentic domains: web research, algorithmic reasoning, software engineering, and knowledge work. EvoAgentBench extracts trace-grounded Abilities from agent executions, canonicalizes them into operational units, and builds domain-specific Ability Graphs linking tasks that share procedural overlap. By design, every test task is backed by verified training-side Ability support. Across a 528/267 train/test split, two scaffolds, and three backbones, curated Ability content transfers reliably across model families, but no current automatic method sustains positive gain in all settings. EvoAgentBench shifts self-evolution evaluation from aggregate accuracy comparison to fine-grained diagnosis of experience encoding, routing, and uptake. The benchmark is publicly available at https://huggingface.co/datasets/EverMind-AI/EvoAgentBench.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

3 major / 6 minor

Summary. The paper introduces EvoAgentBench, a benchmark for evaluating agent self-evolution via procedural ability transfer across four agentic domains (web research, algorithmic reasoning, software engineering, knowledge work). The benchmark's core structural principle is that every test task must be

Significance. The paper addresses a genuine gap in the evaluation of self-evolving agents by isolating procedural transfer from aggregate task accuracy. The construction pipeline is carefully designed: construction backbones are disjoint from evaluation backbones, information access is rigorously controlled (Table 8), and the benchmark is publicly available. The finding that curated ability content transfers across model families while automatic methods remain brittle is a substantive empirical contribution. The cost-accuracy decoupling analysis (Table 4) is a valuable diagnostic.

major comments (3)
  1. §4.2, Table 3: The central claim that 'curated Ability content transfers reliably across model families' is stated as positive in all 24 method-domain-setting cells, but several cells have gains within noise. For example, Anchor on Gemma-4-31B/OpenClaw/Algo shows +0.4 with SE ±4.6, and on Gemma-4-31B/Nanobot/SWE shows +9.5 with SE ±4.8. While the contrast with automatic methods' clearly negative cells (e.g., Memento's −36.3) remains meaningful, describing these small-SE-overlapping gains as 'reliable' transfer overstates the precision of the evidence. The authors should qualify the 'reliably' language or report whether the gains are statistically significant cell-by-cell, not just positive in sign.
  2. §4.2 and §3.2.2–3.2.3: The diagnostic contrast between Anchor Skill and automatic methods confounds two factors that Anchor has but automatic methods lack: (1) curator-side extraction quality and (2) curator-side Ability-label routing. The paper attributes the gap to 'method-side extraction and routing' (§4.2), but without an ablation isolating routing from content quality, the diagnostic is incomplete. A generic-advice ablation — where Anchor's routing is preserved but skill content is replaced with generic domain guidance — would clarify whether the gains stem from operationally specific procedural content or from the benefit of injecting any well-matched contextual guidance with perfect routing. The paper acknowledges extraction quality as a limitation but does not address this confound, which is load-bearing for the claim that 'Ability content transfers.'
  3. §3.2.2–3.2.3 and Appendix A.3: The entire Ability extraction and canonicalization pipeline depends on a single LLM (Claude Sonnet 4.6) for raw card extraction, with three LLM adjudicators (Kimi-K2.5, GLM-5.1, DeepSeek-V3.2) for merging. The paper provides one illustrative example of an extracted Ability (Figure 2: linear-time substring matching) but offers no systematic validation that the other 169 canonical Ability families are comparably operationally specific. If many Abilities are broad directives rather than specific procedures, the 'guaranteed train-side Ability support' could be vacuous — tasks share a label but not a real transferable procedure. A quantitative or sampled qualitative audit of Ability specificity across families would substantially strengthen the benchmark's validity.
minor comments (6)
  1. Table 3: The formatting of the delta values is inconsistent and sometimes difficult to parse. For example, '−36.3' appears without a preceding '+' or '−' sign convention explanation in the caption for negative values, and some cells like '+0.0' for RB on Gemma/OpenClaw/Web could be clearer. Consider standardizing sign presentation.
  2. §3.2.5: The 'no-evolution baseline reward' is used as a 'soft headroom signal' for split construction. This is listed as a free parameter in the axiom ledger. The paper should briefly discuss whether this headroom-based sampling introduces selection bias that could inflate or deflate transfer gains, beyond the caveat already noted in Limitations.
  3. Figure 2: The example Ability is from algorithmic reasoning, which is the domain where abilities are most naturally procedural. Including a second example from web research or knowledge work would demonstrate that the extraction pipeline produces operationally specific abilities in domains where 'procedures' are less clearly defined.
  4. Table 6: The Web Research subgraph has density 0.404 and average degree 230.9, which is extremely dense. The paper should comment on whether this density reflects genuine procedural overlap or potential over-merging of broad search/verification abilities in this domain.
  5. References: Several cited works have 2026 dates (e.g., Jiang et al. 2026, Li et al. 2026, Zhong et al. 2026). If these are not yet published at the time of review, they should be marked as 'to appear' or 'forthcoming' as appropriate.
  6. Appendix A.4: The embedding similarity thresholds (θ=0.85 and θ=0.82) are domain-specific but the rationale for these particular values is not provided. A brief justification would help reproducibility.

Simulated Author's Rebuttal

3 responses · 0 unresolved

We thank the referee for the careful and constructive review. The referee raises three substantive points: (1) the 'reliably' language overstates precision given that some Anchor cells have gains within noise, (2) the Anchor-vs-automatic-methods diagnostic confounds extraction quality with routing quality, and (3) the Ability extraction pipeline lacks systematic validation of operational specificity across all 170 canonical families. We agree with all three points and will revise accordingly.

read point-by-point responses

  1. Referee: §4.2, Table 3: The central claim that 'curated Ability content transfers reliably across model families' is stated as positive in all 24 method-domain-setting cells, but several cells have gains within noise. The authors should qualify the 'reliably' language or report whether the gains are statistically significant cell-by-cell.

    Authors: The referee is correct. Several Anchor cells have gains whose standard errors overlap zero — for example, +0.4 (SE ±4.6) on Gemma-4-31B/OpenClaw/Algo and +3.6 (SE ±5.9) on Qwen3.5-27B/OpenClaw/KW. Describing these as 'reliable' overstates the precision of the evidence. We will make two changes. First, we will qualify the language: rather than stating that Ability content 'transfers reliably,' we will say that Anchor produces positive point estimates in all 24 cells and is the only condition with no negative cells, while noting that individual cell-level gains are not all statistically significant. Second, we will add a supplementary table reporting per-cell significance (paired t-test or bootstrap CI as appropriate) so readers can distinguish cells with robust gains from those within noise. The core contrast with automatic methods — which have clearly negative cells such as Memento's −36.3 — survives this qualification, but the precision of the claim will be honestly represented. revision: yes

  2. Referee: §4.2 and §3.2.2–3.2.3: The diagnostic contrast between Anchor Skill and automatic methods confounds curator-side extraction quality and curator-side Ability-label routing. A generic-advice ablation would clarify whether gains stem from operationally specific procedural content or from any well-matched contextual guidance with perfect routing.

    Authors: This is a fair and important point. The current manuscript attributes the Anchor–automatic gap to 'method-side extraction and routing' but does not isolate the two factors. We agree that a generic-advice ablation is the right diagnostic. We will conduct an ablation where Anchor's curator-side routing is preserved but skill content is replaced with generic domain guidance (e.g., 'search systematically and verify results' for web research, 'consider edge cases and test on large inputs' for algorithmic reasoning). If generic advice with perfect routing performs comparably to Anchor, the transfer claim would be substantially weakened; if it performs worse, the result supports the conclusion that operationally specific procedural content drives the gains. We will report this ablation in the revised manuscript and adjust the framing of the Anchor–automatic gap to explicitly acknowledge the extraction-quality vs. routing confound pending the ablation results. If the ablation cannot be completed within the revision timeline, we will at minimum add an explicit limitation noting this confound and soften the causal attribution accordingly. revision: partial

  3. Referee: §3.2.2–3.2.3 and Appendix A.3: The entire Ability extraction and canonicalization pipeline depends on a single LLM for raw card extraction, with no systematic validation that the other 169 canonical Ability families are comparably operationally specific. A quantitative or sampled qualitative audit of Ability specificity across families would substantially strengthen the benchmark's validity.

    Authors: The referee is right that a single illustrative example (Figure 2) is insufficient to validate operational specificity across all 170 canonical families. We will address this by conducting a sampled audit: we will draw a stratified random sample of canonical Ability families (stratified by domain and role type — Method, Guard, Workflow), have domain experts rate each on a specificity scale (e.g., 'operationally specific procedure' vs. 'broad directive' vs. 'topic label'), and report the distribution. We will also report inter-annotator agreement. This audit will either confirm that the canonicalization pipeline's operational-equivalence rubric produces specific procedures at scale, or reveal families that should be downgraded to annotation-only status. We will add the audit results to the appendix and reference them in §3.2.3. We note that the pipeline already contains safeguards against generic content — the edge-eligibility check excludes units whose procedures become too generic after merging, and the extraction rubric requires actionable operations rather than topic labels — but these safeguards have not been systematically validated, which is the gap the referee correctly identifies. revision: yes

Circularity Check

0 steps flagged

No significant circularity found; the benchmark's positive-transfer claim is empirical, not definitional.

full rationale

The paper's central claim — that curated Ability content transfers across model families while automatic methods remain brittle — is an empirical finding, not a result forced by construction. The Ability Graph guarantees that every test task shares at least one edge-eligible Ability with a training task (§3.2.5), but this guarantees only the existence of shared procedural structure, not that injecting it will improve performance. The positive Δ values reported in Table 3 are measured outcomes. Anchor Skill's content is constructed exclusively from train-side raw cards (§4.1: 'skill content is constructed exclusively from train-side evidence—canonical family text and skill text derive only from raw cards extracted on training tasks, and raw cards from test tasks contribute to neither'), and construction backbones (Kimi-K2.5, GLM-5.1, DeepSeek-V3.2) are disjoint from evaluation backbones (Qwen3.5, Gemma-4). The test-side Ability labels are used only for routing (which skill to retrieve), not for skill content. The paper is transparent that Anchor is a diagnostic reference, not a deployable method. No self-citation chain is load-bearing: the references are to external work (Reflexion, ExpeL, Voyager, Memento, etc.), and no uniqueness theorem or prior-work ansatz is invoked to force the conclusion. The skeptic's concern that Anchor's gains may confound content quality with routing quality is a validity threat, not circularity — the paper does not define its prediction in terms of its inputs. The use of no-evolution baseline scores as a 'soft headroom signal' in split construction (§3.2.5) introduces sampling bias, acknowledged in Limitations, but all methods share the identical split so comparisons remain fair. Score 1 reflects the minor overlap that the Ability Graph is built from traces of all tasks (including future test tasks), but this does not create a definitional reduction of the claimed result to its inputs.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 2 invented entities

The benchmark introduces two invented entities (Ability, Ability Graph) with falsifiable handles. The free parameters are threshold choices and selection signals, not fitted constants. The axioms are domain assumptions about LLM extraction and adjudication quality that the paper does not independently validate but that are partially supported by downstream Anchor Skill results.

free parameters (2)
  • Embedding similarity threshold θ_d = 0.85 (BrowseComp, GDPVal), 0.82 (SWE-Bench, LiveCodeBench)
    Domain-specific thresholds for embedding blocking in canonicalization (§A.4). Chosen by domain, not derived from theory.
  • No-evolution baseline reward (headroom signal) = per-task
    Used as a soft signal in split construction to prefer tasks with room for improvement (§3.2.5, §A.6). Affects which tasks enter the test split.
axioms (4)
  • domain assumption Trace-grounded Abilities extracted by an LLM represent operationally meaningful and transferable procedural units.
    The entire benchmark depends on this. If extracted Abilities are superficial labels, the 'guaranteed support' is vacuous. Invoked in §3.2.2.
  • domain assumption LLM adjudicators can reliably judge operational equivalence of Ability cards.
    Three LLM adjudicators merge raw cards into canonical families (§3.2.3). If adjudication is unreliable, the Ability Graph structure is noise.
  • domain assumption Louvain community detection on the Ability Graph produces meaningful procedural neighborhoods.
    Communities localize train-side support during split construction (§3.2.4). If communities are arbitrary, the support constraint is weaker than claimed.
  • domain assumption Evaluation backbones have not memorized source benchmark answers during pretraining.
    The paper acknowledges this risk in Limitations but argues it shifts absolute scores, not comparisons. Still, it is an unverified assumption.
invented entities (2)
  • Ability (canonical unit) independent evidence
    purpose: A reusable procedural operation extracted from agent traces, used as the transfer unit in the Ability Graph.
    The paper provides a concrete example (Figure 2) and validates transferability via Anchor Skill results. The concept is falsifiable: if Abilities were meaningless, Anchor would not produce uniform positive gains.
  • Ability Graph independent evidence
    purpose: A graph linking tasks by shared Abilities, used to construct ability-aware train/test splits.
    The graph structure is empirically validated by the fact that Anchor Skill (which uses graph structure for routing) produces positive transfer in all 24 cells.

pith-pipeline@v1.1.0-glm · 21164 in / 2720 out tokens · 184227 ms · 2026-07-07T23:39:46.229294+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

17 extracted references · 17 canonical work pages · 5 internal anchors

  1. [1]

    Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, and 1 others

    Building self-evolving agents via experience- driven lifelong learning: A framework and bench- mark.arXiv preprint arXiv:2508.19005. Zijian Chen, Xueguang Ma, Shengyao Zhuang, Ping Nie, Kai Zou, Andrew Liu, Joshua Green, Kshama Patel, Ruoxi Meng, Mingyi Su, and 1 others. 2025. BrowseComp-Plus: A more fair and transparent eval- uation benchmark of deep-res...

    work page arXiv 2025

  2. [2]

    GLM-5: from Vibe Coding to Agentic Engineering

    Glm-5: from vibe coding to agentic engineer- ing.Preprint, arXiv:2602.15763. Google DeepMind. 2026. Gemma 4 31B IT model card. Naman Jain, King Han, Alex Gu, Wen-Ding Li, Fanjia Yan, Tianjun Zhang, Sida Wang, Armando Solar- Lezama, Koushik Sen, and Ion Stoica. 2024. Live- CodeBench: Holistic and contamination free eval- uation of large language models for...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  3. [3]

    SEA-Eval: A Benchmark for Evaluating Self-Evolving Agents Beyond Episodic Assessment

    SEA-Eval: A benchmark for evaluating self- evolving agents beyond episodic assessment.arXiv preprint arXiv:2604.08988. Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can language mod- els resolve real-world GitHub issues? InInter- national Conference on Learning Representations (...

    work page internal anchor Pith review Pith/arXiv arXiv 2024

  4. [4]

    Skill-Pro: Learning Reusable Skills from Experience via Non-Parametric PPO for LLM Agents

    Evaluating very long-term conversational memory of LLM agents. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 13851– 13870. Qirui Mi, Zhijian Ma, Mengyue Yang, Haoxuan Li, Yisen Wang, Haifeng Zhang, and Jun Wang. 2026. Skill-Pro: Learning reusable skills from experience via non-para...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  5. [5]

    Trace2Skill: Distill Trajectory-Local Lessons into Transferable Agent Skills

    Trace2Skill: Distill trajectory-local lessons into transferable agent skills.arXiv preprint arXiv:2603.25158. OpenClaw Contributors. 2026. OpenClaw: Personal AI assistant. https://github.com/openclaw/ openclaw. GitHub repository. Siru Ouyang, Jun Yan, Yanfei Chen, Rujun Han, Zifeng Wang, Bhavana Dalvi Mishra, Rui Meng, Chun-Liang Li, Yizhu Jiao, Kaiwen Zh...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  6. [6]

    Dynamic Cheatsheet: Test-Time Learning with Adaptive Memory

    Reflexion: Language agents with verbal rein- forcement learning. InAdvances in Neural Informa- tion Processing Systems (NeurIPS). Mirac Suzgun, Mert Yuksekgonul, Federico Bianchi, Dan Jurafsky, and James Zou. 2025. Dynamic cheat- sheet: Test-time learning with adaptive memory. arXiv preprint arXiv:2504.07952. Kimi Team, Tongtong Bai, Yifan Bai, Yiping Bao...

    work page internal anchor Pith review Pith/arXiv arXiv 2025

  7. [7]

    All non-unanimous pairs are reviewed by a domain expert under the same rubric (see Appendix C)

    are accepted automatically. All non-unanimous pairs are reviewed by a domain expert under the same rubric (see Appendix C). Group consistency and canonical families.Ac- cepted merge links define a compatibility graph over raw cards. Connected components are con- verted into canonical Ability families only after a group-level consistency check: all interna...

    work page

  8. [8]

    Tasks with no retained family are excluded from the split

    Filter eligible tasks.Retain only tasks with at least one canonical Ability family. Tasks with no retained family are excluded from the split

    work page

  9. [9]

    Secondary objectives in- clude preserving Ability-family coverage and avoiding test-only families

    Rank candidate test tasks.Prefer tasks with lower or medium baseline reward (more room for improvement). Secondary objectives in- clude preserving Ability-family coverage and avoiding test-only families

    work page

  10. [10]

    A task t enters the test split only if at least one family in F(t) retains train-side support: ∀t∈ Dtest,∃f∈F(t) such that |{t′ ∈D train : f∈F(t ′)}|>0

    Select test tasks under support constraints. A task t enters the test split only if at least one family in F(t) retains train-side support: ∀t∈ Dtest,∃f∈F(t) such that |{t′ ∈D train : f∈F(t ′)}|>0. Domain Test Supp@10 Zero Web Research 65 10.00 0 Algo. Reasoning 86 7.45 0 Software Eng. 56 6.82 0 Knowledge Work 60 9.25 0 Total267 8.34 0 Table 7: Train-side...

    work page

  11. [11]

    Validate family-level support.Every Ability family that appears in test must also appear in train

    work page

  12. [12]

    Select skill-train evidence.For each test- relevant family, select train-side evidence tasks for skill construction, prioritizing failure evidence, success contrast, informative traces, and task diversity within the family. Guarantees.The resulting split satisfies three invariants: (1) every test task shares at least one Ability family with training tasks...

    work page

  13. [13]

    Same role type (Method, Guard, or Work- flow)

    work page

  14. [14]

    Compatible trigger conditions

    work page

  15. [15]

    Equivalent reusable procedures (same mecha- nism, not merely same topic)

    work page

  16. [16]

    Same success mechanism or correction target

    work page

  17. [17]

    search”, “debug

    Compatible applicability boundaries. Shared topic, lexical overlap, or generic verbs (“search”, “debug”, “validate”) are explicitly in- sufficient for a merge. D Ability Examples We illustrate the Ability concept with a concrete example from the algorithmic reasoning domain. Figure 2 shows the full extraction flow: from the source task and baseline failur...

    work page 2025