Artificial Intelligence · arXiv:2608.20614 v1 · submitted 20 Aug 2026 · 38 references
Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills
The paper says
“Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose.”
more of the abstract
Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman ρ = 0.14). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency—signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 27 Aug 2026 · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A genuinely useful paired-evaluation framework for agent skills, but the headline composite Skill Lift overstates marginal value because process metrics and refined ground truth are derived from the skill itself. This line is wrong
4 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the 38 references say.
The paper says, at §4.1, §4.3, §4.7, Table 8, Appendix A, §6.4
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The composite Skill Lift headline is inflated by process metrics that the baseline cannot satisfy by construction. Section 4.1 allows expected_behavior entries such as “read git-skill/SKILL.md before executing”; in the baseline condition the target skill is withheld, so that behavior is unsatisfiable regardless of agent competence.
more of objection 1
The skill_execution sub-checks (activation, script_execution, workflow_order) in §4.3 and behavior_check then measure the presence of the intervention rather than its marginal value, and Table 8 weights each of these equally (1/6) in the composite. Section 6.4 reports process gains of +0.3263, +0.2983, and +0.2758 against accuracy +0.1431, and the abstract and §6.4 headline the composite 0.2134 even though Appendix A concedes that process metrics “may be unable to satisfy when the target skill is withheld.” Because the central empirical claim is the composite lift, this is load-bearing: the authors should either make outcome-only lift (0.1799) the headline, or re-author expected_behavior and process checks blinded to condition and show that the process gains survive.
The paper says, at §4.2, §4.3, §6.4
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The trajectory-grounded refinement step in §4.2 replaces ground_truth with the agent's final answer when it “aligns with author intent”; as written, this makes accuracy self-referential unless the alignment check is an independent, documented human judgment. The accuracy judge in §4.3 compares the final response to ground_truth, so if the reference is derived from the same with-skill trajectory, the accuracy lift of 0.1431 measures self-consistency rather than correctness.
more of objection 2
This also contaminates the outcome-only lift of 0.1799 because accuracy is one of its two components. The paper should report how many ground_truth entries were replaced, specify who performed the alignment check, and show that accuracy conclusions survive when the original references are restored.
The paper says, at §6.2, §6.6, Appendix B
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The frozen 947-case headline excludes condition-run failures, timeouts, and unscored cases unless both condition scores are present, and the reconstruction audit in Appendix B covers only 898 of the 947 pairs. Section 6.6 itself gives an example where the with-skill run produced no scored trials while the baseline completed, so exclusion can plausibly bias lift upward.
more of objection 3
The conservative imputation in Appendix B is helpful but does not apply to all 947 frozen pairs. The authors should report a worst-case bound for the full 947 cases, treating missing with-skill scores as failures and missing baseline scores as successes, or otherwise bound the selection bias.
The paper says, at §7, Table 8
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
Because behavior_check and accuracy are LLM-judged and §7 reports no live inter-judge agreement, human calibration, or judge-uncertainty propagation, the 95% confidence interval in §6.4 should be labeled as covering only case-level sampling variation, not judge noise. The authors should either add a judge-agreement study on a random subsample of trajectories or soften the precision claims attached to the headline lift.
5 minor objections
The paper says, at Abstract
Pith says: minor objection 1 of 5
The abstract contains a typo: “reportsSkill Lift” should read “reports Skill Lift.”
The paper says, at Figure 8
Pith says: minor objection 2 of 5
Figure 8 labels the overall lift as “73%+” while §6.4 states composite lift is positive in 689/947 = 72.8% of paired cases; the figure should use the same rounding convention as the text.
show 3 more
The paper says, at §4.7
Pith says: minor objection 3 of 5
The display equation defining Skill Lift is unnumbered; number it so that the outcome-only formula in Appendix A and later references can be cross-referenced precisely.
The paper says, at Table 7, Appendix A
Pith says: minor objection 4 of 5
Table 7 says the security backfill “found secret-like strings, destructive command patterns, unauthorized-path access” across live artifacts; clarify whether these are confirmed findings or pattern matches, since Appendix A notes that the security metric has pattern false positives.
The paper says, at §6.1, §6.2
Pith says: minor objection 5 of 5
The relationship between the 145 scanned skills, 89 live-evaluated variants, and 64 production skills is stated only in §6.2; stating this filtering chain earlier would help readers connect the abstract's corpus claims to the live subset.
What would settle it
On a random sample of production skills, retrieve the frozen evals.json and classify each expected_behavior entry as (a) author-written before any live run, (b) lifted via --refine from a with-skill ATIF trajectory, or (c) generated from SKILL.md; classify each ground_truth as an original author reference or a refined with-skill final answer. Re-score the paired trajectories three ways: (1) using only category-(a) assets; (2) using assets re-authored by a reviewer blinded to all with-skill trajectories and condition labels; (3) using human-written reference answers independent of agent outputs. Compare composite, behavior_check, and skill_execution lift across these three rescorings.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 27 Aug 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
The thing to know about arXiv:2608.20614 is that the framework it describes is real and useful, but the headline number overstates the marginal value of skills. The composite Skill Lift of 0.2134 is partly predetermined: three of the six components—skill_execution, behavior_check, skill_efficiency—include expectations like "read the target skill's SKILL.md before executing," which the baseline condition cannot satisfy because the skill is withheld. The refine step in §4.2 makes it worse by replacing ground_truth with the agent's own final answer and lifting observed tool calls into expected_behavior. That makes the scoring reference self-referential. To their credit, the authors disclose this in Appendix A and report outcome-only lift (0.1799), which is more honest. But the abstract and §6.4 still lead with the composite, so a reader skimming the paper gets an inflated number.
What is actually new: the combination of paired with-skill/baseline trials, an ATIF trajectory contract across harnesses, author-owned evals, and CI-native reporting. I don't know a prior tool that covers all five capabilities listed in Table 3. The empirical observation that static scans and LLM-judge rubrics correlate at only rho=0.14, and that neither predicts live lift (rho near zero), is a genuinely useful result for anyone building skill gates. The evidence section is unusually transparent: they report confidence intervals, cluster checks, missingness audits, and even show what happens under conservative imputation. The open-source SkillEvaluator implementation is a concrete artifact the community can use.
The soft spots beyond the metric issue: the dataset is internal NVIDIA, so the headline numbers are not independently reproducible. The anonymized skill set helps but does not let anyone verify the distribution. The refinement loop is described as closing the gap between hypothesized and observed behavior, but without an independent human audit it can turn the eval into a tautology.
Proportionately, the paper is a serious engineering contribution with a fixable measurement problem. The framework is sound; the composite metric needs to either exclude process components that are trivially unsatisfiable at baseline or be re-validated blind. The outcome-only result, and the qualitative findings about routing and workflow, survive the critique.
I would send this to peer review. A good referee will ask for exactly the re-authoring audit I mentioned; that's manageable. I'd cite it for the paired-protocol design and the static-vs-live correlation result, and bring it to a reading group if anyone cares about agent evaluation infrastructure.
The strongest objection
The central empirical claim—that adding a skill raises composite task performance by 0.2134—depends on ACES's scoring assets being a neutral measuring instrument. Two mechanisms in the protocol break that neutrality. First, §4.1 allows expected_behavior entries such as "read git-skill/SKILL.md before executing"; in the baseline condition the target skill is withheld, so such behaviors are unsatisfiable by construction. skill_execution and behavior_check then measure the presence of the intervention rather than its added value. Second, §4.2's --refine step replaces ground_truth with the agent's own final answer and lifts observed tool-call sequences from with-skill ATIF trajectories into expected_behavior. EVAL.md directives are preserved, but nothing requires the refined assets to be re-validated against an independent reference or against baseline-blind human judgment. The paper partially discloses this in Appendix A, where it notes that process metrics may be unsatisfiable at baseline and offers the outcome-only view; yet the abstract and §6.4 headline the composite, which includes skill_execution, skill_efficiency, and behavior_check. The component lifts are consistent with the bias: skill_execution +0.326, behavior_check +0.298, and skill_efficiency +0.276 dominate accuracy +0.143 and goal_accuracy +0.217, and the composite 0.2134 exceeds outcome-only 0.1799 by 0.0335. Without evidence that process-metric gains survive baseline-blind re-authoring of expected_behavior and ground_truth, the marginal-value reading of the headline is not established.
The reader's weakest-assumption analysis correctly identifies the central risk: ACES's evaluation assets are not shown to be an independent standard. The refinement loop in §4.2 is the sharpest version of the problem, because it can inject the with-skill agent's own behavior and final answers into the scoring contract, and the baseline condition is structurally unable to satisfy those refined behaviors. This is not an internal inconsistency or a fraud accusation; the paper is unusually transparent about the six metrics, the paired protocol, and the outcome-only alternative. But transparency does not neutralize the bias. The reported metric decomposition shows that the three process metrics contribute most of the gap between composite and outcome-only lift, and those are precisely the metrics most exposed to asset circularity. The open-source implementation is a genuine strength, but the production dataset is withheld, so the frozen 947-case headline cannot currently be independently audited. The paper's own Appendix B reconstruction is useful but does not test asset independence. I therefore keep the reader's CONDITIONAL verdict: the engineering contribution and the qualitative finding that live evaluation reveals behaviors static scans miss are credible, but the quantitative headline needs the asset-independence audit described in concrete_test before it can be taken as a fair measure of marginal skill value.
The referee's report
The paper addresses a real gap: production skill review lacks evidence about whether a skill helps a live agent. The paired differential design, the use of reference and decoy skills, the ATIF portability contract, and the open-source SkillEvaluator implementation are valuable and clearly presented. The empirical comparison of scan-only gates with live lift is a useful contribution, and the authors are transparent about clustering, missingness, and the outcome-only alternative. However, the central numeric claim—the 0.2134 composite lift—depends on the evaluation assets (expected_behavior and ground_truth) being an independent standard, and the current protocol does not demonstrate that independence.
Pith's reading of the paper (a machine's summary, not the authors')
This paper tries to establish that agent skills—reusable packages of procedural know-how that an LLM agent loads at runtime—should be evaluated as executable artifacts by running them, not only by scanning their documentation. To measure a skill's value, the authors run paired live trials: the same task, agent, model, workspace, and grader, once with the target skill available and once with it withheld, and report the difference as Skill Lift. Across 947 scored paired cases from 58 production skills and four agent harnesses, mean composite Skill Lift is 0.2134 on a 0-to-1 scale (95% paired-case CI [0.1967, 0.2301]), with outcome-only lift at 0.1799 and positive composite lift in 72.8% of cases. The authors also report that structural scans and LLM-judge document scores barely agree with each other (Spearman $\rho = 0.14$) and do not predict live lift, so scanning alone cannot support deployment decisions. If correct, the result gives skill authors and platform teams a practical, evidence-based gate for whether a skill helps or hurts a real agent.
What carries the argument. The load-bearing object is the paired differential protocol together with the Skill Lift statistic. For a target skill $s$ and agent $a$, Skill Lift is defined as $Lift_{s,a} = \frac{1}{|M|} \sum_{m \in M} (S^{with}_{s,a,m} - S^{base}_{s,a,m})$, the mean across configured metrics of the with-skill score minus the baseline score, where the baseline withholds the target skill while keeping configured prerequisite, helper, reference, and decoy skills fixed. Three portable contracts carry the protocol: an evaluation-asset contract (an evals.json dataset with prompts, ground_truth, and free-form expected_behavior assertions, plus bring-your-own-task and bring-your-own-grader assets), an adapter that materializes paired with-skill and baseline tasks per case, and the Agent Trajectory Interchange Format (ATIF), a versioned JSON trace schema that lets the same six-metric grading suite—security, skill execution, skill efficiency, accuracy, goal accuracy, behavior check—score any harness that emits, or can be converted to, ATIF. Isolation versus group workspace modes decompose the lift into a content contribution and a routing premium. The expected_behavior list is the most expressive mechanism: free-form natural-language sentences an LLM judge grades yes/no from the trajectory, allowing authors to encode workflow-order, safety, and tool-use requirements that final-answer scoring cannot see.
Core claim. On the paper's own terms: a skill is worth shipping when it raises the agent's composite score over a paired baseline under fixed task, harness, and grading policy. The headline finding is that the mean paired composite Skill Lift across 947 scored task cases is 0.2134 (95% paired-case CI [0.1967, 0.2301]), from absolute condition means of 0.7460 with the skill and 0.5326 without; mean outcome-only lift, the average of accuracy and goal-accuracy deltas, is 0.1799. The largest gains are process signals rather than final-answer quality: skill execution lifts by 0.3263, behavior check by 0.2983, and skill efficiency by 0.2758, while accuracy lifts by 0.1431. The authors further claim that document scans are not runtime evidence: across the 62 production skills with scan metadata, Tier 1 structural score and Tier 2 LLM-judge score correlate with overall live lift at Spearman $\rho = -0.0181$ and $-0.0266$, statistically indistinguishable from zero. They also document 87 negative-lift paired cases and zero- or negative-mean cells, using saved trajectories to separate skills that are never discovered from skills that are discovered but misused.
The premise it rests on. The headline lift rests on the assumption that the expected behaviors and reference answers used for grading are independent, faithful standards, yet the refinement step replaces the reference answer with the agent's own final answer and adds the agent's observed tool calls to the expected behaviors, which can make much of the measured gain self-confirming.
If the paper is right (Pith's inferences, not the authors')
Deployment gates for agent skills, plugins, and bundles should include a live paired run, because scan scores have near-zero correlation with live lift and cannot detect discovery, routing, workflow, or tool-use failures.
A skill that scans cleanly can still be a net regression: the 87 negative-lift paired cases and 11 negative-mean cells turn regressions into traceable pre-merge review targets.
Model upgrades can shrink measured Skill Lift even as absolute performance rises, because a stronger baseline needs the skill less; scheduled re-evaluation after model updates is part of the protocol.
The same protocol extends to bundles and team-skill combinations by treating the bundle itself as the intervention, and group-mode runs produce a routing premium that isolates discoverability problems.
Outcome-only lift (0.1799) is lower than composite lift (0.2134), so roughly a sixth of the measured skill value lives in process—how the agent works—rather than in the final answer alone.
If Skill Lift generalizes across organizations, skill registries and marketplaces could rank packages by independently measured marginal lift rather than by documentation quality or download counts, which would reward skills that actually change agent behavior.
The paper's refinement step—replacing ground truth with the agent's own answer and lifting observed tool calls into expected behaviors—creates a testable vulnerability: a calibration study comparing refined datasets against independently authored human datasets would show how much of the composite lift is self-referential rather than genuine added value.
The routing-premium decomposition suggests an automated decoy-selection policy: if reference skills are chosen by semantic similarity to the target, then group-mode lift becomes a harder, less gameable measure of discoverability than the current author-selected decoys.
The near-zero correlation between static scans and live lift implies scan rules could be repositioned as authoring aids and safety checks rather than quality gates, until a validated static proxy for runtime value is found.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman ρ = 0.14). Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. Composite lift is positive in 72.8% of paired cases. The largest process-metric gains appear in skill execution, behavior check, and skill efficiency—signals about discovery, routing, workflow following, and tool use that document scans cannot observe. An open-source implementation of the methodology is available in NVIDIA SkillEvaluator.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 0.64
“Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95\% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.08 · states limitation 0.54 · human participants 0.05 · controlled comparison 0.97 · randomized assignment 0.03 · states quantitative result 0.99 · compares with prior results 0.91 · mentions code or data availability 0.94
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Enterprise agent programs are moving from prototypes into production, where reusable skills, tools, and workflow packages must be reviewed with evidence rather than prose. | background 1.00 | 0.06 | 0.03 | asserted 1.00 |
| Current gates often scan these artifacts for structure, style, and security, but they do not answer the deployment question: does the capability package help a live agent complete enterprise tasks under the same model, sandbox, and grading policy? | gap 0.98 | 0.07 | 0.04 | asserted 1.00 |
| We present ACES (Agentic Continuous Evaluation of Skills), a repository-native framework for evaluating skills and product capability packages as executable agent artifacts. | contribution 1.00 | 0.85 | 0.08 | asserted 1.00 |
| ACES runs paired live trials with and without a target skill, normalizes trajectories into the Agent Trajectory Interchange Format (ATIF), grades six default runtime metrics, and reports Skill Lift: the target skill's added value for a fixed task, harness, workspace, and scorer. | method 0.97 | 0.91 | 0.09 | asserted 1.00 |
| The same protocol supports product-owned task suites that compare baseline, skill, bundle, team-skill, and plugin targets. | method 0.72 | 0.69 | 0.04 | asserted 1.00 |
| On 145 real skills from internal enterprise repositories and public catalogs, scan-only gates surface useful authoring issues but measure complementary facets (structural versus LLM-judge Spearman $\rho = 0.14$). | result 0.99 | 0.93 | 0.08 | asserted 1.00 |
| Across 947 scored paired cases from 58 of 64 production skills and four primary harnesses, mean composite Skill Lift is 0.2134 (95\% paired-case CI [0.1967, 0.2301]); mean outcome-only lift, the average of accuracy and goal accuracy, is 0.1799. | result 1.00 | 0.97 | 0.33 | asserted 1.00 |
| Composite lift is positive in 72.8\% of paired cases. | result 1.00 | 0.96 | 0.14 | asserted 1.00 |
| The largest process-metric gains appear in skill execution, behavior check, and skill efficiency---signals about discovery, routing, workflow following, and tool use that document scans cannot observe. | interpretation 0.63 | 0.93 | 0.14 | asserted 1.00 |
| An open-source implementation of the methodology is available in NVIDIA SkillEvaluator. | contribution 0.92 | 0.43 | 0.02 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,830 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (10)
Figures are the authors', reproduced from the paper.
References (38)
- [1] Accenture et al. 2025. MCP-Bench: Benchmarking Tool-Using LLMs with the Model Context Protocol.arXiv preprint arXiv:2508.20453(2025). https://arxiv. org/abs/2508.20453 on PithPith's reviewarXiv
- [2] Tejas Singh Anand, Yuet Ying Christina Wang, Wanting Jiang, Steve Masson, Tian Zheng, and Bingjie Zhou. 2026. AEVAL: From Anecdotal to Deterministic Testing for Agentic Skill Workflows.arXiv preprint arXiv:2607.16345(2026). https://arxiv.org/abs/2607.16345 Kevin et al on PithPith's reviewarXiv
- [3] Anthropic. 2025. Agent Skills: Best Practices. Claude Platform Documenta- tion. https://platform.claude.com/docs/en/agents-and-tools/agent-skills/best- practices Accessed April 2026 on Pith
- [4] Anthropic. 2025. Equipping Agents for the Real World with Agent Skills. An- thropic Engineering Blog. https://www.anthropic.com/engineering/equipping- agents-for-the-real-world-with-agent-skills Accessed April 2026 on Pith
- [5] Anthropic. 2025. How to Create Skills: Key Steps, Limitations, and Exam- ples. Claude Blog. https://claude.com/blog/how-to-create-skills-key-steps- limitations-and-examples Accessed April 2026 on Pith
- [6] Anthropic. 2025. skill-creator: Building Agent Skills. GitHub repository anthrop- ics/skills. https://github.com/anthropics/skills/tree/main/skills/skill-creator Accessed April 2026 on Pith
- [7] Anthropic. 2025. Skills Notebooks: Introduction. Claude Platform Cook- book. https://platform.claude.com/cookbook/skills-notebooks-01-skills- introduction Accessed April 2026 on Pith
- [8] Anthropic. 2025. Teach Claude Your Way of Working Using Skills. Claude Support. https://support.claude.com/en/articles/12580051-teach-claude-your- way-of-working-using-skills Accessed April 2026 on PitharXiv
- [9] Anthropic. 2026. Demystifying Evals for AI Agents. Anthropic Engineering Blog. https://www.anthropic.com/engineering/demystifying-evals-for-ai-agents Ac- cessed May 2026 on Pith
- [10] Jialong Chen, Xander Xu, Hu Wei, Chuan Chen, and Bing Zhao. 2026. SWE- CI: Evaluating Agent Capabilities in Maintaining Codebases via Continuous Integration.arXiv preprint arXiv:2603.03823(2026). https://arxiv.org/abs/2603. 03823 on PitharXiv
- [11] Wenbo Chen, Yimin Liu, Shenghan Zheng, Xiaokun Chen, Yifeng He, Yubo Li, Bingran You, Haotian Shen, Jiankai Sun, Shuyi Wang, Qunhong Zeng, Di Wang, Xuandong Zhao, Yuanli Wang, Roey Ben Chaim, Zonglin Di, Yipeng Gao, Junwei He, Yizhuo He, Liqiang Jing, Luyang Kong, Xin Lan, Jia... on PithPith's reviewarXiv
- [12] Confident AI. 2024. G-Eval: The Definitive Guide. Confident AI Blog. https: //www.confident-ai.com/blog/g-eval-the-definitive-guide Accessed April 2026 on Pith
- [13] Shahul Es, Jithin James, Luis Espinosa Anke, and Steven Schockaert. 2024. RA- GAS: Automated Evaluation of Retrieval Augmented Generation. InProceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics (EACL): System Demonstration... on PithPith's reviewarXiv
- [14] Google Cloud. 2024. MLOps: Continuous Delivery and Automation Pipelines in Machine Learning. Google Cloud Architecture Center. https://cloud.google.com/architecture/mlops-continuous-delivery-and- automation-pipelines-in-machine-learning Accessed May 2026 on Pith
- [15] Milan Gritta, Debjit Paul, Xiaoguang Li, Lifeng Shang, Jun Wang, and Gerasimos Lampouras. 2026. Process Evaluation for Agentic Systems. InFindings of the Asso- ciation for Computational Linguistics: EACL 2026. Association for Computational Linguistics, Rabat, Morocco, 2678–269... on Pithdoi:10.18653/v1/2026.findings-eacl.140
- [16] Tingxu Han, Yi Zhang, Wei Song, Chunrong Fang, Zhenyu Chen, Youcheng Sun, and Lijie Hu. 2026. SWE-Skills-Bench: Do Agent Skills Actually Help in Real-World Software Engineering?arXiv preprint arXiv:2603.15401(2026). https://arxiv.org/abs/2603.15401 on PithPith's reviewarXiv
- [17] Harbor Framework. 2026. Harbor: A Framework for Sandboxed Agent Task Evaluation. Project homepage. https://www.harborframework.com/ Accessed April 2026 on Pith
- [18] Harbor Framework. 2026. Harbor BaseAgent API Reference. Harbor documenta- tion. https://mintlify.wiki/harbor-framework/harbor/api/base-agent Accessed April 2026 on Pith
- [19] Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. 2024. SWE-bench: Can Language Models Resolve Real-World GitHub Issues?. InThe Twelfth International Conference on Learning Representations (ICLR). https://arxiv.org/abs/2... on PithPith's reviewarXiv
- [20] LangChain. 2025. Evaluating Skills. LangChain Blog. https://www.langchain. com/blog/evaluating-skills Accessed April 2026 on Pith
show 18 more
- [21] Xiao Liu, Hao Yu, Hanchen Zhang, Yifan Xu, Xuanyu Lei, Hanyu Lai, Yu Gu, Hangliang Ding, Kaiwen Men, Kejuan Yang, Shudan Zhang, Xiang Deng, Ao- han Zeng, Zhengxiao Du, Chenhui Zhang, Sheng Shen, Tianjun Zhang, Yu Su, Huan Sun, Minlie Huang, Yuxiao Dong, and Jie Tang. 2024. Age... on PithPith's reviewarXiv
- [22] Yujian Liu et al. 2026. How Well Do Agentic Skills Work in the Wild: Bench- marking LLM Skill Usage in Realistic Settings.arXiv preprint arXiv:2604.04323 (2026). https://arxiv.org/abs/2604.04323 on PithPith's reviewarXiv
- [23] Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval: NLG Evaluation using GPT-4 with Better Human Alignment. InProceedings of the 2023 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2511–2522. https://aclanthology... on Pith
- [24] Mike A. Merrill and Terminal-Bench Team. 2026. Terminal-Bench 2.0: Evalu- ating Autonomous Agents in Realistic Terminal Environments.arXiv preprint arXiv:2601.11868(2026). https://arxiv.org/abs/2601.11868 on PithPith's reviewarXiv
- [25] NVIDIA Corporation. 2026. SkillEvaluator. Software repository. https://github. com/NVIDIA/SkillEvaluator Version 0.1.0; Apache-2.0; accessed August 2026 on Pith
- [26] OpenAI. 2025. Codex Skills. OpenAI Codex Documentation. https://developers. openai.com/codex/skills Accessed April 2026 on Pith
- [27] OpenAI. 2025. Evaluating Skills. OpenAI Developers Blog. https://developers. openai.com/blog/eval-skills/ Accessed April 2026 on Pith
- [28] OpenAI. 2026. Evaluation Best Practices. OpenAI API Documentation. https: //platform.openai.com/docs/guides/evaluation-best-practices Accessed May 2026 on Pith
- [29] OpenClaw. 2026. Agent Transcript Skill. OpenClaw Agent Skills Repository. https://github.com/openclaw/agent-skills/tree/ 2a409d348a4bcf6f15e41e9a20efd0b298a32528/skills/agent-transcript Commit 2a409d348a4bcf6f15e41e9a20efd0b298a32528; accessed August 2026 on Pith
- [30] Salesforce AI Research. 2025. MCPEval: Deep Evaluation for AI Agents via the Model Context Protocol.arXiv preprint arXiv:2507.12806(2025). https: //arxiv.org/abs/2507.12806 on PithPith's reviewarXiv
- [31] Maksim Shaposhnikov, Nicolas Fortuin, Simon Stipcich, Maria I. Gorinova, Amy Heineike, and Rob Willoughby. 2026. A Framework for Evaluating Agentic Skills at Scale.arXiv preprint arXiv:2606.17819(2026). https://arxiv.org/abs/2606.17819 on PithPith's reviewarXiv
- [32] SkillsBench Team. 2026. SkillsBench: Benchmarking How Well Skills Work Across Diverse Tasks. SkillsBench project site. https://www.skillsbench.ai/ Accessed April 2026 on Pith
- [33] Spillwave. 2025. Mastering Agentic Skills: The Complete Guide to Building Effec- tive Agent Skills. Spillwave Publication. https://pub.spillwave.com/mastering- agentic-skills-the-complete-guide-to-building-effective-agent-skills- d3fe57a058f1 Accessed April 2026 on Pith
- [34] Vercel. 2025. Agent Skills Explained: An FAQ. Vercel Blog. https://vercel.com/ blog/agent-skills-explained-an-faq Accessed April 2026 on Pith
- [35] Leye Wang, Zixing Wang, and Anjie Xu. 2026. SkillTester: Benchmarking Utility and Security of Agent Skills.arXiv preprint arXiv:2603.28815(2026). https: //arxiv.org/abs/2603.28815 on PitharXiv
- [36] Yang Wang et al. 2026. GTA-2: Benchmarking General Tool Agents from Atomic Tool-Use to Open-Ended Workflows.arXiv preprint arXiv:2604.15715(2026). https://arxiv.org/abs/2604.15715 on PithPith's reviewarXiv
- [37] Boming Xia, Qinghua Lu, Liming Zhu, Zhenchang Xing, Dehai Zhao, and Hao Zhang. 2024. An Evaluation-Driven Approach to Designing LLM Agents: Process and Architecture.arXiv preprint arXiv:2411.13768(2024). https://arxiv.org/abs/ 2411.13768 on PitharXiv
- [38] Dexu Yu et al. 2026. SkillAudit: From Fixed-Suite Benchmarking to Skill-Centered Assessment.arXiv preprint arXiv:2606.22613(2026). https://arxiv.org/abs/2606. 22613 Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills A Metric Contract and Weighting Tabl... on PithPith's reviewarXiv
Cited by 3 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Beyond Gain: Measuring Skill Internalization for Agent Skill Retirement cs.AI · Oct 2026
- Scanning the Harness: Configuration Exposures in AI Coding-Agent Supply Chains cs.SE · Sep 2026
- Mining Agent Skills from Production Traces cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment