{"id":"905b5d92-72d5-41d8-a0f8-d1adbe6173e7","arxiv_id":"2608.20614","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"ACES measures a skill's value by paired with-skill and baseline agent runs, reporting positive average lift across 947 enterprise cases, though part of the gain is baked into the scoring.","lead":"This paper introduces ACES, a system that tests an AI helper skill by running the same job twice: once with the skill available and once without, then comparing the results. It reports that skills help agents across 947 paired trials, and that static document checks are poor predictors of real-world skill success.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Skill Lift is partly predetermined: §4.2 refinement derives expected_behavior and ground_truth from with-skill trajectories, so the composite headline overstates added value until asset independence is audited.","rationale":"The reader's weakest-assumption analysis correctly identifies the central risk: ACES's evaluation assets are not shown to be an independent standard. The refinement loop in §4.2 is the sharpest version of the problem, because it can inject the with-skill agent's own behavior and final answers into the scoring contract, and the baseline condition is structurally unable to satisfy those refined behaviors. This is not an internal inconsistency or a fraud accusation; the paper is unusually transparent about the six metrics, the paired protocol, and the outcome-only alternative. But transparency does not neutralize the bias. The reported metric decomposition shows that the three process metrics contribute most of the gap between composite and outcome-only lift, and those are precisely the metrics most exposed to asset circularity. The open-source implementation is a genuine strength, but the production dataset is withheld, so the frozen 947-case headline cannot currently be independently audited. The paper's own Appendix B reconstruction is useful but does not test asset independence. I therefore keep the reader's CONDITIONAL verdict: the engineering contribution and the qualitative finding that live evaluation reveals behaviors static scans miss are credible, but the quantitative headline needs the asset-independence audit described in concrete_test before it can be taken as a fair measure of marginal skill value.","tokens_in":21489,"tokens_out":4632,"duration_ms":42839,"concrete_test":"On a random sample of production skills, retrieve the frozen evals.json and classify each expected_behavior entry as (a) author-written before any live run, (b) lifted via --refine from a with-skill ATIF trajectory, or (c) generated from SKILL.md; classify each ground_truth as an original author reference or a refined with-skill final answer. Re-score the paired trajectories three ways: (1) using only category-(a) assets; (2) using assets re-authored by a reviewer blinded to all with-skill trajectories and condition labels; (3) using human-written reference answers independent of agent outputs. Compare composite, behavior_check, and skill_execution lift across these three rescorings.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—that adding a skill raises composite task performance by 0.2134—depends on ACES's scoring assets being a neutral measuring instrument. Two mechanisms in the protocol break that neutrality. First, §4.1 allows expected_behavior entries such as \"read git-skill/SKILL.md before executing\"; in the baseline condition the target skill is withheld, so such behaviors are unsatisfiable by construction. skill_execution and behavior_check then measure the presence of the intervention rather than its added value. Second, §4.2's --refine step replaces ground_truth with the agent's own final answer and lifts observed tool-call sequences from with-skill ATIF trajectories into expected_behavior. EVAL.md directives are preserved, but nothing requires the refined assets to be re-validated against an independent reference or against baseline-blind human judgment. The paper partially discloses this in Appendix A, where it notes that process metrics may be unsatisfiable at baseline and offers the outcome-only view; yet the abstract and §6.4 headline the composite, which includes skill_execution, skill_efficiency, and behavior_check. The component lifts are consistent with the bias: skill_execution +0.326, behavior_check +0.298, and skill_efficiency +0.276 dominate accuracy +0.143 and goal_accuracy +0.217, and the composite 0.2134 exceeds outcome-only 0.1799 by 0.0335. Without evidence that process-metric gains survive baseline-blind re-authoring of expected_behavior and ground_truth, the marginal-value reading of the headline is not established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes ACES, a repository-native framework that evaluates agent skills by running paired live trials—with and without the target skill—under fixed tasks, harnesses, sandboxes, and grading policies. Trajectories are normalized to ATIF, scored by six default metrics (security, skill execution, skill efficiency, accuracy, goal accuracy, behavior check), and summarized as Skill Lift. On a corpus of 145 skills, the authors report that static structural and LLM-judge scores disagree (Spearman ρ = 0.14) and that across 947 scored paired cases from 58 production skills, mean composite Skill Lift is 0.2134 (95% CI [0.1967, 0.2301]) while outcome-only lift is 0.1799. The paper also documents refinement of evaluation assets from trajectories, an evidence inventory, sensitivity analyses, and a discussion of limitations, including the authors' acknowledgment that process metrics may be unsatisfiable in the baseline.","tokens_in":21769,"tokens_out":6132,"duration_ms":54109,"significance":"The paper addresses a real gap: production skill review lacks evidence about whether a skill helps a live agent. The paired differential design, the use of reference and decoy skills, the ATIF portability contract, and the open-source SkillEvaluator implementation are valuable and clearly presented. The empirical comparison of scan-only gates with live lift is a useful contribution, and the authors are transparent about clustering, missingness, and the outcome-only alternative. However, the central numeric claim—the 0.2134 composite lift—depends on the evaluation assets (expected_behavior and ground_truth) being an independent standard, and the current protocol does not demonstrate that independence.","major_comments":[{"comment":"The composite Skill Lift headline is inflated by process metrics that the baseline cannot satisfy by construction. Section 4.1 allows expected_behavior entries such as “read git-skill/SKILL.md before executing”; in the baseline condition the target skill is withheld, so that behavior is unsatisfiable regardless of agent competence. The skill_execution sub-checks (activation, script_execution, workflow_order) in §4.3 and behavior_check then measure the presence of the intervention rather than its marginal value, and Table 8 weights each of these equally (1/6) in the composite. Section 6.4 reports process gains of +0.3263, +0.2983, and +0.2758 against accuracy +0.1431, and the abstract and §6.4 headline the composite 0.2134 even though Appendix A concedes that process metrics “may be unable to satisfy when the target skill is withheld.” Because the central empirical claim is the composite lift, this is load-bearing: the authors should either make outcome-only lift (0.1799) the headline, or re-author expected_behavior and process checks blinded to condition and show that the process gains survive.","section":"§4.1, §4.3, §4.7, Table 8, Appendix A, §6.4"},{"comment":"The trajectory-grounded refinement step in §4.2 replaces ground_truth with the agent's final answer when it “aligns with author intent”; as written, this makes accuracy self-referential unless the alignment check is an independent, documented human judgment. The accuracy judge in §4.3 compares the final response to ground_truth, so if the reference is derived from the same with-skill trajectory, the accuracy lift of 0.1431 measures self-consistency rather than correctness. This also contaminates the outcome-only lift of 0.1799 because accuracy is one of its two components. The paper should report how many ground_truth entries were replaced, specify who performed the alignment check, and show that accuracy conclusions survive when the original references are restored.","section":"§4.2, §4.3, §6.4"},{"comment":"The frozen 947-case headline excludes condition-run failures, timeouts, and unscored cases unless both condition scores are present, and the reconstruction audit in Appendix B covers only 898 of the 947 pairs. Section 6.6 itself gives an example where the with-skill run produced no scored trials while the baseline completed, so exclusion can plausibly bias lift upward. The conservative imputation in Appendix B is helpful but does not apply to all 947 frozen pairs. The authors should report a worst-case bound for the full 947 cases, treating missing with-skill scores as failures and missing baseline scores as successes, or otherwise bound the selection bias.","section":"§6.2, §6.6, Appendix B"},{"comment":"Because behavior_check and accuracy are LLM-judged and §7 reports no live inter-judge agreement, human calibration, or judge-uncertainty propagation, the 95% confidence interval in §6.4 should be labeled as covering only case-level sampling variation, not judge noise. The authors should either add a judge-agreement study on a random subsample of trajectories or soften the precision claims attached to the headline lift.","section":"§7, Table 8"}],"minor_comments":[{"comment":"The abstract contains a typo: “reportsSkill Lift” should read “reports Skill Lift.”","section":"Abstract"},{"comment":"Figure 8 labels the overall lift as “73%+” while §6.4 states composite lift is positive in 689/947 = 72.8% of paired cases; the figure should use the same rounding convention as the text.","section":"Figure 8"},{"comment":"The display equation defining Skill Lift is unnumbered; number it so that the outcome-only formula in Appendix A and later references can be cross-referenced precisely.","section":"§4.7"},{"comment":"Table 7 says the security backfill “found secret-like strings, destructive command patterns, unauthorized-path access” across live artifacts; clarify whether these are confirmed findings or pattern matches, since Appendix A notes that the security metric has pattern false positives.","section":"Table 7, Appendix A"},{"comment":"The relationship between the 145 scanned skills, 89 live-evaluated variants, and 64 production skills is stated only in §6.2; stating this filtering chain earlier would help readers connect the abstract's corpus claims to the live subset.","section":"§6.1, §6.2"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nThe thing to know about arXiv:2608.20614 is that the framework it describes is real and useful, but the headline number overstates the marginal value of skills. The composite Skill Lift of 0.2134 is partly predetermined: three of the six components—skill_execution, behavior_check, skill_efficiency—include expectations like \"read the target skill's SKILL.md before executing,\" which the baseline condition cannot satisfy because the skill is withheld. The refine step in §4.2 makes it worse by replacing ground_truth with the agent's own final answer and lifting observed tool calls into expected_behavior. That makes the scoring reference self-referential. To their credit, the authors disclose this in Appendix A and report outcome-only lift (0.1799), which is more honest. But the abstract and §6.4 still lead with the composite, so a reader skimming the paper gets an inflated number.\n\nWhat is actually new: the combination of paired with-skill/baseline trials, an ATIF trajectory contract across harnesses, author-owned evals, and CI-native reporting. I don't know a prior tool that covers all five capabilities listed in Table 3. The empirical observation that static scans and LLM-judge rubrics correlate at only rho=0.14, and that neither predicts live lift (rho near zero), is a genuinely useful result for anyone building skill gates. The evidence section is unusually transparent: they report confidence intervals, cluster checks, missingness audits, and even show what happens under conservative imputation. The open-source SkillEvaluator implementation is a concrete artifact the community can use.\n\nThe soft spots beyond the metric issue: the dataset is internal NVIDIA, so the headline numbers are not independently reproducible. The anonymized skill set helps but does not let anyone verify the distribution. The refinement loop is described as closing the gap between hypothesized and observed behavior, but without an independent human audit it can turn the eval into a tautology.\n\nProportionately, the paper is a serious engineering contribution with a fixable measurement problem. The framework is sound; the composite metric needs to either exclude process components that are trivially unsatisfiable at baseline or be re-validated blind. The outcome-only result, and the qualitative findings about routing and workflow, survive the critique.\n\nI would send this to peer review. A good referee will ask for exactly the re-authoring audit I mentioned; that's manageable. I'd cite it for the paired-protocol design and the static-vs-live correlation result, and bring it to a reading group if anyone cares about agent evaluation infrastructure.","headline":"A genuinely useful paired-evaluation framework for agent skills, but the headline composite Skill Lift overstates marginal value because process metrics and refined ground truth are derived from the skill itself.","tokens_in":22365,"tokens_out":3221,"duration_ms":26808},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adding an agent skill to a live agent's workspace raises composite task performance by 0.2134 on average across 947 paired trials, and the paper argues skill review should be built on such live evidence rather than documentation scans.","keywords":["agent skills","LLM agents","evaluation","LLM-as-Judge","Skill Lift","continuous evaluation","ATIF","paired trials"],"falsifier":"Re-run the paired protocol on a sample of high-lift production skills using evaluation assets written independently—expected behaviors and reference answers authored by someone who has not seen the skill or the agent trajectories—and compare the resulting composite Skill Lift with the 0.2134 headline; if it collapses toward zero, the headline lift is an artifact of self-referential grading.","tokens_in":21307,"feed_emoji":"🤖","tokens_out":11460,"duration_ms":87469,"temperature":0.7,"pith_summary":"This paper tries to establish that agent skills—reusable packages of procedural know-how that an LLM agent loads at runtime—should be evaluated as executable artifacts by running them, not only by scanning their documentation. To measure a skill's value, the authors run paired live trials: the same task, agent, model, workspace, and grader, once with the target skill available and once with it withheld, and report the difference as Skill Lift. Across 947 scored paired cases from 58 production skills and four agent harnesses, mean composite Skill Lift is 0.2134 on a 0-to-1 scale (95% paired-case CI [0.1967, 0.2301]), with outcome-only lift at 0.1799 and positive composite lift in 72.8% of cases. The authors also report that structural scans and LLM-judge document scores barely agree with each other (Spearman $\\rho = 0.14$) and do not predict live lift, so scanning alone cannot support deployment decisions. If correct, the result gives skill authors and platform teams a practical, evidence-based gate for whether a skill helps or hurts a real agent.","feed_headline":"Agent skills lift task scores by 0.21 in live trials","feed_subtitle":"Paired with/without runs show skills change how agents work, not just final answers","key_machinery":"The load-bearing object is the paired differential protocol together with the Skill Lift statistic. For a target skill $s$ and agent $a$, Skill Lift is defined as $Lift_{s,a} = \\frac{1}{|M|} \\sum_{m \\in M} (S^{with}_{s,a,m} - S^{base}_{s,a,m})$, the mean across configured metrics of the with-skill score minus the baseline score, where the baseline withholds the target skill while keeping configured prerequisite, helper, reference, and decoy skills fixed. Three portable contracts carry the protocol: an evaluation-asset contract (an evals.json dataset with prompts, ground_truth, and free-form expected_behavior assertions, plus bring-your-own-task and bring-your-own-grader assets), an adapter that materializes paired with-skill and baseline tasks per case, and the Agent Trajectory Interchange Format (ATIF), a versioned JSON trace schema that lets the same six-metric grading suite—security, skill execution, skill efficiency, accuracy, goal accuracy, behavior check—score any harness that emits, or can be converted to, ATIF. Isolation versus group workspace modes decompose the lift into a content contribution and a routing premium. The expected_behavior list is the most expressive mechanism: free-form natural-language sentences an LLM judge grades yes/no from the trajectory, allowing authors to encode workflow-order, safety, and tool-use requirements that final-answer scoring cannot see.","core_discovery":"On the paper's own terms: a skill is worth shipping when it raises the agent's composite score over a paired baseline under fixed task, harness, and grading policy. The headline finding is that the mean paired composite Skill Lift across 947 scored task cases is 0.2134 (95% paired-case CI [0.1967, 0.2301]), from absolute condition means of 0.7460 with the skill and 0.5326 without; mean outcome-only lift, the average of accuracy and goal-accuracy deltas, is 0.1799. The largest gains are process signals rather than final-answer quality: skill execution lifts by 0.3263, behavior check by 0.2983, and skill efficiency by 0.2758, while accuracy lifts by 0.1431. The authors further claim that document scans are not runtime evidence: across the 62 production skills with scan metadata, Tier 1 structural score and Tier 2 LLM-judge score correlate with overall live lift at Spearman $\\rho = -0.0181$ and $-0.0266$, statistically indistinguishable from zero. They also document 87 negative-lift paired cases and zero- or negative-mean cells, using saved trajectories to separate skills that are never discovered from skills that are discovered but misused.","pith_inferences":["If Skill Lift generalizes across organizations, skill registries and marketplaces could rank packages by independently measured marginal lift rather than by documentation quality or download counts, which would reward skills that actually change agent behavior.","The paper's refinement step—replacing ground truth with the agent's own answer and lifting observed tool calls into expected behaviors—creates a testable vulnerability: a calibration study comparing refined datasets against independently authored human datasets would show how much of the composite lift is self-referential rather than genuine added value.","The routing-premium decomposition suggests an automated decoy-selection policy: if reference skills are chosen by semantic similarity to the target, then group-mode lift becomes a harder, less gameable measure of discoverability than the current author-selected decoys.","The near-zero correlation between static scans and live lift implies scan rules could be repositioned as authoring aids and safety checks rather than quality gates, until a validated static proxy for runtime value is found."],"forward_implications":["Deployment gates for agent skills, plugins, and bundles should include a live paired run, because scan scores have near-zero correlation with live lift and cannot detect discovery, routing, workflow, or tool-use failures.","A skill that scans cleanly can still be a net regression: the 87 negative-lift paired cases and 11 negative-mean cells turn regressions into traceable pre-merge review targets.","Model upgrades can shrink measured Skill Lift even as absolute performance rises, because a stronger baseline needs the skill less; scheduled re-evaluation after model updates is part of the protocol.","The same protocol extends to bundles and team-skill combinations by treating the bundle itself as the intervention, and group-mode runs produce a routing premium that isolates discoverability problems.","Outcome-only lift (0.1799) is lower than composite lift (0.2134), so roughly a sixth of the measured skill value lives in process—how the agent works—rather than in the final answer alone."],"supporting_citations":[{"why":"Supplies the rubric-anchored LLM-as-judge pattern used for accuracy and expected-behavior grading.","marker":"[23]"},{"why":"Supplies the reference-based goal-accuracy evaluation used as a default metric.","marker":"[13]"},{"why":"Provides the sandboxed execution and ATIF-native agent interface that ACES builds on for task lifecycle and trace emission.","marker":"[18]"},{"why":"Provides the container and cloud sandbox substrate behind the dynamic task emitter.","marker":"[17]"},{"why":"Earlier skill benchmark establishing that skills improve agents; ACES extends it by scoring individual skills on author-owned tasks.","marker":"[11]"},{"why":"Shows skill benefits degrade under realistic retrieval, motivating ACES's discovery pressure and configured decoys.","marker":"[22]"},{"why":"Supplies the four-bucket dataset seeding taxonomy (explicit, implicit, contextual, negative control).","marker":"[27]"},{"why":"Motivates trajectory-level behavior checks because final-answer scoring hides skipped steps and bypassed procedures.","marker":"[15]"},{"why":"Open-source implementation of the ACES protocol that operationalizes the paired-evaluation tier.","marker":"[25]"}],"fun_headline_variants":["Skill lift hits 0.21 in 947 live agent trials","Skills improve agent process, not just answers: lift 0.21","Scan-only gates fail; live skill lift is 0.21","Agent skills add 0.21 lift over baselines in live runs","0.21 average Skill Lift from paired agent evaluations"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The headline lift rests on the assumption that the expected behaviors and reference answers used for grading are independent, faithful standards, yet the refinement step replaces the reference answer with the agent's own final answer and adds the agent's observed tool calls to the expected behaviors, which can make much of the measured gain self-confirming.","fun_headline_variants_meta":{"raw":{"variants":["Skill lift hits 0.21 in 947 live agent trials","Skills improve agent process, not just answers: lift 0.21","Scan-only gates fail; live skill lift is 0.21","Agent skills add 0.21 lift over baselines in live runs","0.21 average Skill Lift from paired agent evaluations"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1730,"prompt_tokens":1147,"completion_tokens":583,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":763,"completion_tokens_details":{"reasoning_tokens":493}},"tokens_in":763,"tokens_out":583,"duration_ms":5279,"temperature":1.0,"reasoning_tokens":493,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T19:18:11.666845+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Re-run the paired protocol on a sample of high-lift production skills using evaluation assets written independently—expected behaviors and reference answers authored by someone who has not seen the skill or the agent trajectories—and compare the resulting composite Skill Lift with the 0.2134 headline; if it collapses toward zero, the headline lift is an artifact of self-referential grading.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the rubric-anchored LLM-as-judge pattern used for accuracy and expected-behavior grading."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the sandboxed execution and ATIF-native agent interface that ACES builds on for task lifecycle and trace emission."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the container and cloud sandbox substrate behind the dynamic task emitter."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the four-bucket dataset seeding taxonomy (explicit, implicit, contextual, negative control)."},{"cited_title":"Process Evaluation for Agentic Systems","cited_arxiv_id":null,"evidence_quote":"Motivates trajectory-level behavior checks because final-answer scoring hides skipped steps and bypassed procedures."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Open-source implementation of the ACES protocol that operationalizes the paired-evaluation tier."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}