{"id":"c5a0c10e-1c17-4040-a2df-f13f8bd63398","arxiv_id":"2608.14036","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Skills help LLM agents mainly by anchoring procedures rather than injecting knowledge, and retrieval difficulty, not execution, is the main scalability bottleneck.","lead":"This paper runs controlled experiments on LLM agents to find when skills help, why they work, and where they fail. It claims skills mostly act as procedural anchors that stabilize execution, and that retrieval problems are separate from execution problems.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 65.7%-vs-4.5% procedural-anchoring result rests on LLM mechanism labels for the 528 paired triples, a labeling stage with no human agreement statistics; a systematic LLM bias here would undo the paper's central 'why skills work' claim.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the procedural-anchoring conclusion rests on LLM-assisted mechanism labels for the 528 paired triples, with no human agreement statistics for that specific labeling stage. I agree, and I would not move the verdict because the concern is addressable and the paper is transparent about its limitation. The central contribution of the paper is mechanistic: the claim that skills stabilize action rather than inject missing facts is what distinguishes this evaluation from prior aggregate-success studies. The 6.06-point skill-over-workflow delta, though headline-adjacent, is less load-bearing because its confidence interval is wide and the reader already asked for more cautious reporting. A human re-labeling of a stratified subset of triples would settle whether the 65.7% versus 4.5% split is a real behavioral contrast or an artifact of the truncated-excerpt, broad-category labeling protocol. If the human procedural_anchor share falls well below the LLM share, the paper's main interpretation is unsupported and the contribution reduces to a well-controlled but largely confirmatory comparison of representation formats. If the labels reproduce, the central claim is credible. This keeps the verdict at CONDITIONAL rather than ACCEPT or REJECT.","tokens_in":28007,"tokens_out":5738,"duration_ms":48739,"concrete_test":"Select a stratified random subset of 100 of the 528 paired triples (stratified by benchmark and trajectory mixture). Give two independent human annotators the same paired-comparison prompt and taxonomy definitions (Appendix B.3), with the same truncated excerpts, and have them assign the skill_mechanism label for each triple. Compute Cohen's kappa between the LLM judge and the pooled human labels, and compare the human procedural_anchor proportion with the LLM's 65.7%. Pre-register thresholds: the claim is supported only if kappa >= 0.7 and the human procedural_anchor share is within 10 percentage points of the LLM share; otherwise the mechanism percentages must be reported as unvalidated LLM judgments and the central interpretation softened accordingly.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central explanatory claim — skills work primarily through procedural anchoring (65.7%) rather than knowledge injection (4.5%) — is computed from the paired-comparison stage of Appendix A.2, where a single LLM judge assigns one of five mechanism labels (procedural_anchor, knowledge_injection, failure_warning, none, counterproductive) per skill arm. Table 3's human validation covers only two earlier stages: grounding of the 238 open-coding labels (714 trajectory checks) and mapping those raw labels to the 12 canonical modes (Cohen's kappa = 0.952). It does not validate the mechanism labels on any of the 528 triples. The paper concedes this in Appendix A.2: 'the full paired-triple dataset remains LLM-assisted rather than exhaustively human-coded.' Two mechanisms could bias the 65.7% figure. First, each trajectory in the paired prompt is a 4,000-character head plus an 8,000-character tail; the middle of the run, where explicit knowledge consultation or reasoning would appear, is omitted. The judge therefore sees final command sequences and verifier checks, which look procedural by construction. Second, the mechanism schema defines procedural_anchor as 'a usable procedure, ordering, checklist, tool sequence, or verification plan' — a category broad enough to cover almost any useful terminal-task artifact — while knowledge_injection is a narrow residual. If human annotators on the same triples assigned procedural_anchor substantially less often, or disagreed with the LLM at low kappa, the headline 'skills stabilize action rather than inject missing facts' would not be established, even though the underlying success-rate comparisons might stand.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper asks when skills help LLM agents, why they work, and where they fail, moving beyond aggregate success rates. The authors run controlled experiments on Terminal-Bench and SkillsBench, holding source trajectories fixed while varying representation (Raw, Workflow Memory, and distilled Skill), outcome annotations (normal vs. no-hint), and agent framework (Codex and Gemini CLI). They construct a 12-mode taxonomy from an open-coding pass over 240 sampled trajectories (238 valid labels), validate the taxonomy-construction stages with human annotation, and then apply the taxonomy to 528 paired triples of raw/workflow/skill executions. The headline findings are that skills improve over Workflow Memory by 6.06 points in matched comparisons, that skills act primarily as procedural anchors (65.7% of mechanism labels) rather than knowledge injectors (4.5%), that execution-layer failures drop with skills, that outcome annotations matter most when failed trajectories are present, and that retrieval precision collapses as pools grow while downstream success remains relatively stable. The paper concludes that skill use should be understood as a lifecycle problem of representation, retrieval, invocation, and adaptation.","tokens_in":28252,"tokens_out":3931,"duration_ms":36683,"significance":"The paper's controlled design is a genuine step beyond black-box success-rate evaluation: the same trajectory pool is distilled into Workflow Memory and Skill, the paired-triple methodology makes mechanism-level claims inspectable, bootstrap confidence intervals accompany the major deltas, the retrieval ground truth comes from SkillsBench's native annotations rather than from the authors' predictions, and the full prompts and deterministic aggregation rules are included. If the mechanism-labeling results are reliable, the procedural-anchoring claim would be a substantive, falsifiable contribution to the agent-memory literature, and the retrieval analysis usefully separates offline identification from execution-time use. The main weakness is that the central 'why skills work' percentages rest on LLM mechanism labels for the 528 paired triples that lack human agreement statistics; Table 3's validation covers only the earlier taxonomy-construction stages. Given the paper's own admission in Appendix A.2 that the paired-triple dataset remains LLM-assisted, the significance of the core explanatory claim is currently conditional on additional validation rather than fully established.","major_comments":[{"comment":"The 65.7% procedural-anchoring versus 4.5% knowledge-injection result is computed from the paired-comparison stage over 528 triples, but Table 3's human validation covers only the 238 open-coding labels and their mapping to the 12 canonical modes (714 trajectory checks; κ=0.952). The mechanism labels assigned to every skill arm in the paired triples have no reported human agreement statistics; Appendix A.2 explicitly concedes that 'the full paired-triple dataset remains LLM-assisted rather than exhaustively human-coded.' Since the paper's central claim that 'skills work as procedural anchors, not merely external knowledge' is built on these labels, please add a human-annotation study on a stratified sample of triples (or a detailed audit of the evidence quotes behind the mechanism assignments) and report inter-annotator agreement. Without this, the 65.7%-versus-4.5% contrast could reflect systematic LLM judgment bias rather than agent behavior.","section":"§4 and Appendix A.2; Table 3"},{"comment":"The mechanism schema in Table 2 defines procedural_anchor as 'a usable procedure, ordering, checklist, tool sequence, or verification plan,' a category broad enough to cover almost any useful terminal-task artifact, while knowledge_injection is a narrow residual. In addition, the paired-comparison prompt gives the judge only a 4,000-character head and an 8,000-character tail of each trajectory, omitting the middle of the run where explicit knowledge consultation or reasoning would appear. These two design choices could mechanically inflate procedural_anchor relative to knowledge_injection. Please provide a confusion-style analysis or re-annotation by human coders using the same schema, and quantify how the mechanism distribution changes when longer or complete transcripts are provided to the judge.","section":"Table 2 and Appendix A.2"},{"comment":"The paired-triple analysis selects one representative trajectory per arm under a deterministic trial-id ordering rather than averaging the repeated trials available in the manifest, and the taxonomy is induced from a stratified sample covering roughly 3% of normalized records, as the Limitations section acknowledges. These choices make the headline percentages (e.g., the 61.9% vs. 55.9% success gap and the SC2 reduction from 37.3% to 23.5%) depend on a single trajectory per task-setting-arm cell. Please add a sensitivity analysis using the paper's own seeded random selection policy and report how the mechanism percentages and paired deltas vary across random draws; this would materially strengthen confidence that the procedural-anchoring conclusion is not an artifact of the particular trial selected.","section":"Appendix A.2 and Section 7"}],"minor_comments":[{"comment":"The paper reports 8,135 normalized trial records and 7,837 records with transcripts, but percentage statements in the text do not always make clear whether the denominator is all records or only those with transcripts; please state the effective denominator when reporting taxonomy and paired-triple statistics.","section":"§4 and Table 7"},{"comment":"Please specify how many human annotators participated in the validation, whether the reported Cohen's kappa is between two human annotators or between a human and the LLM, and what annotation instructions were used; this information is necessary to interpret the reported κ=0.952.","section":"Table 3"},{"comment":"Because RQ4 uses GPT-5.4 for the Codex pairing while RQ1–RQ3 use GPT-5.3-Codex, the figure captions for the RQ4 results should explicitly mark this model mismatch so readers do not compare absolute values across research questions.","section":"§3.2 and Figure 3"},{"comment":"The stacked-bar color legend for the 12 taxonomy modes is difficult to read in the small printed figure; a table with per-mode percentages (as in Table 11) or a labeled panel for each category would improve accessibility.","section":"Figure 2"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is one of the few papers that actually tries to open the skill black box: the contrastive paired-trajectory setup, the three independent retrieval arms, and the twelve-mode taxonomy are a real step beyond aggregate success rates. Second, the paper's central explanatory claim — procedural anchoring at 65.7% versus 4.5% knowledge injection — is not yet supported at the same standard as the rest of the study, because the paired-triple mechanism labels are LLM-generated with no human agreement statistics.\n\nWhat is genuinely new: the controlled representation comparison holds source trajectories fixed while varying only whether they become workflow memory or a distilled skill; the retrieval dissociation (offline selection degrades with confusable pools, while execution-time parsed-use precision collapses from 29.6% to 3.3% and downstream success stays flat) is an interesting and credible finding. The matched-pairs bootstrap and the no-hint ablation are thoughtful. The paper is also honest: it flags the LLM-assisted nature of the paired data, the 3% sample, and the limited model coverage. The taxonomy construction stages that were human-checked — 238 raw labels grounded in 714 trajectory checks, and the mapping to 12 modes at Cohen's kappa 0.952 — look solid. The citation pattern is fair: SkillsBench and SWE-Skills-Bench are cited as prior benchmarks, and the paper distinguishes itself without overselling.\n\nWhere it is soft. The 65.7%-vs-4.5% split is the headline, but it comes from a single LLM judge assigning one of five mechanism labels to each of 528 triples, using a 4,000-character head and an 8,000-character tail per trajectory. The middle of the run — where explicit knowledge consultation would appear — is cut out, and the procedural_anchor definition is broad enough to cover almost any useful artifact, while knowledge_injection is a narrow residual. So the stress-test concern is on target: if a human annotation pass on the same triples produced low agreement or shifted the split, the \"skills stabilize action rather than inject facts\" conclusion would need to be softened. The success-rate comparisons do not depend on that label, but the explanatory story does. Also, the main +6.06-point advantage over Workflow Memory has a 95% CI of [+0.76, +11.36]; it is positive but not tight. Reproducibility is partly promised, but I do not see an actual public release of code, data, and skill artifacts; that should be a condition.\n\nWho it is for: anyone working on agent memory, skill libraries, or retrieval for LLM agents. It deserves serious peer review, but the referee should push on the paired-label validation and ask for either released labels with agreement statistics or a hedged mechanism claim. I would accept it only with those changes.","headline":"Serious, well-designed empirical study with a real methodological step forward, but the headline mechanism split (65.7% vs 4.5%) rests on an LLM-only labeling pass that needs human agreement data before the paper's central explanatory claim can be taken as established.","tokens_in":28882,"tokens_out":2730,"duration_ms":24106},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Skills help LLM agents mostly by stabilizing action sequences—procedural anchoring—rather than by supplying missing facts, and they fail when retrieval, invocation, or task context breaks that anchoring.","keywords":["agent skills","procedural anchoring","LLM agents","trajectory analysis","retrieval evaluation","workflow memory","skill taxonomy"],"falsifier":"Have two independent human annotators re-label the full 528 paired triples from complete, untruncated transcripts, then recompute the procedural-anchor share; if the share drops below roughly half of the reported 65.7%, or the skill-vs-workflow paired delta loses its positive sign, the central taxonomy claim is not stable.","tokens_in":27769,"feed_emoji":"🧭","tokens_out":5615,"duration_ms":47090,"temperature":0.7,"pith_summary":"The paper asks when, why, and where skills help LLM agents. Using paired trajectories—same task run raw, with workflow memory, and with a distilled SKILL.md—it finds skills mainly work by procedural anchoring: they stabilize setup steps, tool sequences, and checks (65.7% of skill mechanisms) rather than injecting facts (4.5%). This gives a 6.06-point gain over workflow memory on matched tasks. Retrieval is a separate bottleneck, and skills fail when assumptions are brittle, contexts incompatible, or adaptation insufficient. A reader should care because this moves skill evaluation beyond aggregate success rates to a mechanism-level explanation.","feed_headline":"Procedural anchoring explains 65.7% of skill success","feed_subtitle":"A controlled contrastive study shows skills beat workflow memory by 6 points and fail separately at retrieval.","key_machinery":"The central object is the SKILL.md artifact, a compact procedural document distilled from prior trajectories. The paper's method is a contrastive paired-triple design: for each task, it compares a raw run, a workflow-memory run, and a skill-injected run, and an LLM judge labels each trajectory with one of twelve taxonomy modes grouped into three categories—successful procedural anchoring, execution and verification failures, and invocation or boundary failures. The mechanism labels (procedural_anchor, knowledge_injection, failure_warning, none, counterproductive) make the otherwise black-box question 'why did the skill help?' attributable to a concrete behavioral change.","core_discovery":"The paper's central claim is that the value of skills is not knowledge injection but procedural anchoring: a distilled skill acts as a stable, ordered procedure—setup steps, tool calls, intermediate checks, verification plans—that keeps execution on track. The evidence: procedural_anchor accounts for 65.7% of skill mechanisms versus 4.5% for explicit knowledge injection; skill arms beat workflow memory by 6.06 percentage points on matched tasks; and execution-layer failures (environment setup, output-format mismatches, service-lifecycle failures) drop sharply. Retrieval is a separate bottleneck: as the pool grows from 5 to 100 skills, actual-use precision falls from 29.6% to 3.3% while downstream success stays roughly flat. Exact ground-truth invocation is neither sufficient nor necessary for success, because related non-ground-truth skills can still provide procedural support.","pith_inferences":["If procedural anchoring is the operative mechanism, then cheap proxies such as tool-call sequence stability or verbatim reuse of skill steps could predict skill value better than success rates alone.","The precision collapse from 29.6% to 3.3% suggests that skill libraries should be kept small, or that retrieval should be trained on procedural or execution similarity rather than the semantic embedding similarity the paper uses.","The persistence of algorithmic logic errors and static-verification failures hints that skills may have a ceiling for tasks requiring novel reformulation; a testable extension would compare skill gains against a task novelty score.","The 65.7% anchoring share implies that ablating the procedural sections of a skill—keeping only domain facts—should shrink its value; that direct experiment is not run in the paper."],"forward_implications":["Skills should be treated as procedural stabilizers: the actionable gain is in ordering, checks, and verification, not in adding domain facts.","Retrieval quality and downstream success should be reported separately; a skill library that ranks well offline can still fail at execution time.","Outcome annotations (success/failure labels) matter during skill construction, especially when the trajectory pool includes failed runs.","A skill can hurt even when it is present: misapplication and rigid following create invocation failures that raw execution would not have."],"supporting_citations":[{"why":"Supplies the SKILL.md skill abstraction and the standard format that all skill-arm interventions are built from.","marker":"(Anthropic, 2025)"},{"why":"Supplies Terminal-Bench, one of the two benchmark suites used for the controlled skill-vs-workflow experiments.","marker":"(Merrill et al., 2026)"},{"why":"Supplies SkillsBench with native task–skill annotations, the ground truth for retrieval precision and recall in RQ4.","marker":"(Li et al., 2026)"},{"why":"Supplies the Workflow Memory baseline that skills are compared against in the matched paired-triple comparisons.","marker":"(Wang et al., 2024c)"},{"why":"Supplies Qwen3-Embedding-0.6B, the embedding model used for the Arm 1 retrieval diagnostics and distractor construction.","marker":"(Zhang et al., 2025)"}],"fun_headline_variants":["Skills work as procedural anchors, not knowledge injectors","Procedural anchoring drives 65.7% of skill success","Retrieval precision collapses from 29.6% to 3.3% as pools grow","Skills beat workflow memory by 6 points, but retrieval fails","Exact skill invocation neither sufficient nor necessary"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The taxonomy numbers rest on LLM judgments over truncated trajectory excerpts (heads plus tails), with the full 528 paired triples not independently human-coded; if those judgments systematically misattribute what the skill changed, the procedural-anchoring conclusion is biased.","fun_headline_variants_meta":{"raw":{"variants":["Skills work as procedural anchors, not knowledge injectors","Procedural anchoring drives 65.7% of skill success","Retrieval precision collapses from 29.6% to 3.3% as pools grow","Skills beat workflow memory by 6 points, but retrieval fails","Exact skill invocation neither sufficient nor necessary"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000216,"raw_usage":{"total_tokens":1468,"prompt_tokens":1015,"completion_tokens":453,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":631,"completion_tokens_details":{"reasoning_tokens":364}},"tokens_in":631,"tokens_out":453,"duration_ms":3687,"temperature":1.0,"reasoning_tokens":364,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T21:21:18.716272+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Have two independent human annotators re-label the full 528 paired triples from complete, untruncated transcripts, then recompute the procedural-anchor share; if the share drops below roughly half of the reported 65.7%, or the skill-vs-workflow paired delta loses its positive sign, the central taxonomy claim is not stable.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}