{"id":"b4f24d76-a977-43b4-a5f7-8d0988257a11","arxiv_id":"2609.06052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SkillSpec detects defects in AI agent skills by extracting expected and factual specifications from documentation and code through masked intent views, then validating discrepancies in a sandbox, confirming 763 defects across 515 skills.","lead":"SkillSpec checks AI agent skills by comparing what a skill's documentation claims it does with what its instructions and code actually do. Applied to 515 real-world skills it found 763 confirmed defects in 239, and the authors release the tool and the defect dataset.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"Manual confirmation of validated defects is performed by the paper's own authors without external blind review or inter-annotator agreement metrics, risking confirmation bias in the central precision claim.","rationale":"The reader's verdict correctly identifies the intent-faithfulness assumption as a key weakness, but the manual-labeling process is a distinct load-bearing link in the empirical chain. The paper's transparency is commendable, yet the central precision claim is only as strong as the ground-truth labels. Our proposed test directly assesses the reliability of those labels. If the test reveals low agreement or a significantly different confirmation rate, the claim requires adjustment (possibly REJECT). If agreement is high, the current CONDITIONAL verdict stands. We recommend UNCHANGED because the reader already conditioned the verdict on additional evidence; our test is a concrete instance of what that evidence should look like. We differ from the reader's weakest assumption focus but acknowledge the reader also noted the labeling issue in the rationale.","tokens_in":25783,"tokens_out":3882,"duration_ms":38034,"concrete_test":"Select a random subset of 200 validated defects from the GPT-5.6-Sol results across both datasets. Have three independent annotators (not authors, familiar with agent skills) re-label each defect using the same 5-point scale, blinded to the system's verdict. Compute Fleiss' kappa for multi-rater agreement and compare the confirmed defect set to the original. If kappa < 0.6 or if the independent confirmation rate (defects with mean >= 4) differs from the original by more than 10 percentage points, the precision claim is not robustly supported by the manual review process.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of 61.2% precision depends on the manual confirmation of 763 validated defects by two of the paper's own authors. Although they evaluate independently, they are not blind to the fact that these are outputs of their own system. The paper does not report inter-annotator agreement (e.g., Cohen's kappa) or compare against an independent external label set. Given the inherent ambiguity in classifying intent–behavior discrepancies (as acknowledged by the 'unsure' category and the taxonomy of false positives), the manual labels could be influenced by prior knowledge of the system's design and expected outcomes. Without external validation, the precision figure may overstate the system's true ability to identify real-world skill defects.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper presents SkillSpec, a framework for detecting behavioral defects in agent skill repositories. It converts heterogeneous skill artifacts (markdown descriptions, scripts) into a unified graph, derives expected specifications (ExpectSpec) from surrounding declared intent and factual specifications (FactSpec) from encoded behavior under four intent-masked views (self, neighbors, lineage, holistic), flags discrepancies as candidate defects, and validates each candidate in an isolated sandbox. On 515 real-world skills from SkillsBench and a curated SkillsTop dataset, SkillSpec identifies 763 manually confirmed defects across 239 skills, achieving 61.2% precision. The paper provides a defect taxonomy, ablation studies, robustness experiments across five LLM backends, and a cost analysis.","tokens_in":25995,"tokens_out":2205,"duration_ms":22579,"significance":"If the results hold, SkillSpec offers a principled, scalable approach to quality assurance for an emerging class of software artifacts (agent skills). The unified graph representation and intent-masked multi-view reasoning are novel contributions that bridge natural-language intent and code-level behavior. The large-scale evaluation (515 repositories, multiple backends) provides empirical evidence that defects are prevalent at intent-implementation boundaries and that specification reasoning can detect them with moderate precision. The study establishes a baseline for future work on skill correctness, automated repair, and safe agent orchestration.","major_comments":[{"comment":"The central precision claim (61.2%) relies on manual labels assigned by the paper's own authors (two authors independently rated each validated defect on a 5-point scale). Inter-annotator agreement (e.g., Cohen's kappa) is not reported. Given the subjective nature of classifying intent-behavior discrepancies (especially the 'unsure' category), and the fact that the authors are not blind to the system's outputs, the precision figure may be inflated by confirmation bias or annotation disagreements. The paper should report agreement statistics and, ideally, include an independent external annotator or a held-out gold-standard set. Without these, the main claim is not fully supported.","section":"§V-A (Metrics) and §VII-A (Construct Validity)"},{"comment":"The assumption that 'the intent declared in a SKILL.md file faithfully reflects the user's true intent' is foundational to the entire pipeline. While the authors acknowledge this limitation in §VII-A, it means that any defect signal rooted in ExpectSpec derives from an oracle that may be incomplete, aspirational, or stale. The paper does not quantify how often this assumption fails or provide a sensitivity analysis. This limits the construct validity of the defect detection, as a mismatch between factual behavior and a flawed expected specification could be a false positive. A discussion of the empirical scope of this threat (e.g., manual inspection of a random sample of SKILL.md files for accuracy) would strengthen the evaluation.","section":"§III (Specification for Skills)"}],"minor_comments":[{"comment":"The paper is well-structured and the contributions are clearly enumerated. However, the text is occasionally verbose (e.g., §VI-A and §VI-B could be condensed).","section":"General"},{"comment":"The visibility-layer semantics table (Table I) is helpful, but the distinction between Lineage and Neighbors for code nodes could be clarified with an example, as the current description ('ancestor definitions along the resolved call path' vs. 'immediate caller and sibling callees') may be ambiguous for methods with multiple callers.","section":"Table I"},{"comment":"The workflow DAG distinguishes structural and operational units, but the description of how 'context' units are identified and how their content is used in specification extraction is vague. Provide an example of a context unit and its role in ExpectSpec or FactSpec derivation.","section":"§IV-A2"},{"comment":"The defect taxonomy in Table IV is informative, but the classification was performed automatically using an 'LLM-as-a-judge' protocol. No validation (e.g., human agreement on a sample) of this automatic categorization is reported. The paper should assess the accuracy of this classification, as it underpins the lifecycle analysis.","section":"Table IV (Defect taxonomy)"},{"comment":"The discussion of the 'unsure' label states it 'reflects inherent ambiguity ... rather than annotator disagreement.' This claim would be substantiated by reporting exact agreement (not just mean-score thresholds) and by showing the distribution of scores per annotator. Without that data, the statement is speculative.","section":"§VII-A (Construct Validity)"},{"comment":"The bar chart for validation outcomes (Figure 9) would be easier to read if the 'Validated' and 'Refuted' bars were colored consistently across models, and if the unresolved counts were annotated with percentages for clarity.","section":"Figure 9"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"Hi,\n\nThe short version: SkillSpec does something genuinely new—treating agent skill correctness as a specification consistency problem and building a multi-view, intent-masked pipeline to detect defects. The paper is internally consistent, unusually honest about its limitations, and releases a 763-defect corpus. But the headline precision number (61.2%) is softer than it reads: the ground truth is labeled by the paper's own authors without inter-rater agreement or external blind review, there is no baseline comparison against the closest prior work (ref [10]), and recall is not measured. These are addressable issues, and the core approach is worth engaging.\n\nWhat's actually new: The intent mask with four visibility layers (self, neighbors, lineage, holistic) and the bidirectional binding between workflow and code graphs. That design is not present in prior work. The ablation study shows that multi-view reasoning beats any single view by a few points and uncovers defects no single view catches. The robustness across five LLM backends (GPT-5.6, DeepSeek-V4, Qwen) is a nice result: code consistency is high everywhere, plain text remains the bottleneck.\n\nWhat the paper does well: The numbers cross-check (I verified the 763 = 346 + 417, the 6,933 candidates sum, per-backend row counts). The threats section (Sec VII) is candid about the limits: no formal guarantee, reliance on LLM-generated specs, ambiguous manual labels, no recall. The authors explicitly concede that the method \"cannot determine whether it fully captures the author's latent intent.\" That level of honesty is rare and should be acknowledged. The case studies are concrete and illustrative (e.g., the Python 0.0-as-falsy bug).\n\nThe soft spots: The stress-test concern is valid. Two of the paper's own authors manually confirm validated defects on a five-point scale. They do not report Cohen's kappa, and they are not blind to the fact that these are outputs of their own system. The paper's own false-positive taxonomy (Table X) shows over-inference and unreachable triggers as the main sources—exactly the kinds of errors a biased labeler might miss. Independent labeling or at least inter-rater statistics would substantially strengthen the claim. Second, no comparison against the closest prior detector (ref [10], semantic fuzzing for specification violations). Without that, we cannot assess relative performance. Third, the cost ($2.585/skill, mostly on GPT-5.6) is nontrivial for broader adoption, though the paper notes local deployment can reduce it.\n\nSummary: This paper is a solid, well-documented systems contribution for the emerging problem of agent skill correctness. It deserves peer review and should be accepted with major revisions requiring independent manual validation, inter-rater agreement reporting, and a baseline comparison. The central idea—specification reasoning with controlled intent visibility—is publishable and will be cited.","headline":"Novel specification reasoning for agent skills, but precision headline is softer than it reads due to self-labeled ground truth and no baseline comparison.","tokens_in":26623,"tokens_out":2337,"duration_ms":22369},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"SkillSpec catches intent–implementation mismatches in agent skills by treating correctness as specification consistency.","keywords":["agent skills","specification reasoning","Hoare logic","intent mask","skill defects","LLM agents","software correctness","intent–implementation consistency"],"falsifier":"Construct a skill where the SKILL.md accurately describes the behavior, but the code contains a subtle logical error (e.g., a silent infinite loop or an off-by-one bug that only manifests under rare inputs). If SkillSpec fails to flag this as a defect because the ExpectSpec–FactSpec comparison yields no inconsistency, then the framework cannot detect correctness violations that do not correspond to an intent–implementation mismatch.","tokens_in":27279,"feed_emoji":"🔍","tokens_out":2299,"duration_ms":22828,"temperature":0.7,"pith_summary":"Agent skills bundle natural-language instructions with code, but defects often hide as silent failures masked by the LLM's capabilities. SkillSpec reformulates skill correctness as a specification-reasoning problem: it derives an expected specification (ExpectSpec) from the declared intent and multiple factual specifications (FactSpec) from the encoded behavior under varying degrees of context masking, then compares them to flag candidate defects. On 515 real-world skills, SkillSpec identified 763 manually confirmed defects across 239 skills, achieving 61.2% precision. The approach is most reliable for code nodes and breaks down at the boundaries between declared intent and implementation, showing that explicit specifications can serve as practical quality contracts for agent ecosystems.","feed_headline":"One framework catches 763 skill defects by checking intent against code","feed_subtitle":"Treats agent skill correctness as specification consistency, achieving 61.2% precision on real-world skills.","key_machinery":"The intent mask is the central mechanism: it controls how much surrounding declared intent is exposed during factual specification extraction, producing four visibility layers (self, neighbors, lineage, holistic) that balance the bias from excessive context against the risk of unsupported inference from insufficient context. Multi-view specification reasoning then compares ExpectSpec (derived from full intent, excluding the target node itself) against each FactSpec to identify persistent inconsistencies that are robust across views and likely to represent genuine defects.","core_discovery":"SkillSpec shows that agent skill defects can be systematically detected by treating the skill as an executable artifact with intrinsic specifications: the declared intent in SKILL.md defines an expected specification, while the workflow structure and code artifacts define a factual specification. By building a unified graph that aligns these heterogeneous artifacts, and by using an intent mask to produce multiple factual views at different context granularities, SkillSpec jointly reasons over ExpectSpec and FactSpecs to localize candidate defects. The paper's central empirical finding is that 46.4% of real-world skills contain at least one confirmed defect, most of which arise from deviations between what the skill claims to do and what it actually does, rather than from conventional coding errors.","pith_inferences":["Because SkillSpec's ExpectSpec oracle depends on the assumption that the SKILL.md intent is faithful, the framework is vulnerable to ambiguous, aspirational, or stale documentation. A practical extension would cross-reference user reported issues or usage logs to validate the intent against actual usage patterns.","The multi-view reasoning strategy is transferable to other domains where specifications are scattered across heterogeneous artifacts, such as API documentation paired with SDK code or regulatory compliance documents tied to implementation.","The observed defect prevalence (46.4% of skills) suggests that the current skill ecosystem has a substantial quality gap. A natural next step is to feed SkillSpec's defect reports into automated repair loops, where an LLM patches the code or revises the documentation to restore consistency.","Since code defects are validated more reliably than workflow defects, a cost-effective deployment could first run the code analysis and defer the more ambiguous workflow analysis to a manual triage step."],"forward_implications":["SkillSpec provides an automated quality-assurance pipeline for agent skill marketplaces, enabling maintainers to flag skills whose behavior diverges from their descriptions without manual code review.","The finding that most defects occur at the intent–implementation boundary suggests that skill-authoring tools should enforce tighter traceability between natural-language claims and executable code.","Code-level defects are validated with high precision (67.1%), making SkillSpec a practical complement to existing testing regimes for script-based skills.","Plain-text nodes remain a bottleneck, meaning that skills without executable code will require more sophisticated natural-language understanding before automated quality checks become reliable."],"supporting_citations":[{"why":"Supplies the SkillsBench dataset, a manually curated collection of agent skills used as part of the evaluation corpus.","marker":"[5]"},{"why":"Establishes the Hoare logic foundation that SkillSpec adapts for natural-language and code specifications.","marker":"[27]"},{"why":"Provides a prior example of LLM-based Hoare-style reasoning scaled to large systems, which SkillSpec extends to heterogeneous skill artifacts.","marker":"[20]"},{"why":"The skills.sh repository is the primary source for the SkillsTop dataset, representing widely downloaded community skills.","marker":"[23]"},{"why":"OpenCode is used as the agent harness for sandbox validation, providing the runtime environment for executing candidate defect probes.","marker":"[33]"},{"why":"Tree-sitter is used to parse concrete syntax trees for code-graph construction; its declarative grammar specifications enable multilingual support.","marker":"[29]"}],"fun_headline_variants":["Spec reasoning catches 763 defects hidden in agent skills","SkillSpec checks intent against code to find 763 defects","SkillSpec frames skill correctness as specification consistency","Intent-masked specs expose silent agent skill failures","SkillSpec: 763 confirmed defects at 61.2% precision"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The entire defect detection chain assumes that the intent declared in the SKILL.md file faithfully reflects the user's true intent; if the description is ambiguous, aspirational, or stale, the expected specification is corrupted before any comparison happens.","fun_headline_variants_meta":{"raw":{"variants":["Spec reasoning catches 763 defects hidden in agent skills","SkillSpec checks intent against code to find 763 defects","SkillSpec frames skill correctness as specification consistency","Intent-masked specs expose silent agent skill failures","SkillSpec: 763 confirmed defects at 61.2% precision"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000252,"raw_usage":{"cost":0.00025158,"is_byok":false,"cost_details":{"upstream_inference_cost":0.00025158,"upstream_inference_prompt_cost":0.00015778,"upstream_inference_completions_cost":9.38e-05},"total_tokens":1462,"prompt_tokens":1127,"completion_tokens":335,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":0,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":271}},"tokens_in":1127,"tokens_out":335,"cost_source":"openrouter_usage","duration_ms":4794,"temperature":1.0,"reasoning_tokens":271,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-09T01:47:55.886140+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-alltime-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Construct a skill where the SKILL.md accurately describes the behavior, but the code contains a subtle logical error (e.g., a silent infinite loop or an off-by-one bug that only manifests under rare inputs). If SkillSpec fails to flag this as a defect because the ExpectSpec–FactSpec comparison yields no inconsistency, then the framework cannot detect correctness violations that do not correspond to an intent–implementation mismatch.","supporting_citations":[{"cited_title":"SkillsBench: Benchmarking How Well Agent Skills Work Across Diverse Tasks","cited_arxiv_id":"2602.12670","evidence_quote":"Supplies the SkillsBench dataset, a manually curated collection of agent skills used as part of the evaluation corpus."},{"cited_title":"/definitions.dfy","cited_arxiv_id":null,"evidence_quote":"Establishes the Hoare logic foundation that SkillSpec adapts for natural-language and code specifications."},{"cited_title":"FM-Agent: Scaling Formal Methods to Large Systems via LLM-Based Hoare-Style Reasoning","cited_arxiv_id":"2604.11556","evidence_quote":"Provides a prior example of LLM-based Hoare-style reasoning scaled to large systems, which SkillSpec extends to heterogeneous skill artifacts."},{"cited_title":"skills.sh: The agent skills directory,","cited_arxiv_id":null,"evidence_quote":"The skills.sh repository is the primary source for the SkillsTop dataset, representing widely downloaded community skills."},{"cited_title":"OpenCode: The open source ai coding agent,","cited_arxiv_id":null,"evidence_quote":"OpenCode is used as the agent harness for sandbox validation, providing the runtime environment for executing candidate defect probes."},{"cited_title":"tree-sitter: A parser generator tool and an incremental parsing library,","cited_arxiv_id":null,"evidence_quote":"Tree-sitter is used to parse concrete syntax trees for code-graph construction; its declarative grammar specifications enable multilingual support."}],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}