Pith.

Software Engineering · arXiv:2609.07360 v3 · submitted 7 Sep 2026

Scanning the Harness: Configuration Exposures in AI Coding-Agent Supply Chains

Benjamin Kapner, Carmel Soceanu, Alicia Petrunin, Hofni Gartner

The paper says

“AI coding agents rely on repository instructions, skills, hooks, tool-server declarations, and subagent definitions. These artifacts distribute both behavior and access to executable dependencies, making configuration review part of the agent software supply chain.”

more of the abstract

We study 3,171 public GitHub repositories: 2,660 assembled setups and 511 skill collections. Deterministic analysis, mechanical re-derivation, model-assisted adjudication, human review, and platform documentation checks identify six categories of configuration exposure and conformance issues. Unpinned MCP package declarations occur in 9.8% of setups, broad execution grants in 2.5%, and broad skill tool preapproval in 3.8%. Their union covers 409 setups (15.4%); among setups with MCP configuration, 24.5% contain an unpinned declaration. Including required-field and skill-format issues brings the setup rate to 17.9% and the collection rate to 6.8%. Holding the six categories fixed, contextual review changes the setup rate from 18.3% to 17.9%; rule selection explains most of the reduction from the broader candidate set. The findings identify concrete opportunities to pin dependencies, review execution pre approval, and check component conformance before distribution or use. A documented permission exception also exposes a shared error in the scanner and its mechanical audit, motivating version-specific semantic checks. The study provides reproducible evidence about repository declarations; agreement with human judgments informs label validation, while runtime consequences and recall remain unmeasured.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 10 Sep 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. A careful, transparent measurement of a new supply-chain layer; the validation protocol is strong, though the shared-reimplementation and model-adjudication risks are real but disclosed and bounded. This line is wrong

2 major objections · 8 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Section 3 ('Validation') and Section 6

The review did not quote this passage.

Pith says: Objection 1 of 2 · major

The abstract states that the pipeline 'validates every finding before it counts,' and the paper's central claim is that the headline figures are 'confirmed'; for the 586 agreed (repository, rule) pairs, that confirmation rests on two same-team implementations of the same rule documentation, with the stage-three model review agreeing on 99.3% of those pairs and no human having read any of them.

more of objection 1

Section 6's own admission that 'a shared misreading of a format would pass both' accurately describes the residual risk, but the caveat appears only in the threats section, and the sub-class routing that excludes fixtures, templates, and locally pinned packages is admitted to be 'as complete as the cases we knew to write,' so a systematic misreading of a sub-class would shift the agreed pairs (for example, all 258 agreed unpinned-MCP pairs) without ever reaching the adjudicator. I request that the authors either (i) have an independent human labeler, outside the author team, score a random sample of about 10% of the agreed pairs against the released evidence, with per-class agreement reported, or (ii) if that is infeasible before publication, state in the abstract and Section 1 that no verdict has been scored by a human, so that 'confirmed' is read as 'confirmed under the pipeline.'

The paper says, at Section 6 ('stage-three review') and Section 3 ('Validation, stage two')

The review did not quote this passage.

Pith says: Objection 2 of 2 · major

The stage-three model review agreed with the pipeline on only 10 of the 18 adjudicated gating/provisional pairs that touch headline figures, and Section 6 reports that the 8 disagreements were re-examined and corrected by the authors, who are the same team that wrote both implementations and designed the sub-class policy; the stated bound of at most 0.3 percentage points covers …

more of objection 2

… reclassifying templates and fixtures but not the correctness of the remaining verdicts, and the review's overall agreement on the 157 adjudicated pairs is 70.7% with kappa=0.23, making adjudication the least reliable stage of the pipeline. Because these 18 pairs sit directly beneath the security prevalence in the abstract, I ask that the authors obtain an independent human labeling of these 18 pairs using the shipped scoring script and report whether any headline figure changes; if the 0.3-point bound needs revision as a result, the bound and the figures should be updated together.

8 minor objections

The paper says, at Section 1, results paragraph

Pith says: minor objection 1 of 8

The sentence 'against a raw instrument output of 96.8%' contradicts the abstract's 'raw scanner rate of 25.5% on the same rules,' Section 4.1's '25.5% of setups,' and Table 2's 'Raw gate output before audit 25.5%'; if 96.8% refers to a different quantity, such as the output of all instrument rules including heuristics, that quantity should be defined and the sentence reconciled.

The paper says, at Section 3 ('Validation, stage three') and Section 6

Pith says: minor objection 2 of 8

Stage two reports 744 re-derived (repository, rule) pairs, while the stage-three table consistently contains 743 rows; the one-row difference should be explained.

show 6 more

The paper says, at Abstract and Section 4.2

Pith says: minor objection 3 of 8

The 16.0% 'security defect' headline is dominated by the unpinned-MCP class (9.8%), which the paper itself likens to an unpinned build dependency rather than a demonstrated compromise; the abstract should report this class separately from the two pre-approval classes or add a qualifier such as 'including supply-chain-hygiene defects.'

The paper says, at Section 4.2

Pith says: minor objection 4 of 8

The sentence '83 setups carry 161 grant findings; each setup counts once per mechanism, so the mechanisms sum to 159' is terse; since multiple findings per setup are already implied by the 161-to-83 ratio, the sentence should clarify what the 161-to-159 difference represents.

The paper says, at Section 4.7

Pith says: minor objection 5 of 8

The rule is said to fire on six repositories, but the text then describes one (a token-counting routine) and 'the other four' (vendor API key use), leaving the sixth repository's outcome unclear.

The paper says, at Section 4.1

Pith says: minor objection 6 of 8

The comparison with prior per-skill prevalence figures [3, 7] mixes units (per-skill versus per-setup); a sentence stating that the comparison is only order-of-magnitude and not directly commensurable would prevent mis-citation.

The paper says, at Section 4.2

Pith says: minor objection 7 of 8

The two liveness checks (29 of 40 seeded setups; 22 of 65 setups) are used to claim that the flagged grants and servers are actually exercised, but the seeding procedure, inclusion criteria, and confidence intervals are not reported; the sampling basis should be stated.

The paper says, at Abstract

Pith says: minor objection 8 of 8

The joined word 'Thisharness' in the abstract's opening paragraph appears to be a typographical error for 'This harness' and should be corrected.

What would settle it

Have an independent human expert (not an author) review all 18 headline figure pairs (gating and provisional rules that went to adjudication) with access to the repository at its pinned commit, the rule definition, and current platform documentation, blinded to the study's verdict. Record whether each is a defect. Compute the resulting prevalence figures; if they differ from the reported 16.0% security defect rate by more than 0.5 percentage points, the validation pipeline's reliance on models and same-team re-implementation is a significant source of uncertainty.

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 10 Sep 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic agrees 25 s · 17,530 tokens read, 2,789 written
  3. Editor wrote the verdict above 17 s · 18,351 tokens read, 1,778 written
  4. Referee major revision: 2 major objections, 8 minor 3 min 18 s · 17,933 tokens read, 25,205 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

You should know this paper upfront: it is the best empirical study I have seen on the security of AI coding-agent configurations. It measures a real problem—unpinned MCP servers, over-broad permission grants, shell-pre-approving skills—across 3,171 public repositories, and it validates every finding with a two-implementation re-derivation and model adjudication. The artifacts are released. That alone makes it a solid contribution.

What is new: the paper goes beyond prior per-skill analyses by looking at assembled setups and published collections. The headline figure—16.0% of setups carry a confirmed security defect—is clearly stated as a lower bound. The paper is unusually transparent about its failure modes: the threat of shared misreading in the second implementation, the model's imperfect agreement, and the absence of human scoring.

The soft spots are the ones the paper itself flags. The second implementation was written by the same team against the same documentation, so a shared format misreading could pass both. The model adjudication is not an oracle; on the 18 (repository, rule) pairs that touch the headline figure, the two model sessions agreed only 10/18. The authors re-examined and corrected those, but no human has scored any verdict. That said, the impact on the headline figure is likely small: they bound the effect at most 0.3 percentage points for template reclassification, and the agreed pairs (586 of 744) are re-derived by two independent implementations, which is already more rigorous than most measurement studies.

This paper is for security researchers, tool authors, and format stewards. It provides actionable numbers and concrete recommendations. It deserves a serious referee. The authors’ own disclosure of limitations makes the work more trustworthy, not less.

I would accept it for peer review. The conditional verdict from the reader is reasonable, but I lean slightly more positive given the bounded impact of the remaining risks.

The strongest objection

The headline prevalence of 16.0% security defects depends on the correctness of the 18 (repository, rule) pairs for gating and provisional rules that required model adjudication. The paper reports that the stage-three model review agreed with the pipeline on only 10 of these 18 pairs (Section 6). Each of the 8 disagreements was re-examined by the authors and corrected, but no independent human has scored any of them. The paper acknowledges that a model is not an oracle and that the authors are the same team that built the instrument. For an agreed pair (586 of 744), no human has read the evidence at all; the second implementation is from the same team, so a shared misreading of a format would pass undetected. The most load-bearing point is that the final security figure rests on a small set of adjudicated pairs where model judgment was overridden by author judgment, without independent verification. The paper's own estimate (at most 0.3 percentage points shift if templates are reclassified) bounds the potential impact of disagreements on templates, but does not cover all 8 disagreements.

The reader's weakest_assumption is that the validation stack relies on a second implementation by the same team and model adjudication without human scoring for every counted pair. My concern sharpens this to a concrete, load-bearing point: the 18 critical pairs where the two model stages disagree on 8, and those 8 were resolved by the authors rather than an independent human. This is a specific manifestation of the reader's more general concern. The paper is transparent and the likely impact on prevalence is small, so the existing CONDITIONAL verdict remains appropriate. The concrete test proposed (independent human review of those 18 pairs) would definitively settle whether the concern affects the central claim. I agree with the reader that this is the weakest link, and I do not see a stronger concern elsewhere: the deterministic rules, exhaustive re-derivation, and released artifacts all strengthen the case. The concern is real but not fatal; the paper's figures are lower bounds and acknowledged as such.

The referee's report

If the figures hold, this is a solid and useful empirical contribution: it maps a genuinely new supply-chain surface, reports per-class rates that readers can recombine, and demonstrates a validation protocol—deterministic byte-decidable rules, full re-derivation rather than sampling, model adjudication of disagreements, and a second model re-read of every counted pair—that is stronger than the norm for prevalence claims in this area. The released artifacts make every number regenerable and independently checkable, and the explicit lower-bound framing is honest. The paper is unusually transparent about its own failure modes: it discloses the shared-team risk in the second implementation, reports the stage-three review's 10/18 agreement on the 18 adjudicated headline-touching pairs and its low agreement (70.7%, kappa=0.23) on adjudicated pairs generally, and bounds the template-policy sensitivity at 0.3 percentage points.

Pith's reading of the paper (a machine's summary, not the authors')

The paper studies the configuration layer of AI coding agents, called the 'harness,' which consists of instruction files, skills, hooks, MCP server declarations, and subagents installed from marketplaces and public repositories. Through a rigorously validated analysis of 3,171 public GitHub repositories, the authors aim to establish that 16.0% of setups (assemblies of two or more component types) carry a confirmed security defect, with the most common being unpinned MCP servers (9.8%), arbitrary-execution grants that appear scoped (3.1%), and skills that pre-approve shell access (3.8%). The significance lies in demonstrating that this dependency layer lacks fundamental supply-chain hygiene—no lockfiles, no install-time validation, and no mechanism for components to declare what they create—making ordinary drift in configurations a measurable security risk. Every finding was independently re-derived from the repository at its pinned commit, disagreements adjudicated by a language model, and all counted pairs reviewed by a second model session, providing a validated lower bound on defects that prior work had only estimated from raw detector output.

What carries the argument. The harness-eval instrument, an open-source static analysis tool that discovers and parses harness components (CLAUDE.md, AGENTS.md, .mcp.json, SKILL.md, etc.) across 11 AI coding agents, runs 31 deterministic rules that emit structured findings. The central validation protocol has three stages: (1) an independent second implementation, written from the rule documentation alone, re-derives every finding from the repository re-cloned at its pinned commit (8,547 findings re-derived); (2) where the two implementations disagree, a language-model adjudicator with a released prompt reads the evidence and rules on whether the stated consequence holds; (3) a second independent model session re-examines every counted (repository, rule) pair against the repository and current platform documentation, correcting the instrument where the objection holds. Only rules meeting a high agreement bar (≥97% on ≥50 re-derived findings) and a consequence bar (≥80% of pairs confirmed as defects) count as 'gating' rules; the six gating rules yield all headline figures.

Core claim. The central empirical discovery is that 16.0% of the 2,660 setups (95% CI 14.6%–17.4%) and 3.7% of the 511 skill collections carry at least one confirmed security defect, and 16.7% carry any confirmed defect (security or cannot-work). The three surviving security defect classes are: (1) unpinned MCP server packages (9.8% of setups), which fetch unbounded versions from public registries on every session start; (2) arbitrary-execution grants (3.1%) where a permissions.allow entry like Bash(python:*) reads as narrowly scoped but permits unrestricted command execution; and (3) skills whose allowed-tools field pre-approves the shell (3.8% of setups, 3.7% of collections). These defects are ordinary, fixable with a local edit, and traceable to a missing convention in the format or client. Crucially, all gating rules operate at the FILE scope—rules that compare two files or analyze the whole component graph produce findings that are more often intended than not, so the headline figures are a lower bound on what the instrument can express.

The premise it rests on. The second implementation of each rule was written by the same team as the first, against the rules' documentation rather than their code, so a shared misinterpretation of a format or condition would pass undetected through both implementations and would be counted as a confirmed defect without human review.

If the paper is right (Pith's inferences, not the authors')

Marketplaces for AI coding-agent components should implement publication-time checks for skills that pre-approve shell access or violate the specification, catching the 3.7% of collections that ship dangerous allowed-tools.

Clients (e.g., Claude Code, Cursor) should warn or refuse on unpinned MCP server declarations, fo llowing the pattern that pm, PyPI, and Docker established with lockfiles and digests.

The ermission vocabulary should be made explicit: Bash(python:*) should render as "any command" in the permission UI, not as a scoped grant, to eiminate the 3.1% of setups with mislading arbitrary-execution grants.

Team CI/CD pipelines can integrate the released instrument as a SARIF-emitting gate that catches unpinned servers, scoped-ooking grants, and diverged instructions across assistants in under a second per repository.

The zero confirmed credential-exfiltration path suggests that run-time taint analysis, not static config scanning, is needed to detect cross-component data flows; the paper supplies a defined lower bound for that class.

The paper's validation protocol sets a new preedent for empirical studies of software configuration defects: any prevalence rate reported from unudited tool output should be considered a property of the tool, not of the suply chain, until findings have been independently re-derived and their consequences adjudicated. Future studies in other ecosystems (e.g., Terraform modules, Docker compose file

The fact that only FILE-scope rules survived validation suggests that cross-file and setup-scope analyses require conventions that do not yet exist in the harness formats—such as a declared list of outputs or a single source-of-truth across assistants. This points to a design principle for future agent configuration languages: make the analysis predicates align with the structural commitments the

The 18.4% confirmed finding rate (including spec-conformance defects) is an order of magnitude larger than typical vulnerability rates in package-manager ecosystems (npm, PyPI), which hover near 0.1–1%. If this gap is confirmed by independent replication, it indicates that the harness layer is fundamentally higher risk and should receive immediate attention from platform vendors before it becomes

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper

The paper

Abstract

AI coding agents rely on repository instructions, skills, hooks, tool-server declarations, and subagent definitions. These artifacts distribute both behavior and access to executable dependencies, making configuration review part of the agent software supply chain. We study 3,171 public GitHub repositories: 2,660 assembled setups and 511 skill collections. Deterministic analysis, mechanical re-derivation, model-assisted adjudication, human review, and platform documentation checks identify six categories of configuration exposure and conformance issues. Unpinned MCP package declarations occur in 9.8% of setups, broad execution grants in 2.5%, and broad skill tool preapproval in 3.8%. Their union covers 409 setups (15.4%); among setups with MCP configuration, 24.5% contain an unpinned declaration. Including required-field and skill-format issues brings the setup rate to 17.9% and the collection rate to 6.8%. Holding the six categories fixed, contextual review changes the setup rate from 18.3% to 17.9%; rule selection explains most of the reduction from the broader candidate set. The findings identify concrete opportunities to pin dependencies, review execution pre approval, and check component conformance before distribution or use. A documented permission exception also exposes a shared error in the scanner and its mechanical audit, motivating version-specific semantic checks. The study provides reproducible evidence about repository declarations; agreement with human judgments informs label validation, while runtime consequences and recall remain unmeasured.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 0.98

“In total 16.0% of setups carry a security defect and 16.7% a confirmed defect of any kind, against a raw scanner rate of 25.5% on the same rules; the third class ships inside 3.7% of collections, where a marketplace scan can see it.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.05 · states limitation 0.91 · human participants 0.04 · controlled comparison 0.43 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.87 · mentions code or data availability 0.97

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
AI coding agents such as Claude Code, Cursor, GitHub Copilot, and OpenAI Codex are configured through artifacts developers write and share: instruction files, skills, hooks, MCP server declarations, subagents. background 1.00 0.07 0.02 asserted 1.00
This harness is a dependency layer installed from marketplaces and public repositories, running with the developer's privileges, with no lockfile, no install-time check, and no vocabulary for what a component may do. background 0.92 0.13 0.04 asserted 1.00
We study it over 3,171 public GitHub repositories: 2,660 setups that assemble two or more component types and 511 published skill collections. method 0.85 0.92 0.03 asserted 1.00
We measure only rules decidable from bytes whose consequence is a security exposure, a configuration that cannot work, or a departure from the Agent Skills specification, and validate every finding before it counts: an independent implementation re-derives it from the repository at its pinned commit, a language-model adjudicator with a released prompt rules on every disagreement, and a second independent model session re-checks every counted pair. method 1.00 0.96 0.05 asserted 1.00
Three security classes survive: 9.8% of setups install an MCP server with no version pinned, 3.1% pre-approve arbitrary execution behind a scoped-looking grant such as Bash(python:*), and 3.8% carry a skill that pre-approves the shell for whoever installs it. result 1.00 0.96 0.31 asserted 1.00
In total 16.0% of setups carry a security defect and 16.7% a confirmed defect of any kind, against a raw scanner rate of 25.5% on the same rules; the third class ships inside 3.7% of collections, where a marketplace scan can see it. result 1.00 0.96 0.42 asserted 1.00
Rules that compare two files detect differences that are usually intended and are reported as observations. limitation 0.64 0.86 0.03 asserted 0.87
No credential-exfiltration path was confirmed. result 0.99 0.92 0.07 asserted 1.00
The instrument, corpus manifest, prompt, and every verdict are released. contribution 0.84 0.83 0.03 asserted 1.00

What kind of paper the classifier reports this is: empirical study 1.00 · evidence from analysis of existing data 0.58

What was examined, and what was not

Sources read

  • arXiv listing abstract v1

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 7,188 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (1)
Figure 1 of the paper
Figure 1. Prevalence over the 6 gating rules. Left: setups and published collections with any confirmed defect (security or

Figures are the authors', reproduced from the paper.

Cited by 3 Pith papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment