Harness Engineering for Agentic AI Coding Tools: An Exploratory Study

Referee: The description of how the 2,853 repositories were identified and sampled (search terms, inclusion/exclusion criteria, GitHub API or search filters) is absent or insufficiently detailed. This is load-bearing because the central claim that Context Files dominate and are often the sole mechanism could be an artifact of conditioning the corpus on the presence of the very filenames being measured.

Authors: We agree that the sampling procedure must be described in full detail. The 2,853 repositories were obtained via the GitHub Search API using queries for the presence of specific filenames (e.g., filename:AGENTS.md, filename:.cursorrules, filename:AGENT.md and equivalents for the other tools), restricted to public, non-forked repositories with at least one commit in the prior 12 months. Inclusion required at least one matching configuration file; we excluded archived repositories and those whose primary language was not a programming language. We will add a dedicated subsection to the Empirical Study section that lists the exact search strings, API parameters, total hits before filtering, deduplication steps, and inclusion/exclusion criteria. We will also explicitly discuss the sampling bias the referee correctly identifies as a limitation and note that our prevalence figures are conditional on the presence of at least one configuration artifact. revision: yes

Referee: No details are supplied on how the eight mechanisms were systematically identified, how repositories were classified into mechanisms, inter-rater reliability, or any validation of the classification scheme. Without these, the reported adoption rates and the distinction between static vs. executable Skills cannot be assessed for reliability.

Authors: The eight mechanisms were first enumerated by systematically reviewing the official documentation and configuration examples published by each tool vendor, followed by an exploratory scan of 50 high-star repositories to confirm the mechanisms in practice. Repository classification combined automated filename and content heuristics with manual review: two authors independently coded a stratified random sample of 200 repositories, reaching 91% raw agreement (Cohen’s κ = 0.87). Disagreements were resolved by joint discussion and the final coding rules were documented. For Skills we distinguished static instruction files from executable scripts by inspecting file extensions and content (presence of shebang lines or code blocks). We will insert a new Methods subsection that fully documents the identification process, the coding protocol, the inter-rater statistics, and the precise criteria used to separate static versus executable Skills. revision: yes

Referee: The quantitative statements about dominance (e.g., Context Files as sole mechanism in many repositories) lack accompanying counts, percentages, or breakdowns by tool that would allow readers to judge effect sizes and to verify that the patterns survive controls for sampling bias.

Authors: We will augment the Results section with a new table (and accompanying text) that reports exact counts and percentages for every mechanism, the proportion of repositories in which Context Files are the sole mechanism, and tool-specific breakdowns (e.g., percentage of Claude Code repositories using only Context Files versus those using additional mechanisms). We will also add a short discussion of how the observed patterns relate to the sampling frame. These additions will supply the numerical detail needed to assess effect sizes and will be accompanied by a limitations paragraph addressing sampling bias. revision: yes