CrossCommitVuln-Bench: A Dataset of Multi-Commit Python Vulnerabilities Invisible to Per-Commit Static Analysis

Referee: Abstract: The claim that the results 'confirm that snapshot-based SAST tools often miss vulnerabilities whose introduction spans multiple commits' is not supported by the reported measurements. The 15 CVEs were explicitly selected under the filter that each commit is individually benign to per-commit analysis; the resulting 13% per-commit and 27% cumulative rates are therefore expected by construction for this sample and cannot be extrapolated to unselected vulnerabilities without a described sampling frame from the broader CVE population.

Authors: We agree with the referee that the selection criteria make the low detection rates expected for this specific sample, and we do not claim to have a representative sampling frame from all CVEs. The benchmark is curated to highlight cases where vulnerabilities are introduced across commits in a way that evades per-commit analysis. We will revise the abstract to replace 'confirming' with 'illustrating' and clarify that the results demonstrate the potential for such invisibility in multi-commit scenarios, without implying a general prevalence. This revision will be made in the next version of the manuscript. revision: yes

Referee: Dataset Construction and Annotation sections: The manual annotation of commit chains and evasion rationales lacks any reported protocol details, number of annotators, inter-annotator agreement metrics, or blinded validation steps. Because the 'benign' labels directly determine the 87% invisibility figure and the two reported detections (one on a security-fix commit, one partial), this omission is load-bearing for the central empirical claims.

Authors: The referee correctly identifies a gap in the reporting of our annotation process. While the annotations were performed manually by the authors using a consistent schema for identifying contributing commit chains and evasion rationales, the manuscript does not detail the protocol. We will add a new subsection under Dataset Construction that describes the annotation methodology, including the steps for identifying multi-commit chains, the rationale categories used, and the process for validating the 'benign' labels against the CVE descriptions. Although formal inter-annotator agreement metrics were not computed (as the work was conducted by a small team with iterative review), we will describe the reconciliation process to enhance transparency and allow readers to evaluate the reliability of the 87% figure. revision: yes

Referee: Evaluation section: The paper states that both per-commit detections are qualitatively poor, yet provides no details on tool configurations, rule sets, severity thresholds, or false-positive handling for Semgrep and Bandit. Without these, the 13% and 27% rates cannot be independently verified or compared to other studies.

Authors: We acknowledge that the current manuscript lacks the necessary details on the SAST tool configurations to enable full reproducibility and comparison. We will revise the Evaluation section to include a comprehensive description of the setups: for Semgrep, the specific rules (e.g., from the python.security and python.lang.security categories), command options, and how alerts were filtered; for Bandit, the enabled tests, severity levels, and confidence thresholds. Additionally, we will explain the qualitative assessment criteria used to deem the detections 'poor' and how false positives were considered in the cumulative mode. These additions will be provided in the revised manuscript and accompanying code release to support independent verification. revision: yes