CI-Repair-Bench: A Repository-Aware Benchmark for Automated Patch Validation via CI Workflows

Referee: Abstract and benchmark construction: The description of the 567 instances provides no details on selection criteria, filtering process, or analysis of potential selection bias across the 103 repositories; this directly affects the generalizability of the reported 18.9% success rate and the claim that the benchmark reflects typical real-world CI issues.

Authors: We agree that additional details on the benchmark construction are necessary to support claims of representativeness. While the manuscript describes the collection of CI failure instances from GitHub Actions workflows across 103 repositories and their categorization into 12 error types, we did not provide a dedicated analysis of selection criteria or potential biases. In the revised version, we will expand Section 3 to include explicit selection criteria (e.g., repositories with active CI, failure logs containing specific error patterns), filtering steps (such as excluding incomplete workflows or non-reproducible failures), and an analysis of repository characteristics (e.g., size, language distribution) to address generalizability concerns. This will better contextualize the 18.9% success rate. revision: yes

Referee: Evaluation section (empirical results): The central claim that full CI re-execution under original workflows provides an unbiased and complete correctness signal is not supported by any reported controls for workflow non-determinism, such as multiple re-executions per patch, flakiness detection, or handling of cached artifacts/external dependencies; a single run per patch risks noise that could inflate or deflate the 18.9% figure.

Authors: This point highlights an important limitation in our evaluation methodology. Our approach prioritizes realism by re-executing patches in the exact original CI environment and workflow, which we believe provides a more complete signal than simplified test suites. However, we did not implement controls such as repeated executions to detect flakiness or account for non-determinism from external dependencies and caches. We will revise the evaluation section to explicitly discuss this limitation, including potential impacts on the reported success rates, and suggest best practices for benchmark users, such as running evaluations multiple times where feasible. We maintain that for many deterministic error types (e.g., linting failures), the single-run evaluation is reliable, but acknowledge the risk for others. revision: partial

Referee: Results and discussion: No statistical tests (e.g., confidence intervals or significance testing) are mentioned around the 18.9% rate or the per-error-type breakdowns, which weakens the fine-grained claims about which error types are more amenable to automated repair.

Authors: We concur that the absence of statistical analysis limits the strength of our claims regarding differential performance across error types. In the revised manuscript, we will add confidence intervals (using the Wilson score interval for binomial proportions) for the overall 18.9% success rate and for each of the 12 error types. Additionally, we will include a brief discussion of the statistical significance of observed differences where appropriate, to better support the fine-grained analysis. revision: yes