{"id":"3ee61920-4e29-459e-ac16-49e58724d06a","arxiv_id":"2505.22583","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"GitGoodBench introduces a benchmark with 900 evaluation, 120 lite, and 17,469 training samples for three Git scenarios, with a 21.11% GPT-4o baseline solve rate on lite.","lead":"GitGoodBench is a new benchmark for testing AI agents on version control tasks, covering merge conflicts, interactive rebasing, and iterative committing. It ships three datasets and a GPT-4o baseline that solves only 21% of tasks, showing current agents struggle with real Git workflows.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"IR/ICC solve rates hinge on an unvalidated LLM-as-a-judge; the paper's own Section 7 concedes the bias risk, so the reported 26.67%/23.33% rates are not yet objective measures of Git competence.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern: the IR and ICC solve rates rest on an LLM judge whose reliability and bias are unvalidated. My stress-test agrees with this read and sharpens it: the solved criterion is preference of the judge over the original history, and the judge is not calibrated to human judgments. The paper acknowledges this in Section 7, which strengthens rather than weakens the concern. The benchmark itself remains a real contribution: the datasets are released, the scenario construction is documented, and MCR has an exact-match ground truth. The issue is confined to the quantitative baseline claim for FCC samples. Because the reader already issued a CONDITIONAL verdict for exactly this reason, my read does not change the verdict. If the proposed human-panel test shows low agreement or materially different solve rates, then the appropriate outcome would be to report FCC rates as preliminary only, or to require an ensemble-judge protocol before treating the baseline as a reference point.","tokens_in":14018,"tokens_out":3443,"duration_ms":46221,"concrete_test":"Using the released evaluation data for the 60 FCC samples in GitGoodBench Lite, take each agent-generated history and the original history and obtain labels from (a) the current LLM judge, (b) a panel of at least three human developers, and (c) two other LLM judges (e.g., Claude and Gemini). Compute pairwise agreement (Cohen's kappa) and recompute IR and ICC solve rates under each label source. If human-labeled solve rates differ from the reported 26.67% and 23.33% by more than 10 percentage points, or if human-judge agreement falls below 0.6 kappa, the reported FCC rates should not be treated as objective baseline numbers.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative result — GPT-4o solving 26.67% of IR and 23.33% of ICC scenarios — depends entirely on the LLM-as-a-judge protocol in Section 3.3, because there is no deterministic ground truth for \"better Git history.\" The judge is given subjective criteria (commit-message quality, logical cohesion, progression, commit size) and is, at least in family, the same model that produced the agent trajectories. A sample is counted as solved only if the judge prefers the agent history over the original history in both position-swapped runs; this controls position bias but not preference bias. If the judge shares GPT-4o's stylistic preferences, or systematically overweights one criterion, a solved sample reflects judge preference rather than Git competence. The paper itself flags this in Section 7: \"our evaluation methodology may introduce bias, as it is LLM-based.\" No calibration against human ratings, no inter-judge agreement statistic, and no sensitivity analysis are reported. Thus the headline 21.11% solve rate and the FCC scenario rates are not yet established as objective reference numbers. The benchmark and dataset contributions can still stand, but the validity of the baseline evaluation is unresolved.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces GitGoodBench, a benchmark for evaluating AI agents on version-control (Git) tasks, covering three scenario types: Merge Conflict Resolution (MCR), Interactive Rebase (IR), and Iterative Committing of Changes (ICC). The benchmark is constructed by mining permissive open-source Python, Java, and Kotlin repositories, with a comprehensive suite (900 samples), a rapid-prototyping Lite set (120 samples), and a training corpus (17,469 samples). The authors present a baseline using GPT-4o with custom tools on the Lite set, achieving an 88% success rate and a 21.11% overall solve rate, with MCR evaluated by exact match and IR/ICC evaluated by an LLM-as-a-judge protocol that compares agent-generated vs. ground-truth Git histories. The paper also documents data filters, stratification, and a difficulty heuristic for MCR based on conflict counts and distribution, while acknowledging that the proposed FCC purity-based difficulty heuristic for IR/ICC does not correlate with empirical solve rates.","tokens_in":14278,"tokens_out":4832,"duration_ms":57576,"significance":"GitGoodBench addresses a real gap: existing SE agent benchmarks such as SWE-bench focus on issue resolution and do not evaluate VCS-specific workflows like interactive rebase, merge-conflict resolution, or iterative committing. The dataset construction is careful and externally grounded in real repositories, with transparent filters, stratification, and per-sample metadata; the release of a training corpus and a Lite/Full split is a useful contribution. If the evaluation methodology is validated, the benchmark could meaningfully advance research on VCS-aware agents. The baseline results, however, are only indicative: the solve rates for IR and ICC rest entirely on an unvalidated LLM judge, and the reported numbers come from a single run without error bars. These issues must be resolved before the quantitative claims can be taken as established reference points.","major_comments":[{"comment":"The IR and ICC solve rates (26.67% and 23.33%) are computed via an LLM-as-a-judge protocol, but the judge is never validated against human judgment, no inter-judge agreement is reported, no sensitivity analysis with alternative judge models is given, and the paper does not even specify which LLM served as the judge. The paper's own Section 7 concedes that this 'evaluation methodology may introduce bias, as it is LLM-based.' Because these two scenario types constitute two-thirds of the benchmark's evaluations, the headline 21.11% overall solve rate is not yet an objective measure of Git competence. The authors should calibrate the judge against human ratings on a subsample, report agreement statistics, test at least one other judge model, and state the judge model and decoding parameters explicitly.","section":"§3.3, §7"},{"comment":"The FCC purity-based difficulty heuristic is explicitly acknowledged not to correlate with empirical solve rates: Table 4 shows easy IR samples with a 13.33% solve rate while medium and hard samples have 31.82% and 30.43% respectively, i.e., the 'easy' label corresponds to the worst performance. Since this heuristic is used for stratification and is included in the dataset metadata, the difficulty labels for FCC-based scenarios are not valid and should not be presented as meaningful. The authors should either revise the heuristic, relabel the samples as provisional, or remove the difficulty field from the released FCC samples.","section":"§3.2, §A.2.3, §5, Table 4"},{"comment":"All baseline results are based on a single run per scenario, with no error bars or repeated trials. GPT-4o trajectories are stochastic; a single sample can be solved in one run and not another, so the reported 21.11% overall solve rate and the per-scenario rates in Tables 2–4 lack an estimate of variability. The authors should report aggregated results over at least a few repeated runs (e.g., 3 runs) with per-scenario and overall variance, or explain why the evaluation is deterministic despite the stochasticity of the underlying model.","section":"§5"}],"minor_comments":[{"comment":"The command 'git show –remerge-diff' uses an en dash; it should be 'git show --remerge-diff'.","section":"§3.2"},{"comment":"The paper does not state which LLM model was used as the judge in the LLM-as-a-judge protocol, nor the temperature/sampling parameters for either the agent or the judge; this information is needed for reproducibility.","section":"§3.3, §4"},{"comment":"The overall solve rate of 21.11% is computed over 180 scenario-evaluations (60 FCC samples evaluated once as IR and once as ICC, plus 60 MCR samples), while the Lite set contains 120 unique samples. The paper should clarify this double-counting of the FCC samples in the overall rate, or report a per-unique-sample rate in addition.","section":"§5, Table 2"},{"comment":"The statement that 'Easy samples have a ≈3 times higher solve rate than hard samples' is inaccurate because the hard sample solve rate is 0%; the ratio is undefined, not three.","section":"§5"},{"comment":"There is a typo in 'a approximately 60% improvement' in the Introduction.","section":"§1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark construction and dataset release are solid contributions, and the paper is honest about the limitations it identifies. However, the central quantitative claims for two of the three scenario types depend on an unvalidated LLM judge, and the authors' own acknowledged failure of the FCC difficulty heuristic undermines part of the dataset's metadata. These are load-bearing issues for a benchmark paper, not mere presentation concerns. A major revision that validates the judge, fixes or removes the FCC difficulty labels, and reports variability of the baseline would make the paper acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"GitGoodBench is the first benchmark we actually have for end-to-end evaluation of AI agents on Git workflows, and that alone is a real contribution. The three scenario types—merge conflict resolution, interactive rebase, iterative committing—cover genuinely different VCS skills, and the authors mine real permissive repositories, apply sensible filters, stratify for language and repository diversity, and release the data, including a 17k-sample training split. MCR evaluation uses exact-match against the ground-truth merge commit, which is deterministic and hard to game. The paper also ships its prompts and raw evaluation data, so the community can reproduce and extend the baseline.\n\nThe soft spots are concentrated in the measurement of the two FCC-based scenarios. The IR and ICC solve rates (26.67% and 23.33%) come from an LLM-as-a-judge comparing agent histories against the original history. The judge is given subjective criteria, and it is the same model family that generated the trajectories. The paper does the position-swap control, but that only handles position bias, not preference bias. There is no calibration against human raters, no inter-judge agreement, no sensitivity analysis. The authors themselves concede in Section 7 that the evaluation \"may introduce bias, as it is LLM-based.\" So the headline solve rates for IR and ICC are not yet objective reference numbers. The MCR numbers are on firmer ground, but 120 samples with no error bars or multiple runs means they are also just a baseline sketch.\n\nThe failed FCC difficulty heuristic is honestly reported, which is good, but it means the difficulty distribution of the FCC samples is currently unvalidated. That matters for interpreting aggregate results, though it doesn't undermine the benchmark's basic usability.\n\nIn short: the resource is real and useful. The baseline evaluation methodology needs work—human-judge validation on a subset, multiple runs, error bars, and a sensitivity check on the judge—before the solve rates should be quoted as stable. This is a solid benchmark paper that deserves a serious referee. I would engage with it.","headline":"First real end-to-end Git-agent benchmark with released data; useful resource, but the LLM-judged solve rates for IR/ICC are provisional until validated.","tokens_in":14787,"tokens_out":1881,"would_cite":true,"duration_ms":22217,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GitGoodBench presents a benchmark that measures AI agents on Git workflows—merge conflict resolution, interactive rebase, and iterative committing—and a baseline model solves only 21.11% of the prototype tasks.","keywords":["benchmark","Git","AI agents","version control","merge conflict resolution","interactive rebase","LLM-as-a-judge","software engineering agents"],"falsifier":"Take a random sample of the 60 FCC samples from the Lite set and have the two histories (agent-generated and ground truth) rated by experienced developers; if the LLM judge agrees with the human majority in fewer than roughly 70% of cases, or if agreement varies strongly with the choice of judge model, the reported 26.67% and 23.33% solve rates are not a stable measure of Git competence.","tokens_in":13837,"feed_emoji":"🔀","tokens_out":7700,"duration_ms":81354,"temperature":0.7,"pith_summary":"GitGoodBench is a benchmark that tests AI agents on version-control workflows rather than just code fixing. It covers three scenarios extracted from open-source Python, Java, and Kotlin repositories: resolving merge conflicts, rewriting commit history with interactive rebase, and turning scattered changes into a clean sequence of commits. The benchmark provides a 900-sample full evaluation set, a 120-sample rapid-prototyping set, and a 17,469-sample training corpus. A baseline agent built on a frontier language model with custom Git tools solves 21.11% of the prototype tasks, showing that Git competence is a largely unsolved part of software-engineering automation. If the benchmark is right, evaluating agents on Git tasks will be a necessary step toward comprehensive software-engineering agents.","feed_headline":"New Git benchmark: AI agents solve only 21% of tasks","feed_subtitle":"Coding benchmarks skip version control; this one tests merge, rebase, and commit workflow skills.","key_machinery":"The central objects are the File-Commit Chain (a sequence of consecutive commits modifying a single file, used as a heuristic to find histories that could be improved) and the LLM-as-a-Judge evaluation protocol (an LLM compares agent-generated and ground-truth histories twice with positions swapped, scoring on commit-message quality, cohesion, progression, and commit size). These support the two FCC-based scenarios; Merge Conflict Resolution instead uses exact match against the ground-truth merge commit. The benchmark also provides three datasets and a difficulty heuristic for merges based on the number of conflicts and their spread across files.","core_discovery":"The paper claims to introduce the first benchmark that evaluates AI agents on multiple end-to-end Git scenarios, rather than on isolated sub-tasks. In the Merge Conflict Resolution scenario the agent must reproduce the ground-truth merge commit when given a set of conflicts; in Interactive Rebase it must produce an improved history from existing commits; in Iterative Committing of Changes it must turn a disorganized set of changes into a coherent history. The benchmark is built by mining repositories with permissive licenses, using File-Commit Chains as a heuristic for histories that could be improved, and stratifying samples by scenario type, language, and difficulty. Baseline results with a GPT-4o-based system equipped with custom tools show a 21.11% solve rate overall, with exact-match scoring for conflicts and LLM-as-a-judge scoring for histories; performance drops sharply on Merge Conflict Resolution samples as difficulty increases, with a 0% solve rate on the hardest stratum.","pith_inferences":["The authors leave implicit that the LLM-judge step could be a major source of variance; an ensemble of judges or human ratings would be a direct test.","A natural next benchmark axis would be diagnostic workflows such as git bisect, which the paper mentions only as future work.","A way to separate model reasoning from tool design would be to compare the custom scaffold against a generic Git protocol adapter, something the paper notes it did not investigate.","If Git proficiency becomes a standard axis, composite evaluations that pair code-fixing and history-quality scores would better reflect real developer workflows."],"forward_implications":["Git-specific tooling and evaluation become a recognized part of agentic software engineering, complementary to code-fixing benchmarks.","Current agents fail on interactive Git workflows such as rebase and partial staging, so new agent designs must handle delayed feedback and structured state.","The 21.11% baseline on the Lite set gives a concrete target: any proposed Git agent should report success on the same 120 samples.","The Train split (17,469 samples) is intended for collecting agent trajectories, enabling smaller models to be trained on Git tasks.","The MCR difficulty heuristic is validated by the baseline: easy samples solve at 22.58%, medium at 7.69%, and hard at 0%."],"supporting_citations":[{"why":"Defines the standard coding-agent benchmark that GitGoodBench positions itself as complementing.","marker":"Jimenez et al., 2024"},{"why":"Supplies the LLM-as-a-Judge methodology used to score the ICC and IR histories.","marker":"Zheng et al., 2023"},{"why":"Provides the SEART repository-mining tool used to gather the open-source data.","marker":"Dabic et al., 2021"},{"why":"The GPT-4o system that is evaluated as the baseline agent.","marker":"OpenAI et al., 2024b"},{"why":"Provides the Git-workflow framework used to motivate which scenarios are core.","marker":"Cortés Ríos et al., 2022"}],"fun_headline_variants":["Git benchmark exposes AI's blind spot: version control","Coding benchmarks skip Git; new one reveals 21% solve rate","GitGoodBench: AI agents fail hardest at merge conflicts","Version control is AI's coding gap: 21% solve rate","New benchmark tests Git skills, not just code: AI scores 21%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The solve rates for Interactive Rebase and Iterative Committing rest on the assumption that an LLM judge can reliably and unbiasedly tell which of two Git histories is higher quality; the authors acknowledge in Section 7 that this may introduce bias.","fun_headline_variants_meta":{"raw":{"variants":["Git benchmark exposes AI's blind spot: version control","Coding benchmarks skip Git; new one reveals 21% solve rate","GitGoodBench: AI agents fail hardest at merge conflicts","Version control is AI's coding gap: 21% solve rate","New benchmark tests Git skills, not just code: AI scores 21%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000507,"raw_usage":{"total_tokens":2440,"prompt_tokens":884,"completion_tokens":1556,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":500,"completion_tokens_details":{"reasoning_tokens":1467}},"tokens_in":500,"tokens_out":1556,"duration_ms":12276,"temperature":1.0,"reasoning_tokens":1467,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T13:03:02.884894+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of the 60 FCC samples from the Lite set and have the two histories (agent-generated and ground truth) rated by experienced developers; if the LLM judge agrees with the human majority in fewer than roughly 70% of cases, or if agreement varies strongly with the choice of judge model, the reported 26.67% and 23.33% solve rates are not a stable measure of Git competence.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the LLM-as-a-Judge methodology used to score the ICC and IR histories."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the SEART repository-mining tool used to gather the open-source data."}],"review_version":1}