{"id":"f63d79f1-09d9-440d-b95f-ec47479e444a","arxiv_id":"2505.04606","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"OmniGIRL introduces a 959-instance multilingual, multimodal GitHub issue resolution benchmark; the best result, GPT-4o with Agentless-X, resolves only 8.6% of issues.","lead":"OmniGIRL is a new benchmark of 959 GitHub issue-resolution tasks covering Python, JavaScript, TypeScript, and Java, plus 19 issues that require reading images. On it, the strongest configuration tested, GPT-4o with Agentless-X, resolves only 8.6% of issues, showing that multilingual and multimodal issue resolution is still largely unsolved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark validity depends on unvalidated Docker environments and FAIL2PASS tests; without independent verification, the headline 8.6% and image-subset 10.5% rates are not trustworthy.","rationale":"The reader's weakest_assumption identifies the execution-based verification pipeline as the load-bearing premise; I agree. The paper's headline numbers are only as credible as the FAIL2PASS judgments and Docker reproducibility, and the paper provides no independent validation of either. This is not a hypothetical issue: existing SWE-bench-style pipelines are known to contain tests that pass for incidental reasons, and the paper's own §6 acknowledges filtering limitations without quantifying their effect. The image-subset result compounds the problem: 10.5% is 6/57 runs, not 6/19 issues, so the multimodal claim is even less robust. These are correctable issues, so the verdict should remain conditional rather than reject, but the benchmark's central measurement must be independently verified before the specific rates are used.","tokens_in":162,"tokens_out":5984,"duration_ms":67680,"concrete_test":"Independently rebuild 20 random instances per language from the released base commits and Dockerfiles, reproduce the FAIL2PASS transitions claimed in §3.4.3 (tests fail before gold patch, pass after), and have two developers blind to the PR assess whether each FAIL2PASS test specifically targets the issue. If the independent rebuild disagrees on more than 10% of instances, or if either developer judges a FAIL2PASS test as not issue-related in more than 10% of cases, the headline rates should be re-derived after fixing the pipeline.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the execution-based verification pipeline in Sections 3.4.1-3.4.3. The pipeline assumes: (a) Docker environments built from README instructions reproduce the exact pre-fix state; (b) the FAIL2PASS tests extracted from merged PRs encode the target issue and fail for issue-related reasons before the gold patch; and (c) the gold patch's test-passing behavior is solely attributable to the fix. The paper provides no evidence for these assumptions: no environment build success rate, no count of instances discarded at each filtering stage, no inter-annotator agreement for the manual FAIL2PASS check in §3.4.3, and no human-validated subset analogous to SWE-bench Verified. Since every resolve rate, including the headline 8.6%, is computed from these FAIL2PASS judgments, a failure of any one assumption directly invalidates the benchmark's core measurement. This is a concrete risk: SWE-bench's same pipeline has been criticized for tests that pass for unrelated reasons and for environment reproducibility issues. The paper itself admits in §6 that filtering 'may filter out some instances' and that test-file detection 'could miss files,' but it does not quantify how often the retained FAIL2PASS tests might be wrong. The image-subset result is an additional interpretive problem: §5.3 reports 10.5% as 6/57 runs rather than unique resolved issues out of 19, so the multimodal claim rests on at most a handful of distinct successes. Both issues need addressing before the numbers can be cited.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces OmniGIRL, a GitHub issue-resolution benchmark of 959 instances drawn from 15 repositories across Python, JavaScript, TypeScript, and Java, with a manually filtered 19-instance subset containing images that are essential for resolving the issue. The construction pipeline follows SWE-bench: merged pull requests are filtered by issue references and changed test files, task instances are assembled with base commits, test patches, gold patches, and FAIL2PASS/PASS2PASS annotations, and Docker-based execution is used to verify environments and filter instances. The paper evaluates GPT-4o, Claude-3.5-Sonnet, and DeepSeek-V2.5 using oracle retrieval, Agentless-X, and AutoCodeRover-X, reporting a best resolve rate of 8.6% (GPT-4o with Agentless-X) on the full benchmark and 10.5% (Claude-3.5-Sonnet with oracle retrieval and image-augmented text) on the 19-image subset. It also presents failure analyses of parsing errors in Agentless-X and of poor performance on multi-file issues.","tokens_in":24066,"tokens_out":3156,"duration_ms":30869,"significance":"If the benchmark's execution-based verification is sound, OmniGIRL is a useful contribution: it extends issue-resolution evaluation beyond Python-only SWE-bench, covers diverse domains and languages, and introduces a small but genuinely multimodal subset. The evaluation is carried out with reasonable, established baseline methods, and the code and data are open-sourced. The headline result that current LLMs resolve only 8.6% of issues is plausible and consistent with the broader SWE-bench literature. The failure analysis on prompt-format sensitivity and multi-file edits is informative. The main significance hinges, however, on demonstrating that the FAIL2PASS tests and Docker environments actually encode each issue, and on clarifying the statistical basis of the multimodal results; these points currently receive insufficient evidence.","major_comments":[{"comment":"The validity of every reported resolve rate depends on the assertion that FAIL2PASS tests extracted from merged PRs fail before the gold patch and pass after it for issue-related reasons, and that Docker environments faithfully reproduce the pre-fix state. The paper does not report the number of PRs collected, the number removed at each attribute filter, the environment build success rate, the number of instances discarded because the gold patch did not produce a FAIL2PASS test, or any measure of inter-annotator agreement for the manual FAIL2PASS check in §3.4.3. The paper's own §6 acknowledges that the path-keyword test-file detection 'could miss files' and that filtering 'may filter out some instances,' but it does not quantify how often the retained tests could be wrong. I ask the authors to provide a full filtering funnel and a human-validated subset (analogous to SWE-bench Verified) or an independent verification study on a random sample, since without this the 8.6% headline is not yet trustworthy.","section":"§3.4.1–3.4.3"},{"comment":"The image-subset results report percentages over 57 runs for 19 unique instances (three repetitions each), so the 10.5% best result corresponds to only 6/57 runs and may represent as few as two distinct resolved issues (if all three runs of two instances resolved). With 19 instances, each unique resolved instance changes the rate by approximately 5.3 percentage points, making the multimodal claim extremely sensitive to a handful of successes. Please report the number of unique resolved instances and their instance IDs for each setting, and discuss the per-instance run-to-run agreement.","section":"§5.3, Table 9"},{"comment":"There is an internal inconsistency in the central failure-analysis statistic: the text of §5.4 states Claude-3.5-Sonnet's parsing success rate is 11.8%, while Table 10 reports 10.0% (96/959). These cannot both be correct, and the discrepancy directly affects the claimed cause of the 1.9% resolve rate. Please reconcile the numbers and verify the corresponding apply rates (52.2% in Table 10 vs. 52.0% in Table 8 for DeepSeek-V2.5 Agentless-X).","section":"§5.4 and Table 10"},{"comment":"Table 8 contains internally inconsistent percentages that must be corrected: for Claude-3.5-Sonnet with Agentless-X on Python, the Apply Rate is shown as '1.6% (55/374)', but 55/374 is 14.7%, not 1.6%; the Java row also contains a typo ('4/105|'). Because apply rate is one of the three evaluation metrics, these errors need to be fixed and all percentages recomputed from the raw counts before publication.","section":"Table 8"}],"minor_comments":[{"comment":"The modality distribution states that 4.1% (39/959) of instances contain image information, but Table 2 shows that after filtering only 19 instances remain with images judged necessary. Please clarify whether the 4.1% figure refers to the pre-filtering count and report the post-filtering distribution (19/959 ≈ 2.0%) consistently.","section":"§4.2.3"},{"comment":"There are several typos and misspellings: 'intergrated' in §5.1.3, 'AutoCoderOver' in the §5.2 text, 'Statitics modeling' in Table 6, 'preformance' in the RQ1 summary, and the duplicated '(d)' label in Figure 2. These should be cleaned up.","section":"Throughout"},{"comment":"The abbreviation 'IAG-Text' is defined in the table caption as 'Image-augmented Text', but the caption for Table 9 says 'Image-augmented Text' is shortened to 'IAG-Text' while the table header uses 'IAG-Text'; please use the abbreviation consistently and define it explicitly the first time it appears in the text.","section":"Table 9"},{"comment":"The oracle retrieval baseline is described as providing the exact files from the gold patch, but the paper does not specify whether the oracle baseline also receives the exact function or line locations; please clarify the input granularity, since it affects the interpretation of the 7.8% result.","section":"§5.1.2"},{"comment":"Table 1 lists SWE-bench-java as having 91 instances, while Section 2.3 also says 91; however, the abstract-level related-work text in Section 1 states 93 instances. Please ensure the counts are consistent.","section":"§2.3 and Table 1"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid benchmark-construction effort with a plausible central result, but its validity currently rests on unquantified pipeline assumptions. The internal inconsistencies in Tables 8 and 10 and the run-based (rather than instance-based) multimodal statistics are concrete, fixable issues. If the authors provide the filtering funnel, environment build statistics, a human-validated sample, and corrected numbers, the paper could be suitable for publication. I do not see grounds for rejection, but the load-bearing verification evidence is not yet in the manuscript."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a solid SWE-bench extension, not a breakthrough. The benchmark itself is a useful resource, but the headline numbers in the current draft have a few inconsistencies and the image-subset rate is described in a way that overstates what was measured.\n\nWhat's genuinely new: 959 instances across Python, JavaScript, TypeScript, and Java from 15 repos, plus a small hand-filtered image subset (19 instances) and website-link annotations. That's a real addition to a field otherwise dominated by Python-only SWE-bench variants. The construction pipeline follows SWE-bench closely, and the authors are transparent about the steps: attribute filtering, Docker setup, FAIL2PASS selection, manual checks on images and links. They also open-source the collection code. The failure analysis is useful, especially the finding that Claude-3.5-Sonnet's structured-output parsing failures crush its Agentless-X performance and that a one-line prompt fix recovers most of the gap.\n\nWhere it's soft. First, there are internal numeric inconsistencies. The parsing success rate for Claude is 11.8% in the text and 10.0% in Table 10; the apply rate for Claude with Agentless-X is 14.6% in Table 8 and 17.6% in Table 10; and some Table 8 cells look like typos (e.g., the Python apply rate of \"1.6%\" is actually 55/374 ≈ 14.7%). These need to be reconciled before the paper is citable. Second, the image-subset result is reported as 6/57 runs = 10.5%, but 57 is 19 issues × 3 repetitions. Per-run resolve rate is not the same as per-issue resolve rate; if the same issue resolved in multiple runs, the per-issue rate is lower. The authors should report how many distinct issues were resolved. Third, they don't report how many instances were discarded at each filtering stage, nor the Docker build success rate, nor inter-annotator agreement for the manual FAIL2PASS check. That's a transparency gap. It doesn't sink the benchmark—SWE-bench has the same unvalidated assumptions—but it makes the exact numbers hard to trust without more detail. The stress-test's worry that FAIL2PASS tests might not encode the issue is real but speculative; the manual check provides some mitigation.\n\nOverall: the central claim, that current LLMs resolve under 9% of these multilingual issues, is credible and consistent with prior work. This is worth a serious review. I'd recommend sending to peer review with a request to fix the numbers and clarify the image-subset metric.","headline":"A useful SWE-bench extension to multiple languages and a small image subset, but the current draft has internal numeric inconsistencies and an over-stated image-subset rate that need fixing before the numbers are citable.","tokens_in":24639,"tokens_out":3171,"would_cite":true,"duration_ms":29544,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The OmniGIRL benchmark shows that current LLMs resolve only 8.6% of real multilingual GitHub issues.","keywords":["GitHub issue resolution","multilingual benchmark","multimodal benchmark","large language models","execution-based verification","cross-file code changes","image understanding","software engineering agents"],"falsifier":"Run each designated fail-to-pass test suite multiple times on the untouched base-commit environment from the released data. If any of those tests ever passes before the gold patch is applied, or fails after it, that instance does not cleanly encode the issue and the affected resolve rates are over- or understated.","tokens_in":23579,"feed_emoji":"🐙","tokens_out":10719,"duration_ms":87734,"temperature":0.7,"pith_summary":"The paper introduces OmniGIRL, a benchmark for automatic GitHub issue resolution built from 959 real resolved issues across four programming languages, fifteen repositories, and eight domains. The authors want to show that current large language models are far weaker at this task than single-language, text-only benchmarks suggest. On the full benchmark the strongest configuration, GPT-4o with an adapted workflow called Agentless-X, resolves 8.6% of issues. When an issue description includes an image that is essential to the fix, the best reported resolve rate is 10.5%, achieved by Claude-3.5-Sonnet with oracle file retrieval and image-augmented text over three repeated runs. The paper also isolates two recurring failure modes: models that do not emit parseable localization output, and models that edit only one file when the gold fix requires changes across several files.","feed_headline":"Best LLM resolves only 8.6% of real GitHub issues","feed_subtitle":"OmniGIRL spans four languages and image-bearing issues; the 8.6% top score leaves room to grow.","key_machinery":"The central object is the OmniGIRL task instance: a real merged pull request paired with its linked issue text, optional images and website links, a base commit, a gold patch, and a set of fail-to-pass tests that must turn from failing to passing after the fix is applied. The construction pipeline filters merged pull requests by issue linkage and test-file changes, then verifies each instance in a Docker environment built from the base commit, retaining only instances with at least one fail-to-pass test and manually checking that those tests match the pull request's test patch. The evaluation machinery compares three baselines (oracle retrieval, a hierarchical localization-then-repair workflow, and an agentic search-and-repair loop) on each instance. Its work is to convert 'issue resolved' into an executable, reproducible pass/fail check, and the paper's numbers inherit all their meaning from that conversion.","core_discovery":"OmniGIRL's central claim is that current LLM-based issue resolution degrades sharply as soon as evaluation leaves the single-language, text-only setting: on 959 validated instances from Python, JavaScript, TypeScript, and Java repositories, the best tested model resolves 8.6% of issues, and on the 19 instances whose images are essential, the best resolve rate is 10.5%. The paper argues this is not simply a retrieval problem, because giving models the exact files to edit (oracle retrieval) improves some results, and because rewriting image content into text raises Claude-3.5-Sonnet's image-subset rate from 3.5% to 10.5%. It attributes the failures to concrete causes: parse-format errors in the localization stage, which a one-line prompt constraint fixes for Claude-3.5-Sonnet (1.9% to 7.4%), and a systematic single-file editing bias on fixes that require multi-file changes. The implied conclusion is that multilingual and multimodal issue resolution remains largely unsolved and needs new mechanisms rather than prompt tweaks alone.","pith_inferences":["A direct validity check a reader could run on the released data is to execute each fail-to-pass suite several times on the untouched base commit; any test that sometimes passes before the fix would inflate every reported resolve rate.","The visual subset is only 19 instances (57 runs per setting), so the 10.5% figure is directionally suggestive but not a precise capability estimate.","Because the benchmark annotates website links but does not evaluate them, a natural next experiment is to give a repair agent browsing access to those links and measure the gain over text-only and image-augmented inputs.","The same image-augmented-text rewrite could be transferred to other multimodal code-repair benchmarks to test whether the gain is specific to this dataset or a general property of current vision-language models."],"forward_implications":["If the 8.6% full-benchmark rate holds, claims of software-engineering agent competence should be scoped to single-language, text-only benchmarks until multilingual evidence is available.","Multimodal issue resolution is a distinct bottleneck; the jump from 3.5% to 10.5% on the image subset with image-augmented text suggests converting visual content into text is a cheap and effective first step.","Prompt robustness can move a model's resolve rate by a factor of about four: adding an explicit format constraint raised Claude-3.5-Sonnet from 1.9% to 7.4%.","Cross-file issue resolution is a separate weakness; even with oracle file retrieval, models modify only a single file in roughly 73% to 86% of cross-file cases.","Syntax-tree-based localization favors languages whose structure is class- and method-centric (Python, Java) and underperforms on JavaScript and TypeScript, where interfaces and anonymous functions carry much of the structure."],"supporting_citations":[{"why":"Supplies the collection, attribute-filtering, and execution-based verification pipeline that OmniGIRL extends to four languages.","marker":"[27]"},{"why":"Provides the prior single-language extension to Java whose scope OmniGIRL generalizes.","marker":"[59]"},{"why":"Provides the human-verified subset and leaderboard context used to justify the evaluation setup.","marker":"[38]"},{"why":"Defines the hierarchical localization-and-repair workflow that the paper adapts into Agentless-X.","marker":"[52]"},{"why":"Defines the agentic search-and-repair method adapted into AutoCodeRover-X.","marker":"[63]"},{"why":"Supplies the oracle-retrieval evaluation idea and a multi-agent issue-resolution baseline.","marker":"[47]"},{"why":"Used to select high-performance, low-cost baselines from the issue-resolution leaderboard.","marker":"[15]"},{"why":"Identifies GPT-4o, one of the three evaluated models and the full-benchmark best performer.","marker":"[37]"},{"why":"Identifies Claude-3.5-Sonnet, the best performer on the image-bearing subset.","marker":"[12]"}],"fun_headline_variants":["OmniGIRL: LLMs resolve only 8.6% of GitHub issues","GPT-4o tops but fails 91.4% on OmniGIRL benchmark","Image issues trip up LLMs in multilingual benchmark","OmniGIRL exposes big shortfall in GitHub issue fixing"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the fail-to-pass tests extracted from each merged pull request actually fail before the fix, pass after it, and fully capture the reported issue; if any test is flaky, environment-dependent, or unrelated to the issue text, every resolve rate in the benchmark is unreliable.","fun_headline_variants_meta":{"raw":{"variants":["OmniGIRL: LLMs resolve only 8.6% of GitHub issues","GPT-4o tops but fails 91.4% on OmniGIRL benchmark","Image issues trip up LLMs in multilingual benchmark","OmniGIRL exposes big shortfall in GitHub issue fixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000263,"raw_usage":{"total_tokens":1654,"prompt_tokens":1051,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":667,"completion_tokens_details":{"reasoning_tokens":521}},"tokens_in":667,"tokens_out":603,"duration_ms":5445,"temperature":1.0,"reasoning_tokens":521,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T23:23:57.653764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run each designated fail-to-pass test suite multiple times on the untouched base-commit environment from the released data. If any of those tests ever passes before the gold patch is applied, or fails after it, that instance does not cleanly encode the issue and the affected resolve rates are over- or understated.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the human-verified subset and leaderboard context used to justify the evaluation setup."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Used to select high-performance, low-cost baselines from the issue-resolution leaderboard."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies GPT-4o, one of the three evaluated models and the full-benchmark best performer."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies Claude-3.5-Sonnet, the best performer on the image-bearing subset."}],"review_version":1}