{"id":"df6d6f8d-cc52-44c0-8454-e874b721735f","arxiv_id":"2607.15205","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MM-IssueLoc is a controlled multimodal repository-level issue-localization benchmark showing current systems achieve at most 38.96% file Acc@5 and that visual evidence helps only unevenly.","lead":"This paper introduces MM-IssueLoc, a benchmark of 652 real GitHub issue–pull-request pairs with images, designed to measure whether visual evidence helps localize the code that must change. It evaluates agents and retrievers and finds the best file-level accuracy is only 38.96%, showing multimodal issue localization is far from solved.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"PR-diff gold labels are unvalidated; incidental edits in merged PRs could make Acc@K measure patch alignment rather than localization skill, undermining the 'visual evidence as controlled variable' claim.","rationale":"The reader identified the same weakest assumption: merged-PR edit scope is used as gold without validation. This is the most load-bearing concern because all metrics, ablations, and cross-benchmark conclusions derive from this gold. Other issues—lack of confidence intervals, missing release artifacts—are real but secondary; they affect interpretation or reproducibility rather than the fundamental validity of the benchmark's target. The paper's own §6 admission supports the concern. My proposed audit is a concrete, feasible check that would either confirm the gold labels are clean or show that the headline scores need substantial qualification. Since this aligns with the reader's CONDITIONAL verdict and strengthens the case for requiring annotation-quality evidence, no change to that verdict is needed.","tokens_in":20821,"tokens_out":4165,"duration_ms":39418,"concrete_test":"Sample 50 instances (25 human-annotated, 25 AI-annotated). Have two independent annotators, blind to the paper's system rankings, mark each edited file/function as 'required for the fix' or 'incidental' (refactor, rename, docs, generated, unrelated). Compute the proportion of instances with at least one incidental gold entry. Then recompute OpenHands GPT-5.2 File@5 and MM-IssueLoc-VL-Emb-8B Func@10 after restricting the gold set to required entries only, using the same released harness and tie-handling. If either score shifts by more than 3 points, or if the with-image minus text-only difference changes sign for any model, the headline conclusions are not robust to gold-label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The benchmark's core validity rests on treating merged-PR edit scope as the ground-truth localization target (§3.1). The gold file set F*_i and function set G*_i are exactly the files/functions edited by the fixing pull request; they are extracted from unified diffs and tree-sitter matches, with no human verification that each gold entry is necessary for the fix. The human annotation pipeline (§3.2) validates image category and relevance, not the gold labels. With the strict all-gold Acc@K metric (Eq. 4), a single incidental edit—a mass rename, a lockfile update, an unrelated refactor, or a generated-file change—causes an otherwise correct localization to score 0 on that instance. The paper concedes 'such labels are not minimal causal explanations' (§6), but it does not quantify how often PR diffs contain such noise. If incidental edits correlate with repositories, languages, or image categories, then every headline comparison—text-only vs. with-image, cross-benchmark transfer, difficulty buckets—may reflect patch-alignment skill rather than localization skill, so the central claim that MM-IssueLoc 'turns visual evidence into an explicit evaluation variable' is not yet established.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces MM-IssueLoc, a repository-level issue-localization benchmark designed to make visual evidence an explicit evaluation variable. It contains 652 real issue-PR instances (1,050 images, 23 languages), with file-level gold labels for all instances and function-level labels for 343, plus per-image category/relevance annotations and harmful-image controls. The paper evaluates LLM-based agents and retrieval-based systems under four input modes (text-only, with-image, VCE, VCE+image), including a newly trained controlled retriever MM-IssueLoc-VL-Embedding. Results show a large capability gap: the best agent achieves 38.96 file Acc@5 and 22.45 function Acc@10; the best retriever achieves 33.86 function Acc@10. The paper also reports cross-benchmark comparisons indicating that strong text-only localization performance on SWE-bench does not transfer to multimodal localization, and concludes that visual evidence can help but is used unevenly by current systems.","tokens_in":21112,"tokens_out":3419,"duration_ms":32784,"significance":"If the benchmark is valid, it addresses a real gap: existing SWE benchmarks either strip images or evaluate end-to-end repair, making it impossible to isolate whether visual evidence helps localization. MM-IssueLoc provides a multi-lingual, multi-granularity dataset with per-image annotations, paired input modes, harmful-image stress tests, and a controlled retriever probe. The pipeline is described in sufficient detail to be reproducible, and the authors release per-instance predictions, which is a strength. The paper also makes a falsifiable claim—current systems localize multimodal issues poorly—that seems robust to moderate label noise. However, the central validity of the benchmark rests on unvalidated PR-diff gold labels, and all headline numbers are single-run without variance or significance testing. These issues need to be addressed before the benchmark can fully support the paper's 'controlled variable' claim.","major_comments":[{"comment":"The gold file/function sets F*_i and G*_i are exactly the files/functions edited by the merged human PR, with no human verification that each gold entry is necessary for the fix. The paper acknowledges in §6 that 'such labels are not minimal causal explanations,' but this is load-bearing for the benchmark's central claim that it 'turns visual evidence into an explicit evaluation variable.' If merged PR diffs contain incidental edits (mass renames, lockfile updates, unrelated refactors, generated-file changes), then the strict all-gold Acc@K metric (Eq. 4) penalizes correct localization on those instances, and the measured effect of images may reflect patch-alignment skill rather than localization skill. This concern applies to every headline comparison, including the cross-benchmark transfer results. Please quantify the incidence of such incidental edits (e.g., by human review of a rando","section":"§3.1 and §6"},{"comment":"All performance numbers are single-run, with no variance, confidence intervals, or significance tests. This is particularly problematic for the RQ2 conclusion that visual evidence is useful: the retriever gains of 4.91 and 4.44 File@5 points after removing images could be within run-to-run noise, and the agent-level Δimg values (−2.66, −0.31, +0.92) are even smaller. Similarly, in Table 2 the 1.21-point difference between MM-IssueLoc-VL-Emb-8B and Qwen3-VL-Emb-8B on Func@10 (33.86 vs 32.65) is interpreted as an advantage, but no significance is reported. Since LLM-based agents are stochastic and the benchmark size (652/343 instances) is modest, please report at least three seeds (or bootstrap intervals) for the main tables, or explicitly justify why single-run comparisons are stable for this setting.","section":"§5.2, Table 3"},{"comment":"The VCE diagnostic table shows very large variance across frameworks and backends (ρ_pix ranges from −11.69 to +4.74). The claim that 'raw images do not consistently add value once VCE is provided' is based on single-run differences that overlap substantially in magnitude with the observed framework-level variation. This may be a presentation issue if the authors add significance testing, but currently the raw-pixel residual claim is not supported by the evidence as presented. Please provide error bars or a clearer statistical treatment, and temper the conclusion accordingly.","section":"§5.3, Table 4"}],"minor_comments":[{"comment":"The abstract and Figure 1 say '23 programming languages,' while §3.4 and the 'benchmark statistics' sentence say '24 programming languages.' Please reconcile.","section":"Abstract vs §3.4"},{"comment":"The text says 650 repository snapshots for 652 instances; please clarify whether two instances share a snapshot or whether the counts differ for another reason.","section":"§3.2"},{"comment":"Model names are abbreviated inconsistently (MM-IssueLoc-VL-Emb vs MM-IssueLoc-VL-Embedding, Qwen3-VL-Emb vs Qwen3-VL-Embedding). Please standardize.","section":"Table 2 / Table 3"},{"comment":"The Radford et al. reference contains a typo ('PmLR' should be 'PMLR'). Please proofread the reference list.","section":"References"},{"comment":"Panel C labels 'OpenHands harness' and 'Mini-SWE-Agent harness' are confusing: the x-axis is valid submission rate and the y-axis is File@5. Consider a clearer caption describing that these are two different agent frameworks with the same backend.","section":"Figure 3"}],"recommendation":"major_revision","confidential_remarks":"The benchmark is potentially valuable and the construction pipeline is detailed. The main concerns are (1) the unvalidated PR-diff gold labels, which directly affect the benchmark's claim to control visual evidence, and (2) the complete absence of variance/significance information for single-run LLM comparisons. Both are fixable within the manuscript's scope: add a gold-label noise analysis and add multiple runs or confidence intervals. I do not see a fundamental correctness error that would justify rejection, but the paper needs these additions before the central claims are fully supported."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is the first benchmark I know that treats visual evidence in repo-level issue localization as a controlled variable rather than as part of end-to-end repair. The central finding—current systems are weak at this, with best file Acc@5 below 39%—is probably robust. But the paper’s fine-grained comparisons are thinner than the headline, and the PR-diff gold labels need validation before the benchmark fully earns its claim.\n\nWhat’s new: paired text-only/with-image evaluation, per-image relevance and category labels, harmful-image controls, file/function golds, and a VCE side channel that separates image content from raw pixels. The construction is detailed and mostly honest. The AI-assisted annotation is marked and sanity-checked against the human subset. The controlled retriever is trained on disjoint data, so there is no circularity in the benchmark claim. The limitations list is candid, including the §6 admission that merged-PR labels are not minimal causal explanations.\n\nSoft spots, in order. First, all headline numbers are single-run. Differences like 33.86 vs 32.65 Func@10 or 32.82 vs 32.06 could easily be noise; there are no confidence intervals or significance tests. That matters because the paper leans on a few points to say visual evidence helps. Second, the gold labels are exactly the files/functions edited by the merged PR. The stress-test concern is real: strict all-gold Acc@K scores an instance 0 whenever any gold entry is missed, so one incidental edit—a mass rename, lockfile change, or unrelated refactor—turns a correct localization into a failure. The paper acknowledges this but does not quantify how often PR diffs contain such noise. If that noise correlates with image category or language, the category and cross-benchmark comparisons could partially measure patch alignment rather than localization skill. This does not sink the central capability gap, but it should be addressed with a human audit of gold entries and perhaps a relaxed metric. Third, annotation quality: no inter-annotator agreement is reported for the 450 human instances, and the AI subset is checked only by consistency between two VLMs, not by human verification. Fourth, synthetic harmful images are fine as a stress test, but they should not be read as natural frequency.\n\nThe paper deserves a serious referee. I would ask for variance/confidence reporting, gold-label validation, and a released eval harness with a commit hash. If those land, I’d cite it and bring it to the reading group.","headline":"A genuinely useful controlled benchmark for visual evidence in issue localization, with a real gold-label caveat and missing variance; send it to referees.","tokens_in":21608,"tokens_out":2586,"would_cite":true,"duration_ms":24334,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A controlled benchmark for repository-level issue localization shows that visual evidence carries real signal but that current systems use it unevenly, with the best agent reaching only 38.96 file Acc@5 and the best retriever 33.86 function","keywords":["issue localization","multimodal","visual evidence","repository-level benchmark","retrieval","LLM agents","software engineering","Acc@K evaluation"],"falsifier":"Train the paper's controlled retriever on the same training rows with all images stripped, and run it on the 343-instance function subset; if its function Acc@10 matches or exceeds the multimodal retriever's 33.86, then the benchmark's 'visual signal' is actually explainable by text cues, refuting the claim that images carry distinct localization information.","tokens_in":20717,"feed_emoji":"🖼️","tokens_out":8726,"duration_ms":63618,"temperature":0.7,"pith_summary":"The paper sets out to establish that screenshots and other images attached to repository issues are not decoration: they carry localization signal that current AI systems use unevenly. By building a controlled benchmark where each issue can be evaluated with and without images, and by annotating every image's category and relevance, the authors make it possible to ask whether, when, and how visual evidence helps find the right file or function. The headline finding is that no existing system is reliable: even the best agent finds all correct files in the top 5 only 38.96% of the time, and the best retriever finds all correct functions in the top 10 only 33.86% of the time. The paper also shows that strong performance on text-only issue-resolving benchmarks does not transfer to multimodal issues, so a dedicated benchmark is needed.","feed_headline":"Screenshots in bug reports stump AI: top systems land the fix file only 39% of the time","feed_subtitle":"A controlled benchmark shows images carry real localization signal, but agents use them unevenly and text-only skill doesn't transfer to vis","key_machinery":"The central mechanism is the benchmark's paired evaluation protocol: each instance can be run text-only, with raw images, with structured visual text (VCE), or with both, so visual evidence is an independent variable rather than a confound. Per-image annotations (7 evidence categories, 4 relevance levels, including a human-reviewed harmful-image subset) plus strict all-gold Acc@K scoring turn image use into a testable cause. The paper also contributes a controlled multimodal retriever, MM-IssueLoc-VL-Embedding, trained with a contrastive objective and a file-to-function curriculum, whose no-image ablation isolates the image contribution from the retriever's text understanding.","core_discovery":"The paper claims that visual evidence in repository issues is a measurable, and currently under-exploited, variable for code localization. It constructs a controlled benchmark of 652 real issue-PR pairs with per-image relevance and category annotations and both file- and function-level gold labels, then shows three things: no current system localizes reliably under strict all-gold Acc@K (best file Acc@5 = 38.96, best function Acc@10 = 33.86); removing images costs a controlled retriever 4.44-4.91 points of file Acc@5, so images themselves carry signal; and strong localization on text-only issue-resolving benchmarks does not transfer cleanly to multimodal issue localization. The paper's contr","pith_inferences":["The paired with/without-image protocol is a template: future 'does vision help?' questions in software engineering could reuse this design to avoid the confound where a model's patch ability masks localization failures.","Because VCE converts images to text, the results suggest that lightweight OCR-and-structured-extraction pipelines might let existing text-only localizers gain some multimodal ability without retraining on image data.","The benchmark's harmful-image subset, though synthetic, implies that future localizers should be evaluated not just on whether they use images but on whether they can resist misleading ones; scaling up naturally occurring harmful images is a testable extension.","If the cross-benchmark non-transfer holds, then text-only scores on issue-resolving benchmarks are not a valid proxy for multimodal localization ability; benchmark designers should treat multimodal localization as a separate evaluation axis."],"forward_implications":["Visual evidence is a measurable localization variable: removing images from the paper's controlled retriever drops file Acc@5 by 4.44 to 4.91 points, showing the images themselves, not just the issue text, carry signal.","Current systems are far from reliable: the strongest agent misses all correct files in its top 5 for over 60% of instances (38.96 Acc@5), and the strongest retriever reaches only 33.86 Acc@10 at function level.","Multi-edit issues are the main bottleneck: Acc@10 collapses from roughly 83 on easy instances to roughly 3 on hard instances for the top agent, so strict all-gold accuracy on hard issues is near zero.","Strong text-only localization does not transfer to multimodal settings: the same agent family that scores above 90 File@5 on text-only issue-resolving benchmarks drops to around 43 on a multimodal issue-resolving benchmark, motivating dedicated multimodal localization tests.","Structured textualization of images (VCE) can be as helpful as raw pixels for some agents, and sometimes more: several agent-framework combinations improve file Acc@5 with VCE alone, while raw-pixel residuals vary widely across models."],"fun_headline_variants":["AI fixers land the right file only 39% on visual bug reports","Screenshots add real signal, but AI locators use them unevenly","Text-only mastery doesn't transfer to visual bug localization","New benchmark makes visual evidence a testable variable in bug fixing"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The benchmark assumes that the files and functions edited by the merged pull request are exactly the locations a good localizer should output; if real pull requests contain unrelated refactors, mass renames, or incomplete fixes, every Acc@K score measures patch alignment rather than localization skill.","fun_headline_variants_meta":{"raw":{"variants":["AI fixers land the right file only 39% on visual bug reports","Screenshots add real signal, but AI locators use them unevenly","Text-only mastery doesn't transfer to visual bug localization","New benchmark makes visual evidence a testable variable in bug fixing"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000272,"raw_usage":{"total_tokens":1514,"prompt_tokens":833,"completion_tokens":681,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":577,"completion_tokens_details":{"reasoning_tokens":606}},"tokens_in":577,"tokens_out":681,"duration_ms":6061,"temperature":1.0,"reasoning_tokens":606,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T23:50:13.604431+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train the paper's controlled retriever on the same training rows with all images stripped, and run it on the 343-instance function subset; if its function Acc@10 matches or exceeds the multimodal retriever's 33.86, then the benchmark's 'visual signal' is actually explainable by text cues, refuting the claim that images carry distinct localization information.","supporting_citations":[],"review_version":1}