{"id":"d1d3e54c-8554-4811-a6f9-f56ea147762b","arxiv_id":"2602.06090","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A multimodal program-repair system that converts bug screenshots into semantic scene graphs and iteratively crops to bug regions, reporting 36.47% on SWE-Bench M, 38.02% on MMCode, and 95.12% on CodeVision.","lead":"SVRepair turns bug screenshots and diagrams into structured text scene graphs, then uses a coding agent to find and fix bugs. It reports top scores on three code benchmarks, but the claims rest on very thin margins and some numbers in the paper disagree.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SWE-Bench M result is arithmetically inconsistent: 186/517 vs benchmark size 617; if the true denominator is 617, Pass@1 is 30.15%, not 36.47%, invalidating the SOTA claim.","rationale":"The central claim of the paper is state-of-the-art Pass@1 on SWE-Bench M (36.47%). This number is arithmetically inconsistent with the paper's own description of the benchmark (617 instances) and the reported resolved count (186/517). If the official benchmark size is 617, then the actual Pass@1 is 30.15%, which is below several baselines in Table 1, directly contradicting the abstract and Section 4.3. If the paper used a filtered 517-instance subset, then all baseline comparisons in Table 1 must have been run on that same subset; the paper does not state this, making the comparison to GUIRepair and OpenHands invalid. This issue is more immediately load-bearing than the transfer-learning assumption: even if SVR transfers perfectly, an incorrect evaluation denominator invalidates the headline result. The reader's rationale does mention 'mixes denominators' but places the primary weakness on domain transfer; I regard the denominator discrepancy as the more fundamental problem. Therefore, the paper cannot be accepted as-is; it should be rejected until the evaluation protocol is corrected and the results recomputed. The additional inconsistencies (CodeVision 95.73 vs 95.12; OmniGIRL only in abstract) reinforce the need for a single consistent reporting protocol.","tokens_in":12689,"tokens_out":5520,"duration_ms":54602,"concrete_test":"Inspect the released code (github.com/codefuse-ai/CodeFuse-SVR) and the predictions to count the exact number of SWE-Bench M instances evaluated; cross-check against the official SWE-Bench Multimodal split (which Section 4.2 says has 617). If the evaluated set has 517 instances, list the exclusion criteria and rerun GUIRepair on that same subset; if instead the official split is 617, recompute SVRepair's Pass@1 as 186/617 = 30.15% and compare with Table 1 baselines, which would overturn the headline claim. Also verify the CodeVision number (95.73 vs 95.12) from the released logs.","verdict_should_be":"REJECT","load_bearing_attack":"The most load-bearing concern is the internal inconsistency in the headline SWE-Bench M result. The abstract reports '186/517 SWE-Bench M instances (35.98% over all instances; 36.47% over submitted runs)', implying a 517-instance evaluation (510 submitted runs, since 186/510 = 36.47%). However, Section 4.2 states 'SWE-Bench M consists of 617 task instances across 17 JavaScript repositories.' If the benchmark truly has 617 instances, then 186 resolved gives 30.15% Pass@1, which is below GUIRepair (35.98%) and OpenHands-Versa (34.43%) in Table 1, destroying the claimed SOTA status. If instead the benchmark actually has 517 instances, then Section 4.2 contains a factual error and, more importantly, the paper must show that all baselines (GUIRepair, OpenHands, etc.) were evaluated on the same 517-instance subset. The abstract and body also disagree on CodeVision (95.73% vs 95.12%), and OmniGIRL appears only in the abstract; these numbers must be reconciled. Without a consistent denominator and a single evaluation subset shared by all methods, the central claim that SVRepair outperforms prior multimodal APR systems is not verifiable.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SVRepair, a multimodal automated program repair framework. It fine-tunes Qwen3-VL-8B on WebSight HTML/DOM pairs and staticfg control-flow graphs to produce a textual Mermaid-based Semantic Scene Graph (SSG) from bug-related images. A coding agent (GPT-o3) uses the SSG plus the repository to localize bugs and generate patches, with an iterative visual-artifact segmentation loop (Qwen3-VL-235B) that crops to bug-centered regions when patch validation fails. Experiments are reported on SWE-Bench M (186/517 resolved, 36.47% Pass@1 over submitted runs / 35.98% over all instances), MMCode (38.02%), and CodeVision (95.12%/95.73% in abstract), with ablations V1-V4 and a Mermaid rendering fidelity study (Table 4). The central claim is state-of-the-art performance on multimodal APR through structured visual representation.","tokens_in":13026,"tokens_out":3845,"duration_ms":33204,"significance":"If the reported numbers hold, the paper makes a useful contribution: it provides evidence that a compact fine-tuned VLM can convert screenshots into a code-relevant intermediate graph that improves fault localization and patch generation, and it quantifies the benefit of iterative visual cropping. The method is largely reproducible in design (open model, public code link, detailed prompts in appendices), and the SVR training setup is simple and transparent. The strength of the paper is the idea of normalizing visual artifacts into a structured textual graph rather than feeding raw images to the coding agent. However, the load-bearing evidence is currently undermined by arithmetic inconsistencies in the headline SWE-Bench M result and ambiguous evaluation-subset definitions, so the significance assessment is conditional.","major_comments":[{"comment":"The headline Pass@1 numbers are arithmetically inconsistent. The abstract reports 186/517 SWE-Bench M instances (35.98% over all instances; 36.47% over submitted runs), which implies 186/510 ≈ 36.47% submitted runs. But §4.2 states that SWE-Bench M consists of 617 task instances. If the benchmark has 617 instances, then 186/617 = 30.15%, which is below GUIRepair (35.98%) and OpenHands-Versa (34.43%) in Table 1, invalidating the claimed SOTA. If the evaluation was restricted to a 517-instance subset, the paper must state this explicitly, justify the subset, and show that all baselines in Table 1 were evaluated on the same subset. Without a consistent denominator and a shared evaluation set, the central claim is not verifiable.","section":"Abstract / §4.2–4.3, Table 1"},{"comment":"The CodeVision result is reported as 95.73% in the abstract and as 95.12% in Table 2 and §4.3. Similarly, OmniGIRL appears only in the abstract (4/19, 21.05%) and is never mentioned again in the experimental sections, including the benchmark description, Table 1, or the effectiveness section. These discrepancies must be reconciled. The stated CodeVision value affects the 'outperforming all baselines' claim, and the OmniGIRL result is part of the abstract's evidence but no method or evaluation details are provided for it.","section":"Abstract vs. §4.3, Table 2"},{"comment":"The training description is internally inconsistent and incomplete. The paper says SVR is fine-tuned on WebSight (HTML/DOM) and 37 GitHub repositories' staticfg control-flow graphs, but no dataset sizes, class balance, or preprocessing details are given. More importantly, §4.1 states 'The training is performed on 8 NVIDIA H20 96GB GPUs, and the process includes three epochs' and then two sentences later says 'The model is trained for 2 epochs.' This is a concrete contradiction that matters for reproducibility. The transfer claim — that a model trained only on clean HTML/DOM and CFG images produces useful SSGs for noisy, in-the-wild screenshots in SWE-Bench M — is central to the method, but no in-domain or noisy-image evaluation is provided beyond the 1,300 CFG parse benchmark in §4.4. The paper should report the actual epoch count and provide evidence on real bug-report screenshots (e.g.","section":"§3.1 / §4.1"},{"comment":"The ablation table is missing numbers: row (3) has '35.0138.02' for SWE-Bench M (presumably '35.01' and '38.02' concatenated), and row (4) has only SWE-Bench M. The narrative explains that feedback was not applied to MMCode/CodeVision, but the table does not show the V4 rows for those benchmarks, making it impossible to verify the claim that V4 equalled V3 on those benchmarks. Also, V2 (+Vision) on SWE-Bench M is only 33.08% vs V1's 32.88%, which is a small gain; this is fine but the text's phrase 'nuanced' is vague. Please report complete numbers.","section":"§4.4, Table 3"},{"comment":"The baseline numbers appear to be taken from other papers, but the evaluation conditions are not fully specified: are all baselines run on the same Docker environment, same test harness, same subset (if any), same number of attempts, and same compute budget? The only methodological comparison given is that GUIRepair uses multi-sampling up to 40 candidates, while SVRepair uses greedy decoding. However, Table 1 reports a single Pass@1 number per method without clarifying whether Pass@1 is the best-of-N or first-attempt rate for baselines. Since the central claim is SOTA, the comparison must be apples-to-apples or at least clearly qualified.","section":"§4.2 / Table 1"}],"minor_comments":[{"comment":"The abstract says '36.47% accuracy on SWE-Bench M', '95.12% on CodeVision', and '186/517', while the body says '95.12%' and '36.47%'. Please ensure all numbers are consistent across abstract, introduction, and results.","section":"Abstract / §1 / §4.3"},{"comment":"The term 'directed cyclic graph' is odd for a DOM tree or control-flow graph. A CFG may have cycles, but a DOM tree is acyclic. Please clarify whether the SSG is a general directed graph.","section":"§3.1"},{"comment":"The Mermaid parsing benchmark is described as '1,300 code-control flow graph pairs' and later 'code-control flow graph pairs from high-starred GitHub repositories'. It is unclear whether these are held-out from the SVR training set. Please specify the split to avoid optimism bias.","section":"§4.4 / Table 4"},{"comment":"The case study legend is difficult to read: 'SVRepairw/o feedback loop' and 'SVRepair' are concatenated. Please clean up the figure captions and labels.","section":"§4.5, Figure 3"},{"comment":"The limitation section says 'threshold of three rounds is generally sufficient' in §3.3 and 'e.g., k=2' in §6. Please reconcile the default iteration threshold.","section":"§6"},{"comment":"Some references are duplicated or incomplete (e.g., Laurençon et al. appears twice; the Refact URL is split across lines). Please check formatting.","section":"Related Work / References"},{"comment":"The implementation details are too terse: no learning rate schedule, warmup, batch size, LoRA rank if any, or image resolution. For a paper that introduces a fine-tuned model, these details are needed for reproducibility.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is interesting and the ablation supports the direction of the method, but the headline SWE-Bench M numbers are currently not self-consistent (186/517 vs 617 benchmark size) and the CodeVision/OmniGIRL discrepancies add to the credibility risk. These are fixable in a revision if the authors clarify the evaluation subset and reconcile all numbers. I do not think rejection is warranted, but the revision must be substantive, not cosmetic."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the Semantic Scene Graph (SSG) intermediate representation plus the iterative visual-artifact segmentation loop is a real idea, and the authors deserve credit for training a small VLM to turn screenshots and CFGs into Mermaid-style scene graphs. The ablation, despite a few mismatches, points in the right direction. But the paper currently cannot be accepted because the headline SWE-Bench M numbers do not hold together. The abstract reports 186/517 resolved; Section 4.2 says the benchmark has 617 instances. If the denominator is 617, the pass rate is 30.15%, which is below GUIRepair (35.98) and OpenHands (34.43) in their own Table 1. If they used a subset of 517, they need to say so and show all baselines were evaluated on that same subset. That is not a minor typo; it is the central empirical claim. The same goes for CodeVision (95.73 in the abstract vs 95.12 in the body) and OmniGIRL, which appears only in the abstract. I also caught a few smaller inconsistencies: the ablation text says the vision boost on CodeVision took 60.36 to 89.02, but Table 3 lists 51.83 to 85.36; Section 4.1 says 'three epochs' then 'two epochs'; and CodeV is cited in related work but missing from the baseline table. None of these are fatal to the method, but they suggest the evaluation section was assembled in a hurry. On the substance: the SSG idea is a genuine extension of GUIRepair, not a repackaging. The iterative segmentation loop is a sensible way to reduce visual noise, and the ablation (at face value) shows each component helps. The limitation section is honest about scope. The main open scientific question is whether SVR, trained only on WebSight HTML/DOM pairs and staticfg CFGs, transfers to noisy real-world screenshots in SWE-Bench M. The paper does not show any domain adaptation or per-instance analysis to support that transfer. So: the idea deserves a serious referee, but the paper needs a major revision focused on reporting consistency, a clear evaluation subset, and stronger evidence for the transfer. If they fix the numbers and include CodeV, I'd believe the direction of the result.","headline":"A novel structured-visual-reasoning idea for APR, undermined by inconsistent headline numbers—the SOTA claim needs a denominator fix before it can be taken seriously.","tokens_in":13523,"tokens_out":3598,"would_cite":false,"duration_ms":36920,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SVRepair claims that translating bug screenshots into structured semantic scene graphs, then iteratively cropping to bug-centered regions, lets a coding agent fix visual bugs at higher rates than unimodal or raw-image methods.","keywords":["automated program repair","multimodal LLM","semantic scene graph","visual artifact segmentation","fault localization","SWE-Bench M","Mermaid","vision-language model"],"falsifier":"Measure whether SVRepair's Pass@1 on SWE-Bench M drops to the unimodal baseline when the SVR module is replaced by a generic image captioner (or by SVR fine-tuned on a small held-out set of real SWE-Bench M images). A drop to baseline would show the scene graph, not the agent's search, is doing the work; no drop would undermine the paper's causal story.","tokens_in":12569,"feed_emoji":"🩹","tokens_out":3882,"duration_ms":43449,"temperature":0.7,"pith_summary":"The paper tries to establish that structured visual reasoning improves automated program repair. Its central move is to fine-tune a vision-language model to turn screenshots and control-flow graphs into a textual 'semantic scene graph' that lists GUI elements and their hierarchical relations, then feed that graph to a coding agent that localizes and patches the fault. A second mechanism recursively segments the original image, focusing each repair round on the region most likely to contain the bug. On SWE-Bench M the full system reports a Pass@1 of 36.47%, and on the two supplementary code-from-vision benchmarks it reaches 38.02% and 95.12%.","feed_headline":"Semantic scene graphs lift visual bug fixing to 36.47%","feed_subtitle":"SVRepair turns screenshots into structured graphs before patching; beats prior visual-repair systems on SWE-Bench M.","key_machinery":"Semantic Scene Graph (SSG): a directed graph whose nodes are visual elements (HTML nodes, CFG basic blocks) and whose edges carry relation types (composition hierarchy, control flow, data flow), serialized into Mermaid syntax for the coding LLM. The SVR model is a fine-tuned Qwen3-VL-8B trained on WebSight HTML/DOM pairs and staticfg control-flow graphs. The third component is the iterative visual-artifact segmenter, a larger VLM that predicts a bounding box for the buggy region and crops the image for the next round.","core_discovery":"The authors contend that the bottleneck in multimodal program repair is not visual perception but representation: raw screenshots fed to a multimodal LLM cause context loss and noise, so the LLM cannot ground the image in code. SVRepair's contribution is to replace the raw image with a normalized intermediate structure — a Semantic Scene Graph serialized as Mermaid text — and to iterate on it: a segmenter crops the image to the suspected bug region, the SVR model re-graphs that crop, and the coding agent retries localization and patching. The reported gains over prior visual-repair systems and over the authors' own unimodal baseline are attributed to this representation and to the iterative","pith_inferences":["The SSG format is agnostic to artifact type; sequence diagrams, UML class diagrams, or architecture diagrams could be supported by fine-tuning SVR on those pairs, making the framework a general visual-to-code bridge.","Because the segmentation step relies on an external large VLM, the marginal cost of SVRepair scales with the number of failed patch attempts; a cheaper, learned segmenter could be a natural follow-up.","A direct test of the transfer assumption would be to fine-tune SVR on a handful of real SWE-Bench M screenshots and measure whether Pass@1 changes; the paper's current numbers leave the domain gap unmeasured.","The unimodal-to-vision gap on CodeVision (51.83% to 95.12%) suggests that the biggest wins are in tasks where the image is essentially a specification; in noisy settings the graph's filtering role, not just its structure, may be what matters."],"forward_implications":["Visual bug reports can be repaired automatically without losing the diagnostic signal present in screenshots.","An 8B-parameter vision-language model fine-tuned on structured pairs can match or beat much larger general VLMs on diagram-to-structure parsing.","Iterative cropping of the visual artifact is what unlocks gains on noisy real-world repo issues; it does not help on noise-free code-generation benchmarks.","The same pipeline transfers, with no change, to code generation from flowcharts and competition problems (MMCode, CodeVision)."],"fun_headline_variants":["Replace screenshots with scene graphs to fix code bugs","Scene graphs from screenshots lift repair rate to 36%","Graph the UI, then patch: visual repair at 36.47%","From pixels to patches: scene graphs for code repair","Turn screenshots into structured graphs to boost patching"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The SVR model is trained only on clean, synthetic HTML/DOM pairs and control-flow graphs extracted with staticfg, yet it is deployed on noisy, real-world SWE-Bench M screenshots with no in-domain fine-tuning; the entire pipeline's benefit on that benchmark depends on this transfer producing code-relevant scene graphs.","fun_headline_variants_meta":{"raw":{"variants":["Replace screenshots with scene graphs to fix code bugs","Scene graphs from screenshots lift repair rate to 36%","Graph the UI, then patch: visual repair at 36.47%","From pixels to patches: scene graphs for code repair","Turn screenshots into structured graphs to boost patching"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000689,"raw_usage":{"total_tokens":3015,"prompt_tokens":858,"completion_tokens":2157,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":2074}},"tokens_in":602,"tokens_out":2157,"duration_ms":18698,"temperature":1.0,"reasoning_tokens":2074,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T06:07:18.374680+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Measure whether SVRepair's Pass@1 on SWE-Bench M drops to the unimodal baseline when the SVR module is replaced by a generic image captioner (or by SVR fine-tuned on a small held-out set of real SWE-Bench M images). A drop to baseline would show the scene graph, not the agent's search, is doing the work; no drop would undermine the paper's causal story.","supporting_citations":[],"review_version":1}