{"id":"2b12176e-e5e4-4859-8f74-46c902450422","arxiv_id":"2412.17315","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CodeV converts images and videos in GitHub issues into detailed text and structured summaries, then feeds them to a text-only LLM to generate patches, improving resolution rates on the new Visual SWE-bench benchmark.","lead":"CodeV uses a vision-language model to turn screenshots and diagrams in GitHub issues into text descriptions, then feeds those to a standard coding LLM to produce fixes. On a new benchmark of 133 visual issues, it resolves about 13% of issues versus about 8% for a text-only baseline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ablation shows the gain is driven by structured summarization, not by visual descriptions; the claim that visual data are causal is unproven.","rationale":"The reader's weakest assumption concerns the fidelity of VLM descriptions, but the ablation study raises a more direct issue: the component that injects visual content into the pipeline (fine-grained description) contributes little, while the component that reformats the entire issue (structured summarization) accounts for the whole improvement over baseline. This makes the causal role of visual data uncertain, regardless of description fidelity. Even perfect descriptions would not help if the only effective component is a text-level summary. The reader's concern is related but focuses on information loss in conversion; the deeper issue is that the paper never isolates the contribution of visual content. The recommended verdict remains conditional acceptance (UNCHANGED), but the revision should require the text-only summary control. If that control shows no drop in performance, the central claim that visual data are essential would be unsupported and the verdict should move toward rejection.","tokens_in":13881,"tokens_out":5920,"duration_ms":61716,"concrete_test":"Run a text-only control: use Qwen2.5-Coder-32B (or DeepSeek-V2.5) to generate the structured summary from the issue text with all images removed, then feed that summary plus the original issue to Agentless. If the resolved count remains at or near 15/133, visual data are not causal; the central claim would need to be revised. Also run the same with fine-grained descriptions omitted; if the count stays around 15, the entire method reduces to text reformatting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim, that visual data improve issue resolution, is not supported by the ablation evidence. In Table 3, CodeV + Agentless resolves 15/133 instances. Removing the fine-grained description component—the only component that explicitly converts visual data into text—drops performance to 13/133. Removing structured summarization instead drops performance to 10/133, exactly equal to the Agentless baseline (Table 2). Therefore the entire measurable gain over baseline is attributable to the structured summary, which processes the whole issue and is not specific to visual content. The paper provides no control condition in which the structured summary is generated from the textual issue alone, with images removed. Without that control, the improvement could come solely from reformatting the text into a clearer structure. Moreover, the fine-grained visual descriptions show no standalone benefit, which is inconsistent with the claim that visual data 'convey additional knowledge that text alone cannot.' This is an internal inconsistency: the component that isolates visual information has no independent effect, while the component that reorganizes text accounts for all gain.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CodeV, a two-phase approach for automatically resolving GitHub issues that contain visual data such as screenshots and videos. In the data-processing phase, a vision-language model (Qwen2-VL) generates fine-grained descriptions of each image/video and a structured summary of the entire issue. In the patch-generation phase, these textual artifacts are spliced into the original issue and passed to an existing textual issue-resolving approach (Agentless) to produce a patch. The authors construct a new benchmark, Visual SWE-bench, containing 133 instances from 11 repositories, selected so that visual data is essential for resolution. Experiments on this benchmark show that CodeV + Agentless resolves 12.78% of instances versus 7.52% for Agentless alone (17 vs. 10 instances) in the best configuration, and ablations indicate that removing structured summarization drops performance back to the Agentless baseline.","tokens_in":14086,"tokens_out":3780,"duration_ms":37567,"significance":"If the central claim were established, the paper would make a useful contribution by drawing attention to a previously ignored modality in issue resolution and by providing a benchmark for future work on visual GitHub issues. The construction of Visual SWE-bench, with its execution verification and human filtering, is a substantial effort and could serve as a community resource. The paper also provides transparent limitation statements and uses only open-source models, which improves reproducibility. However, the significance is tempered by the fact that the ablation evidence does not isolate the contribution of visual data, and the benchmark is constructed in a way that may amplify the apparent benefit of visual processing.","major_comments":[{"comment":"","section":"Section 4.2.2, Table 3"},{"comment":"","section":"Section 3.1, construction step 4"},{"comment":"","section":"Table 2 and Table 3"}],"minor_comments":[{"comment":"","section":"Section 1, Figure 1 caption"},{"comment":"","section":"Table 1"},{"comment":"","section":"Section 2.1, structured summarization"},{"comment":"","section":"Section 4.2.1 and Appendix C"},{"comment":"","section":"Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The core issue is that the ablation in Table 3 does not isolate the visual contribution; the structured-summarization component, which processes the entire issue, accounts for all of the gain over the Agentless baseline in the Qwen2-VL-72B configuration. The authors need to add a text-only control for the structured summary to support the central claim. The self-constructed benchmark filtering further compounds this concern. If these points are addressed, the contribution could be valuable, but as presented the evidence is not sufficient for acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Visual SWE-bench is the real contribution; CodeV is plausible, but the paper overclaims what the ablation shows. Worth sending to review, but the authors should be pushed to add a control.\n\nWhat is genuinely new: a curated benchmark of 133 GitHub issues with images or videos, built with four filtering stages and execution verification, plus a simple two-phase VLM pipeline (fine-grained descriptions plus structured summary) that feeds enriched text into Agentless. The benchmark statistics are useful, the construction is transparent, and the authors honestly list limitations including best-of-two runs and the lack of an independent benchmark.\n\nThe soft spots are real. Table 3 shows the full system resolves 15/133; dropping fine-grained description gives 13/133; dropping structured summarization gives 10/133, exactly the Agentless baseline. The stress-test is right that the measurable gain over baseline disappears when you remove structured summarization. The authors interpret this as showing both components help, but the component that isolates visual content (fine-grained description) adds only a small, likely non-significant 2 instances. Since structured summarization is generated from the whole issue including the images, the gain could come from better text organization, better visual integration, or both. The missing control is a structured summary produced from the textual issue alone, with images removed. Without it, the headline claim that visual data convey additional knowledge text alone cannot is not established. This is fixable, not fatal.\n\nThe benchmark inclusion criteria deliberately exclude OCR-convertible images and issues where visual data was judged non-essential, so the set is reasonable for evaluating image-augmented methods, but it inflates the apparent gap to text-only baselines. The absolute counts (17 vs 10) are fragile under small perturbations, and no confidence intervals or repeated-run variance are reported.\n\nThis paper is for people working on repository-level automation and multimodal coding; the benchmark will likely be reused. It deserves a serious referee, and the referee should ask for the text-only-summary control and more careful statistical reporting. I would bring it to our reading group for the benchmark construction and the ablation lesson.","headline":"Visual SWE-bench is a useful new artifact, but the paper's central claim that visual data drive the improvement is missing a key control.","tokens_in":14593,"tokens_out":3129,"would_cite":true,"duration_ms":33094,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Converting the visual data in GitHub issues into text lets a text-only LLM resolve over 50% more issues, the paper reports.","keywords":["GitHub issue resolving","visual data","vision-language model","SWE-bench","benchmark construction","patch generation","software engineering"],"falsifier":"Have humans compare each generated description and summary against the original image or video for all 133 instances, scoring omitted or hallucinated visual facts, and then check whether the seven instances CodeV resolves beyond Agentless are exactly the ones with accurate descriptions; if those instances contain key description errors, the visual-to-text fidelity mechanism is undercut.","tokens_in":13736,"feed_emoji":"🖼️","tokens_out":10045,"duration_ms":85645,"temperature":0.7,"pith_summary":"The paper claims that existing GitHub issue-resolving systems leave useful evidence on the table: screenshots, diagrams, and videos attached to issues. CodeV is a two-phase pipeline that first uses a vision-language model to turn each piece of visual data into a fine-grained text description and to write a structured summary of the whole issue, then feeds that enriched text into an ordinary text-only issue resolver. On a new benchmark called Visual SWE-bench, built from 133 visual issues drawn from SWE-bench and visualization-library pull requests, CodeV plus Agentless resolves 12.78 percent of instances compared with 7.52 percent for Agentless alone, a relative improvement of more than 50 percent. The authors argue this shows visual data carries information that text alone cannot, and that it can be exploited cheaply with open-source models.","feed_headline":"Turning screenshots into text helps LLMs fix 50% more issues","feed_subtitle":"A two-phase CodeV pipeline converts images into descriptions, lifting Agentless from 7.5% to 12.8% resolved.","key_machinery":"The machinery is the visual-to-text translation step. Fine-grained description is generated in two passes: an independent description that records the raw content of each image or video, and a contextual description that explains what the visual data demonstrates relative to the reported bug. Structured summarization then reorganises the entire issue into a template with clear fields such as problemSummary, stepsToReproduce, expectedResults, and actualResults. These textual artifacts replace the original visual data in the prompt, allowing a text-only LLM operating through Agentless to reason about evidence it could never see directly.","core_discovery":"The central claim is that visual data in an issue can be converted into text and that this conversion, not the coding model's innate multimodal ability, is what unlocks better issue resolution. CodeV's vision-language model produces two artifacts: fine-grained descriptions (an independent description of each image or video, then a contextual description that interprets the image's role in the issue) and a structured summary with problem summary, reproduction steps, expected results, and actual results. These are spliced into the original issue text and passed to Agentless, an open-source localization-and-repair pipeline. The result is 17 of 133 Visual SWE-bench instances resolved (12.78 percent) versus 10 (7.52 percent) for Agentless alone; a variant that feeds images directly to a vision-language model for repair resolves only one instance, which the authors attribute to the VLM's weak coding ability. Ablations show that removing any component lowers performance, with structured summarization removal having the largest effect.","pith_inferences":["The authors discard the original visual data after converting it to text; a natural extension is to keep both modalities and let the LLM refer back to the image, which could recover details the VLM misreads or omits.","Because the benchmark is dominated by visualization and documentation libraries (matplotlib, sphinx, seaborn), the observed gain may be largest for output- and plot-oriented bugs; other software domains with different visual genres may see a smaller effect.","The headline gain rests on 7 additional resolved instances (17 versus 10), so the over-50-percent relative improvement is a small absolute effect; replicating on a larger benchmark would test whether the advantage is stable.","A direct test of the fidelity assumption would be to have humans mark the factual accuracy of the VLM's descriptions on all 133 instances and check whether the instances CodeV uniquely solves are exactly those with accurate descriptions."],"forward_implications":["Visual GitHub issues can be attacked with open-source text-only coding LLMs, avoiding the estimated thousands of dollars needed to run closed multimodal agents over a benchmark.","Existing text-only issue resolvers such as SWE-agent, AutoCodeRover, and RAG can be given visual awareness simply by preprocessing issues with CodeV's pipeline, without changing the resolver.","Structured summarization is the highest-value component: removing it cuts resolved instances from 15 to 10, more than removing either description step.","VLM size is not decisive: Qwen2-VL-2B and 7B pipelines resolved 17 instances, matching or beating the 72B version's 15, so cheap small VLMs can carry the translation step.","Visual SWE-bench provides a reusable, test-verified set of 133 visual issue instances, filling a gap left by existing benchmarks whose evaluation scripts are not public."],"supporting_citations":[{"why":"Provides SWE-bench, whose 128 visual instances seed Visual SWE-bench and whose RAG baseline is compared.","marker":"Jimenez et al., 2024"},{"why":"Provides Agentless, the open-source localization-and-repair pipeline that CodeV augments and the main baseline.","marker":"Xia et al., 2024"},{"why":"Provides Qwen2-VL, the vision-language model that converts visual data into descriptions and summaries.","marker":"Wang et al., 2024"},{"why":"Provides Qwen2.5-Coder-32B, one of the two LLMs used to generate patches.","marker":"Hui et al., 2024"},{"why":"Provides DeepSeek-V2.5, the other LLM used to generate patches.","marker":"DeepSeek-AI, 2024"},{"why":"Provides SWE-agent, a commercial-model baseline and the basis of the cost estimate for closed multimodal agents.","marker":"Yang et al., 2024a"},{"why":"Provides SWE-bench Multimodal, the prior visual-issue benchmark that the authors say lacks public evaluation fields and scripts.","marker":"Yang et al., 2024b"},{"why":"Provides AutoCodeRover, a baseline issue-resolving approach in the comparison.","marker":"Zhang et al., 2024"}],"fun_headline_variants":["Turning screenshots into text boosts LLM issue-fixing by 70%","Visual data converted to text helps LLMs resolve more GitHub issues","CodeV: Descriptive text from screenshots improves SWE-bench results","Images in issues become text, making LLM fix rates jump","Explain screenshots as text: LLMs then fix 70% more issues"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire improvement depends on the assumption that the vision-language model's text descriptions and structured summary preserve every task-relevant detail in the screenshot, diagram, or video, because the original visual data is discarded and never checked for accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Turning screenshots into text boosts LLM issue-fixing by 70%","Visual data converted to text helps LLMs resolve more GitHub issues","CodeV: Descriptive text from screenshots improves SWE-bench results","Images in issues become text, making LLM fix rates jump","Explain screenshots as text: LLMs then fix 70% more issues"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00056,"raw_usage":{"total_tokens":2631,"prompt_tokens":887,"completion_tokens":1744,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1650}},"tokens_in":503,"tokens_out":1744,"duration_ms":12336,"temperature":1.0,"reasoning_tokens":1650,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T05:35:09.316008+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have humans compare each generated description and summary against the original image or video for all 133 instances, scoring omitted or hallucinated visual facts, and then check whether the seven instances CodeV resolves beyond Agentless are exactly the ones with accurate descriptions; if those instances contain key description errors, the visual-to-text fidelity mechanism is undercut.","supporting_citations":[],"review_version":1}