Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

VisualRepair claims that routing each bug-report image through a type-specific tool chain, then zooming around multiple grounded region candidates, resolves 196 of 517 SWE-bench Multimodal test issues—10 more than the best prior system—at $

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 02:49 UTC pith:4KW3Q7UO

load-bearing objection A coherent visual-APR system with plausible gains, but the headline margins are partly tuned on the test set and the dev margin shrinks from 11 to 5 under a matched backbone. the 4 major comments →

arxiv 2607.14075 v1 pith:4KW3Q7UO submitted 2026-07-15 cs.SE

VisualRepair: Dynamic Tool Calling and Region Focusing for Visual Software Issue Repair

Jingyu Xiao , Zhongyi Zhang , Haoran Hou , Yuxuan Wan , Yuan Jiang , Yintong Huo , Michael R. Lyu This is my paper
classification cs.SE
keywords automated program repairmultimodal large language modelsvisual software issue repairimage type classificationbug region groundingGUI bugsSWE-bench Multimodaltool calling
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper tries to establish that visual software issue repair is bottlenecked by two problems that prior systems treat uniformly: the heterogeneity of image types in bug reports (UI screenshots, IDE snapshots, GIFs, plain-text images) and imprecise grounding of the bug-relevant region inside a complex screenshot. It proposes VisualRepair, which routes each image through a deterministic type-aware tool chain and then grounds multiple candidate bug regions, zooming in and out on each to produce a diverse patch pool. On SWE-bench Multimodal, VisualRepair resolves 196 of 517 test issues (37.91%) and 25 of 102 dev issues, surpassing the strongest baselines by 10 and 11 instances at an average cost of $0.47 per issue. A sympathetic reader would care because the reported gains come from perception-side preprocessing and focused region sampling rather than from a more powerful backbone model alone.

Core claim

The central discovery is that a deterministic, image-type-aware tool chain combined with multi-candidate region zooming yields a reproducible improvement over uniform-pipeline baselines. Removing the type-aware tool chain (ITTC) drops resolved issues from 196 to 171; removing the region focusing (DTRF) drops them to 166; removing both leaves 151. The two components are complementary, with their combination adding 45 resolved issues over the no-module baseline. Gains concentrate on repositories with GIF-based interaction reports, noisy screenshots, or rendering-library code templates, indicating that the modules address distinct failure modes.

What carries the argument

The key machinery is two-fold. ITTC (Image Type-aware Tool Calling) is a rule-based classifier that checks the file suffix, uses UIED to detect UI elements, crops away blank regions, and then branches: GIFs are decomposed into MAE-selected key frames, text-heavy images go through OCR and a code-tag check, and render-library repos pull matching code templates from a pre-built library. DTRF (Dynamic Test-time Region Focusing) prompts a grounding MLLM to produce M=3 candidate bounding boxes for the bug region, then creates a zoomed-in (half-area) and zoomed-out (double-area) variant of each, yielding 10 visual inputs per issue, each generating one candidate patch. Patch selection compiles candi

Load-bearing premise

The headline numbers rely on hyperparameters (M=3, zoom factor 1/2, keyframe coefficient k=1.2) that were chosen by maximizing resolved counts on a sample drawn from the same SWE-bench Multimodal test set used for the final comparison; if those parameters partly overfit these test instances, the 10-instance edge over the best baseline could shrink.

What would settle it

Re-run the full VisualRepair pipeline on a fresh random split of SWE-bench Multimodal (or on a new set of visual issue reports from the same 17 repositories) with all hyperparameters fixed a priori from the dev set only, and check whether the 196/25 resolved counts and the 10/11 margins over the best baselines persist.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Type-aware visual routing and multi-region zoom focusing each add 15–20 resolved issues over a uniform visual pipeline on SWE-bench Multimodal, and together they add 45.
  • The code-library templates enable automated visual validation: candidate patches are rendered and compared against the original bug screenshot, filtering out no-effect patches before selection.
  • Deterministic tool routing avoids the instability and overhead of letting the LLM itself choose tools for visual inputs, claims the paper.
  • The gains concentrate on GIF-heavy, noisy-screenshot, and render-library repositories, suggesting the modules target specific failure modes that uniform pipelines systematically mishandle.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The hyperparameters (M=3 regions, zoom factor 1/2, keyframe coefficient k=1.2) were selected by maximizing resolved counts on a 100-instance sample drawn from the same test split used for the final evaluation; a genuinely held-out benchmark could shrink the 10-instance margin over the best baselines.
  • The same recipe of type routing plus multi-scale region focusing could plausibly transfer to other multimodal bug domains, such as mobile-app UI repair or design-to-code tools, where reports scatter heterogeneous visual artifacts.
  • The reported 100% classification accuracy is verified on the benchmark itself, not on out-of-distribution images; the paper's own failure analysis lists long screenshots, GIFs with excessive frames, and high-density images as remaining weak spots.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. VisualRepair is an MLLM-based framework for repairing visual software issues on SWE-bench Multimodal. It combines an Image Type-aware Tool Calling (ITTC) module, which classifies images (UI, IDE, GIF, text) and routes them through deterministic tool chains (GIF keyframe extraction, OCR, cropping, and repository-specific code templates), with a Dynamic Test-time Region Focusing (DTRF) module, which grounds multiple bug-relevant regions, applies zoom-in/zoom-out augmentation, and generates diverse candidate patches. The paper reports that VisualRepair resolves 196/517 test instances and 25/102 dev instances, outperforming the best baselines by 10 and 11 instances, with ablations, parameter sensitivity studies, and case studies supporting the contributions. The central claim is empirical: ITTC and DTRF provide a reproducible improvement over GUIRepair and SVRepair when using the same o3 backbone.

Significance. If the reported results hold, this is a practically useful advance in visual automated program repair. The system combines several plausible mechanisms—deterministic type-aware routing, adaptive region focusing, and issue-agnostic code templates—and reports a low average cost ($0.47/issue) with a solid margin over existing open-source baselines. The paper also contains a useful analysis of heterogeneous visual input types and grounding failures. I see no definitional circularity in the method: the components are not fitted to reproduce the final metric. However, the evaluation protocol has a load-bearing test-set tuning problem, and several presentation inconsistencies in the parameter and dev-set comparisons prevent me from accepting the claims as currently stated. The strengths—especially the deterministic classifier, the explicit attempt to avoid test leakage in code-template construction, and the ablation design—deserve credit; they are undermined only by the lack of released artifacts and by the tuning protocol.

major comments (4)
  1. [§V-C, Tables IV–VI] The hyperparameters M=3, zoom factor 1/2, and k=1.2 are selected by maximizing resolved counts on “100 test instances” (§V-C.1) and “50 GIF-based issues” (§V-C.3) sampled from the same SWE-bench Multimodal test set used to produce the headline 196-instance result in Table I. This is test-set fitting: the 10-instance margin over GUIRepair-o3 may be inflated. Please re-select parameters on the dev split or use a nested/held-out protocol, and report the resulting test numbers. A version that reports both “tuned on dev” and “tuned on test” would make the sensitivity of the claim transparent.
  2. [§V-A, Tables VII–VIII] The abstract’s “surpassing the best baseline by 10 and 11 instances” mixes backbones. The 11-instance dev margin comes from Table VII, where all methods use GPT-4o-20240806. Table VIII, using the same o3-20250416 backbone as the main test result, shows VisualRepair at 37 vs. GUIRepair at 32, a margin of only 5. The dev claim should be reported against the strongest same-backbone baseline, or the abstract should be qualified. This does not invalidate the test result, but it changes the strength of the advertised margin.
  3. [§III-B.2] The paper claims “100% accuracy with zero misclassified instances” for the end-to-end image type classifier on the full SWE-bench Multimodal benchmark, but provides no confusion matrix, per-type accuracy, prompt details, or released classification outputs. Since the final text-vs-code discrimination uses an MLLM judgment, this is an empirical claim, not a deterministic certainty. Please provide the evaluation data or release the classifier pipeline and outputs; otherwise the ITTC contribution is not independently verifiable.
  4. [§IV-A vs. §V-C.1] The definition of P is inconsistent. §IV-A states “P=1 patches generated per visual input” and a final pool of 1×(3M+1)=10 patches per issue, but Table IV reports P=4,7,10,13 for M=1,2,3,4, and §V-C.1 says “we reduced P to 10” and discusses “4 patches per region.” As written, the candidate-pool size, the sensitivity analysis, and the cost calculation are not reproducible. Please define P unambiguously (per visual input vs. per issue), align Table IV with the implementation, and state the total patch budget for the final configuration.
minor comments (5)
  1. [Table II] The column totals do not match the row sums. For example, OpenHands-Versa rows sum to 181, but the table reports 178; and the repo-level rows sum to 510, not the stated total of 517. Please correct the arithmetic or clarify that some repositories are omitted from the row-level display.
  2. [§V-A] The statement “We run VisualRepair three times and report the average results, with a variance of 1” is vague. Please specify the variance of what (resolved count, resolve rate, or cost) and report the three individual runs or a confidence interval.
  3. [§I, §V-A] The abstract and text say results are “as of March 2026.” Since leaderboard baselines may change, please provide the leaderboard snapshot date and, ideally, the exact commit/version of the benchmark used.
  4. [§III-B.1] In Algorithm 1, the condition “if repo not in CODELIBRARY” generates a template using “official documentation”; the paper asserts this is issue-agnostic. Please state explicitly how the MLLM is prevented from using any issue-derived content when generating templates, beyond the textual claim in §III-B.1.
  5. [§V-C.1] The sentence “M=3 with 4 patches per region would yield a comparable total” is confusing given the later statement “P=10 (approximately 1 patch per region per zoom level).” Please rewrite this paragraph to make the relationship between regions, zoom levels, and patch count precise.

Circularity Check

0 steps flagged

No definitional or self-citation circularity; the headline count is empirical, but the §V-C test-set hyperparameter selection is an evaluation-protocol concern rather than a circular reduction.

full rationale

The paper's claimed derivation chain is empirical rather than formal: ITTC is a rule-based image-type routing pipeline and DTRF is a multi-region grounding/zoom procedure, and neither module is defined in terms of SWE-bench M outcomes. No equation in §III reduces to a fitted target, and the reported 196 test resolutions are measured by benchmark execution, not computed from the hyperparameters. The main internal-evaluation concern is that §V-C selects M=3, zoom factor 1/2, and k=1.2 by maximizing resolved counts on 100 instances sampled from the same SWE-bench M test set used for the headline result; this can inflate the 10-instance margin over GUIRepair, but it is a test-set selection bias rather than a construction-level circularity because the final count is not a deterministic function of the chosen parameters. The abstract's dev margin of 11 comes from Table VII (GPT-4o backbone), while Table VIII shows a 5-instance margin under the same o3 backbone used for the main test result; this is a comparison-consistency issue, not circularity. Self-citations (e.g., DesignBench, Interaction2Code, EfficientUICoder, ComUICoder) appear only in related work and are not load-bearing for the central empirical claim. No uniqueness theorem or ansatz is imported from the authors' prior work. Therefore no significant circularity.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

The paper introduces no new theoretical entities. Its load-bearing free parameters are the keyframe threshold, zoom factor, and region/patch counts, all tuned on test-set samples. The main unverified assumptions concern the fidelity of the code-template leakage claim and the 100% classification accuracy claim.

free parameters (3)
  • keyframe threshold coefficient k (also written δ) = 1.2
    Selected by sensitivity analysis on 50 GIF-based test instances (Table VI) to maximize solved count while reducing keyframes; tuned on the evaluation set.
  • zoom factor for region augmentation = 1/2
    Selected on 100 test instances (Table V) as the best balance between coverage ratio and overlap ratio; tuned on the evaluation set.
  • number of grounded regions M and patch count P = M=3, P=10
    Selected on 100 test instances (Table IV); M=3 achieves the highest resolved count (37) and is used for the final test-set evaluation.
axioms (5)
  • domain assumption SWE-bench Multimodal resolved-instance counts are a valid proxy for visual software repair quality, and the benchmark's patch/test labels are correct.
    All headline claims are measured against the SWE-bench M test/dev splits (§II-B, §V-A).
  • domain assumption Compile validation plus MLLM visual-fix assessment (and pixel comparison for code-library repositories) is a sufficient patch oracle; no full test-suite execution is described.
    Patch selection in §III-E relies on compilation, pixel-level rendering comparison, and MLLM judgment rather than executing the repository's test suite.
  • ad hoc to paper Code-library templates constructed from repository structure and official documentation without issue text do not leak SWE-bench M test information.
    The authors assert this in §III-B.1: "Since construction relies solely on repository-level information with no access to issue content, this process eliminates any risk of test-set leakage," but no audit or artifact is provided.
  • ad hoc to paper The deterministic rule-based image type classifier achieves the claimed 100% accuracy with zero misclassified instances on the full benchmark.
    Stated in §III-B.2 without a released evaluation protocol or detailed per-stage accuracy numbers; the claim is load-bearing for ITTC's contribution.
  • domain assumption Leaderboard baseline results are directly comparable to VisualRepair's results despite differing submission dates and backbones.
    The main comparison in Table I mixes GPT, Claude, and closed-source commercial systems; the authors use the same o3 backbone for the closest baselines (GUIRepair, SVRepair), but other rows use different models.

pith-pipeline@v1.3.0-alltime-deepseek · 19179 in / 11047 out tokens · 106933 ms · 2026-08-02T02:49:27.299595+00:00 · methodology

0 comments
read the original abstract

Automated Program Repair (APR) has witnessed significant progress with the advent of Large Language Models (LLMs). However, as modern software systems increasingly expose rich graphical user interfaces, effectively leveraging visual information from bug screenshots has become essential for understanding bugs and generating accurate fixes in multimodal scenarios. Real-world issue reports frequently contain heterogeneous visual attachments including UI screenshots, IDE snapshots, GIFs, and text-centric images, each with distinct visual patterns and domain-specific semantics that impose substantial perceptual demands on MLLMs. Furthermore, bug screenshots often contain large expanses of uninformative and bug-irrelevant regions, distracting the model's attention and limiting patch diversity. To address these challenges, we propose VisualRepair, an MLLM-based framework for visual software issue repair comprising two core modules: Image Type-aware Tool Calling (ITTC), which classifies input images and dynamically invokes a tailored tool-calling chain for robust visual interpretation, and Dynamic Test-time Region Focusing (DTRF), which grounds multiple bug-related region candidates and refines them via an adaptive zoom-in and zoom-out strategy to improve fault localization and promote diverse patch generation. Extensive experiments on the SWE-bench Multimodal benchmark demonstrate that VisualRepair consistently outperforms state-of-the-art approaches. VisualRepair resolves 196 and 25 instances on the test and dev sets, respectively, surpassing the best baseline by 10 and 11 instances. These results highlight the effectiveness of type-aware visual understanding and region-focused localization for automated visual software issue repair.

Figures

Figures reproduced from arXiv: 2607.14075 by Haoran Hou, Jingyu Xiao, Michael R. Lyu, Yintong Huo, Yuan Jiang, Yuxuan Wan, Zhongyi Zhang.

Figure 1
Figure 1. Figure 1: Four visual tasks with heterogeneous inputs for SWE-bench M [22]. [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Image types distribution in SWEbench MM benchmark. Others means [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Bug-related region grounding example. Observation 2: MLLMs exhibit imprecise grounding ca￾pability, frequently failing to localize the bug-relevant region, which undermines repairing performance. III. METHOD A. Overview [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: The overview of VisualRepair. <html> <head> <link rel="stylesheet" href="./themes/prism.css"> </head> <body> <pre> <code class="language-sql"> INSERT INTO MESSAGESEARCHPATTERN (name, query)VALUES ('a', 'select ''hello world'' from a’) -- this is a comment INSERT INTO MESSAGESEARCHPATTERN (name, query)VALUES ('a','select ''hello world ''from a’) </code> </pre> <script src="./prism.js"></script> <script src=… view at source ↗
Figure 5
Figure 5. Figure 5: The reproduced code of two issues of PrismJS. [PITH_FULL_IMAGE:figures/full_fig_p004_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Venn diagram of fixed issues for Top-5 methods. [PITH_FULL_IMAGE:figures/full_fig_p007_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Case study for tool using. figure, two of the initial grounding results (red boxes) are incorrectly localized, demonstrating that single-shot grounding alone is insufficient to reliably capture the bug-relevant region at the bottom-right corner. DTRF further applies zoom-in (blue boxes) and zoom-out (green boxes) augmentations, among which the candidates marked with successfully cover the defective area an… view at source ↗
Figure 8
Figure 8. Figure 8: Case study for dynamic test-time region focusing. [PITH_FULL_IMAGE:figures/full_fig_p010_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. CodeShrink: Adaptive Visual Compression for Efficient Multimodal Code Understanding

    cs.CV 2026-07 conditional novelty 6.0

    A three-part pipeline (blank-free rendering, instruction-aware visual-token pruning, and per-input compression settings) cuts visual tokens for code-image MLLM inputs by up to 71% without losing accuracy.

Reference graph

Works this paper leans on

72 extracted references · 4 linked inside Pith · cited by 1 Pith paper

  1. [1]

    A survey of learning- based automated program repair,

    Q. Zhang, C. Fang, Y . Ma, W. Sun, and Z. Chen, “A survey of learning- based automated program repair,”ACM Transactions on Software Engi- neering and Methodology (TOSEM), vol. 33, no. 2, pp. 1–69, 2023

    2023

  2. [2]

    Automatic software repair: A survey,

    L. Gazzola, D. Micucci, and L. Mariani, “Automatic software repair: A survey,” inProceedings of the 40th International Conference on Software Engineering, 2018, pp. 1219–1219

    2018

  3. [3]

    Genprog: A generic method for automatic software repair,

    C. Le Goues, T. Nguyen, S. Forrest, and W. Weimer, “Genprog: A generic method for automatic software repair,”Ieee transactions on software engineering, vol. 38, no. 1, pp. 54–72, 2011

    2011

  4. [4]

    Tbar: Revisiting template-based automated program repair,

    K. Liu, A. Koyuncu, D. Kim, and T. F. Bissyand ´e, “Tbar: Revisiting template-based automated program repair,” inProceedings of the 28th ACM SIGSOFT international symposium on software testing and anal- ysis, 2019, pp. 31–42

    2019

  5. [5]

    Semfix: Program repair via semantic analysis,

    H. D. T. Nguyen, D. Qi, A. Roychoudhury, and S. Chandra, “Semfix: Program repair via semantic analysis,” in2013 35th International Conference on Software Engineering (ICSE). IEEE, 2013, pp. 772– 781

    2013

  6. [6]

    An empirical study on fine-tuning large language models of code for automated program repair,

    K. Huang, X. Meng, J. Zhang, Y . Liu, W. Wang, S. Li, and Y . Zhang, “An empirical study on fine-tuning large language models of code for automated program repair,” in38th International Conference on Automated Software Engineering (ASE), 2023, pp. 1162–1174

    2023

  7. [7]

    Impact of code language models on automated program repair,

    N. Jiang, K. Liu, T. Lutellier, and L. Tan, “Impact of code language models on automated program repair,” in45th International Conference on Software Engineering (ICSE), 2023, pp. 1430–1442

    2023

  8. [8]

    The plastic surgery hypothesis in the era of large language models,

    C. S. Xia, Y . Ding, and L. Zhang, “The plastic surgery hypothesis in the era of large language models,” in38th International Conference on Automated Software Engineering (ASE), 2023, pp. 522–534

    2023

  9. [9]

    Morepair: Teaching llms to repair code via multi-objective fine-tuning,

    B. Yang, H. Tian, J. Ren, H. Zhang, J. Klein, T. Bissyande, C. Le Goues, and S. Jin, “Morepair: Teaching llms to repair code via multi-objective fine-tuning,”ACM Transactions on Software Engineering and Method- ology (TOSEM), 2025

    2025

  10. [10]

    Less training, more repairing please: revisiting automated program repair via zero-shot learning,

    C. S. Xia and L. Zhang, “Less training, more repairing please: revisiting automated program repair via zero-shot learning,” in30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE), 2022, pp. 959–971

    2022

  11. [11]

    Gamma: Re- visiting template-based automated program repair via mask prediction,

    Q. Zhang, C. Fang, T. Zhang, B. Yu, W. Sun, and Z. Chen, “Gamma: Re- visiting template-based automated program repair via mask prediction,” in38th International Conference on Automated Software Engineering (ASE), 2023, pp. 535–547

    2023

  12. [12]

    Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,

    C. S. Xia and L. Zhang, “Automated program repair via conversation: Fixing 162 out of 337 bugs for $0.42 each using chatgpt,” in Proceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024). New York, NY , USA: Association for Computing Machinery, 2024, p. 819–831. [Online]. Available: https://doi.org/10.1145/3...

    arXiv 2024

  13. [13]

    Thinkrepair: Self- directed automated program repair,

    X. Yin, C. Ni, S. Wang, Z. Li, L. Zeng, and X. Yang, “Thinkrepair: Self- directed automated program repair,” in33rd International Symposium on Software Testing and Analysis (ISSTA), 2024, pp. 1274–1286

    2024

  14. [14]

    Repairagent: An autonomous, llm-based agent for program repair,

    I. Bouzenia, P. Devanbu, and M. Pradel, “Repairagent: An autonomous, llm-based agent for program repair,” in47th International Conference on Software Engineering (ICSE), 2025, pp. 694–694

    2025

  15. [15]

    Defects4j: A database of existing faults to enable controlled testing studies for java programs,

    R. Just, D. Jalali, and M. D. Ernst, “Defects4j: A database of existing faults to enable controlled testing studies for java programs,” inPro- ceedings of the 2014 international symposium on software testing and analysis (ISSTA 2014), 2014, pp. 437–440

    2014

  16. [16]

    SWE-bench: Can language models resolve real-world github issues?

    C. E. Jimenez, J. Yang, A. Wettig, S. Yao, K. Pei, O. Press, and K. R. Narasimhan, “SWE-bench: Can language models resolve real-world github issues?” inThe Twelfth International Conference on Learning Representations (ICLR 2024), 2024. [Online]. Available: https://openreview.net/forum?id=VTF8yNQM66

    2024

  17. [17]

    Demystifying llm-based software engineering agents,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Demystifying llm-based software engineering agents,”Proceedings of the ACM on Software Engineering (FSE), vol. 2, pp. 801–824, 2025

    2025

  18. [18]

    Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press, “Swe-agent: Agent-computer interfaces enable automated soft- ware engineering,”Advances in Neural Information Processing Systems (NeurIPS 2024), vol. 37, pp. 50 528–50 652, 2024

    2024

  19. [19]

    Specrover: Code intent extraction via llms,

    H. Ruan, Y . Zhang, and A. Roychoudhury, “Specrover: Code intent extraction via llms,” in47th International Conference on Software Engineering (ICSE), 2025, pp. 617–617

    2025

  20. [20]

    Designbench: A comprehensive benchmark for mllm-based front-end code generation,

    J. Xiao, M. Wang, M. H. Lam, Y . Wan, J. Liu, Y . Huo, and M. R. Lyu, “Designbench: A comprehensive benchmark for mllm-based front-end code generation,”arXiv preprint arXiv:2506.06251, 2025

    arXiv 2025

  21. [21]

    Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping,

    J. Xiao, Y . Wan, Y . Huo, Z. Wang, X. Xu, W. Wang, Z. Xu, Y . Wang, and M. R. Lyu, “Interaction2code: Benchmarking mllm-based interactive webpage code generation from interactive prototyping,” in2025 40th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE 2025). IEEE, 2025, pp. 241–253

    2025

  22. [22]

    SWE-bench multimodal: Do AI systems generalize to visual software domains?

    J. Yang, C. E. Jimenez, A. L. Zhang, K. Lieret, J. Yang, X. Wu, O. Press, N. Muennighoff, G. Synnaeve, K. R. Narasimhan, D. Yang, S. Wang, and O. Press, “SWE-bench multimodal: Do AI systems generalize to visual software domains?” inThe Thirteenth International Conference on Learning Representations (ICLR 2025), 2025. [Online]. Available: https://openrevie...

    2025

  23. [23]

    Beyond nl2code: A structured survey of multimodal code intelligence,

    X. Zhao, Q. Sun, J. Xiao, X. Liu, H. Yang, Q. Chen, X. Luo, J. Huang, Y . Zhong, L. Chenet al., “Beyond nl2code: A structured survey of multimodal code intelligence,”arXiv preprint arXiv:2606.15932, 2026

    Pith/arXiv arXiv 2026

  24. [24]

    Slidecoder: Layout-aware rag-enhanced hierarchical slide generation from design,

    W. Tang, J. Xiao, W. Jiang, X. Xiao, Y . Wang, X. Tang, Q. Li, Y . Ma, J. Liu, S. Tanget al., “Slidecoder: Layout-aware rag-enhanced hierarchical slide generation from design,” inProceedings of the 2025 Conference on Empirical Methods in Natural Language Processing (EMNLP 2025), 2025, pp. 9026–9050

    2025

  25. [25]

    Efficientpostergen: Semantic-aware efficient poster generation via token compression and accurate violation detection,

    W. Tang, J. Xiao, Y . Gong, F. Ran, T. Xia, J. Liu, M. H. Lam, W. Wang, and M. R. Lyu, “Efficientpostergen: Semantic-aware efficient poster generation via token compression and accurate violation detection,” arXiv preprint arXiv:2603.00155, 2026

    arXiv 2026

  26. [26]

    Webtestpilot: Agentic end-to-end web testing against natural language specification by inferring oracles with symbolized gui elements,

    X. Teoh, Y . Lin, D.-M. Nguyen, R. Ren, W. Zhang, and J. S. Dong, “Webtestpilot: Agentic end-to-end web testing against natural language specification by inferring oracles with symbolized gui elements,”Pro- ceedings of the ACM on Software Engineering, vol. 3, no. FSE, 7 2026

    2026

  27. [27]

    Uibenchkit: A unified toolkit for design-to-code model evaluation,

    C. T. Le, T. O. Y . Siang, J. Xiao, Y . Wan, and Y . Huo, “Uibenchkit: A unified toolkit for design-to-code model evaluation,”arXiv preprint arXiv:2605.13141, 2026

    Pith/arXiv arXiv 2026

  28. [28]

    Guipilot: A consistency-based mobile gui testing approach for detecting application-specific bugs,

    R. Liu, X. Teoh, Y . Lin, G. Chen, R. Ren, D. Poshyvanyk, and J. S. Dong, “Guipilot: A consistency-based mobile gui testing approach for detecting application-specific bugs,”Proc. ACM Softw. Eng., vol. 2, no. ISSTA, Jun. 2025. [Online]. Available: https://doi.org/10.1145/3728909

    doi:10.1145/3728909 2025

  29. [29]

    Designrepair: Dual-stream design guideline-aware frontend repair with large language models,

    M. Yuan, J. Chen, Z. Xing, A. Quigley, Y . Luo, T. Luo, G. Mohammadi, Q. Lu, and L. Zhu, “Designrepair: Dual-stream design guideline-aware frontend repair with large language models,” in2025 IEEE/ACM 47th International Conference on Software Engineering (ICSE 2025). IEEE, 2025, pp. 2483–2494

    2025

  30. [30]

    Seeing is fixing: Cross-modal reasoning with multimodal llms for visual software issue repair,

    K. Huang, J. Zhang, X. Xie, and C. Chen, “Seeing is fixing: Cross-modal reasoning with multimodal llms for visual software issue repair,” in 2025 40th IEEE/ACM International Conference on Automated Software Engineering (ASE 2025). IEEE, 2025, pp. 1156–1168

    2025

  31. [31]

    Swe-bench multimodal leaderboard,

    SWE-bench, “Swe-bench multimodal leaderboard,” 2026. [Online]. Available: https://www.swebench.com/index.html#multimodal

    2026

  32. [32]

    A design system built by ibm

    C. D. System, “A design system built by ibm.” 2026. [Online]. Available: https://github.com/carbon-design-system/carbon

    2026

  33. [33]

    Paddleocr 3.0 technical report,

    C. Cui, T. Sun, M. Lin, T. Gao, Y . Zhang, J. Liu, X. Wang, Z. Zhang, C. Zhou, H. Liu, Y . Zhang, W. Lv, K. Huang, Y . Zhang, J. Zhang, J. Zhang, Y . Liu, D. Yu, and Y . Ma, “Paddleocr 3.0 technical report,”

  34. [34]

    Uied: a hybrid tool for gui element detection,

    M. Xie, S. Feng, Z. Xing, J. Chen, and C. Chen, “Uied: a hybrid tool for gui element detection,” inProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering, 2020, pp. 1655–1659

    2020

  35. [35]

    Lightweight, robust, elegant syntax highlighting

    PrismJS, “Lightweight, robust, elegant syntax highlighting.” 2026. [Online]. Available: https://github.com/PrismJS/prism

    2026

  36. [36]

    Javascript syntax highlighter with language auto- detection and zero dependencies

    HighlightJS, “Javascript syntax highlighter with language auto- detection and zero dependencies.” 2026. [Online]. Available: https: //github.com/highlightjs/highlight.js

    2026

  37. [37]

    A markdown parser and compiler. built for speed

    MarkedJS, “A markdown parser and compiler. built for speed.” 2026. [Online]. Available: https://github.com/markedjs/marked

    2026

  38. [38]

    ReAct: Synergizing reasoning and acting in language models,

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao, “ReAct: Synergizing reasoning and acting in language models,” in International Conference on Learning Representations (ICLR), 2023

    2023

  39. [39]

    Openhands: An open platform for ai software developers as generalist agents,

    X. Wang, B. Li, Y . Song, F. F. Xu, X. Tang, M. Zhuge, J. Pan, Y . Song, B. Li, J. Singhet al., “Openhands: An open platform for ai software developers as generalist agents,”arXiv preprint arXiv:2407.16741, 2024

    Pith/arXiv arXiv 2024

  40. [40]

    [Online]

    OpenAI, “o3,” 2026. [Online]. Available: https://developers.openai.com/ api/docs/models/o3

    2026

  41. [41]

    text-embedding-3-small,

    ——, “text-embedding-3-small,” 2025. [Online]. Available: https: //platform.openai.com/docs/models/text-embedding-3-small

    2025

  42. [42]

    [Online]

    ——, “Gpt-4o,” 2026. [Online]. Available: https://platform.openai.com/ docs/models/gpt-4o

    2026

  43. [43]

    Claude 3.5 sonnet,

    Anthropic, “Claude 3.5 sonnet,” 2025. [Online]. Available: https: //docs.anthropic.com/en/docs/about-claude/models/all-models

    2025

  44. [44]

    Robertson and H

    S. Robertson and H. Zaragoza,The probabilistic relevance framework: BM25 and beyond. Now Publishers Inc, 2009, vol. 4

    2009

  45. [45]

    Agentless lite: Rag-based swe-bench software engineering scaffold,

    C. S. Xia, Y . Deng, S. Dunn, and L. Zhang, “Agentless lite: Rag-based swe-bench software engineering scaffold,” 2026. [Online]. Available: https://github.com/sorendunn/Agentless-Lite

    2026

  46. [46]

    Globant code fixer agent,

    Globant, “Globant code fixer agent,” 2026. [Online]. Available: https://ai.globant.com/

    2026

  47. [47]

    The ai coding agent,

    Zencoder, “The ai coding agent,” 2026. [Online]. Available: https: //zencoder.ai/

    2026

  48. [48]

    Refact ai agents,

    R. AI, “Refact ai agents,” 2026. [Online]. Available: https://refact.ai/

    2026

  49. [49]

    Svrepair: Structured visual reasoning for automated program repair,

    X. Tang, J. Wang, L. Luo, J. Xu, S. Zhou, D. Chen, W. Jiang, and Y . Li, “Svrepair: Structured visual reasoning for automated program repair,” arXiv preprint arXiv:2602.06090, 2026

    arXiv 2026

  50. [50]

    About a bpmn 2.0 rendering toolkit and web modeler

    Bpmn-io, “About a bpmn 2.0 rendering toolkit and web modeler.”

  51. [51]

    A high-performance, feature-packed library for creating interactive maps on the web

    Openlayers, “A high-performance, feature-packed library for creating interactive maps on the web.” 2026. [Online]. Available: https: //github.com/openlayers/openlayers

    2026

  52. [52]

    Automated auditing, performance metrics, and best practices for the web

    GoogleChrome, “Automated auditing, performance metrics, and best practices for the web.” 2026. [Online]. Available: https://github.com/ GoogleChrome/lighthouse

    2026

  53. [53]

    Simple yet flexible javascript charting for designers and developers

    ChartJS, “Simple yet flexible javascript charting for designers and developers.” 2026. [Online]. Available: https://github.com/chartjs/chart. js/

    2026

  54. [54]

    p5.js is a free and open-source javascript library for accessible creative coding

    processing, “p5.js is a free and open-source javascript library for accessible creative coding.” 2026. [Online]. Available: https: //github.com/processing/p5.js/

    2026

  55. [55]

    Automated repair of programs from large language models,

    Z. Fan, X. Gao, M. Mirchev, A. Roychoudhury, and S. H. Tan, “Automated repair of programs from large language models,” in45th International Conference on Software Engineering (ICSE), 2023, pp. 1469–1481

    2023

  56. [56]

    Automated program repair in the era of large pre-trained language models,

    C. S. Xia, Y . Wei, and L. Zhang, “Automated program repair in the era of large pre-trained language models,” in2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE 2023). IEEE, 2023, pp. 1482–1494

    2023

  57. [57]

    Enhancing automated program repair with solution design,

    J. Zhao, D. Yang, L. Zhang, X. Lian, Z. Yang, and F. Liu, “Enhancing automated program repair with solution design,” in39th International Conference on Automated Software Engineering (ASE), 2024, pp. 1706– 1718

    2024

  58. [58]

    Automated and context- aware repair of color-related accessibility issues for android apps,

    Y . Zhang, S. Chen, L. Fan, C. Chen, and X. Li, “Automated and context- aware repair of color-related accessibility issues for android apps,” in31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE), 2023, pp. 1255–1267

    2023

  59. [59]

    DesignRepair: Dual-Stream Design Guideline- Aware Frontend Repair with Large Language Models ,

    M. Yuan, J. Chen, Z. Xing, A. Quigley, Y . Luo, T. Luo, G. Mohammadi, Q. Lu, and L. Zhu, “ DesignRepair: Dual-Stream Design Guideline- Aware Frontend Repair with Large Language Models ,” inIEEE/ACM 47th International Conference on Software Engineering (ICSE). Los Alamitos, CA, USA: IEEE Computer Society, May 2025, pp. 646–646

    2025

  60. [60]

    Divide-and-conquer: Generating ui code from screenshots,

    Y . Wan, C. Wang, Y . Dong, W. Wang, S. Li, Y . Huo, and M. Lyu, “Divide-and-conquer: Generating ui code from screenshots,”Proceed- ings of the ACM on Software Engineering (FSE 2025), vol. 2, pp. 2099– 2122, 2025

    2025

  61. [61]

    Latcoder: Converting webpage design to code with layout-as-thought,

    Y . Gui, Z. Li, Z. Zhang, G. Wang, T. Lv, G. Jiang, Y . Liu, D. Chen, Y . Wan, H. Zhanget al., “Latcoder: Converting webpage design to code with layout-as-thought,” inProceedings of the 31st ACM SIGKDD Conference on Knowledge Discovery and Data Mining, 2025, pp. 721– 732

    2025

  62. [62]

    Effi- cientuicoder: A bidirectional token compression framework for efficient mllm-based ui code generation,

    J. Xiao, Z. Zhang, Y . Wan, Y . Huo, Y . Liu, and M. R. Lyu, “Effi- cientuicoder: A bidirectional token compression framework for efficient mllm-based ui code generation,”Proceedings of the ACM on Software Engineering, vol. 3, no. FSE, pp. 2396–2418, 2026

    2026

  63. [63]

    Envisioning future interactive web development: Editing webpage with natural language,

    T. H. Dang, J. Xiao, and Y . Huo, “Envisioning future interactive web development: Editing webpage with natural language,”2025 2nd IEEE/ACM International Conference on AI-powered Software (AIware), pp. 61–66, 2025. [Online]. Available: https://api.semanticscholar.org/ CorpusID:282592059

    2025

  64. [64]

    Comuicoder: Component-based reusable ui code generation for complex websites via semantic segmentation and element-wise feedback,

    J. Xiao, J. Qin, S. Li, M. H. Lam, Y . Wan, J.-t. Huang, Y . Huo, and M. R. Lyu, “Comuicoder: Component-based reusable ui code generation for complex websites via semantic segmentation and element-wise feedback,”arXiv preprint arXiv:2602.19276, 2026

    arXiv 2026

  65. [65]

    On using gui interaction data to improve text retrieval-based bug localization,

    J. Mahmud, N. De Silva, S. A. Khan, S. H. Mostafavi, S. H. Mansur, O. Chaparro, A. Marcus, and K. Moran, “On using gui interaction data to improve text retrieval-based bug localization,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE 2024), 2024, pp. 1–13

    2024

  66. [66]

    Toward the automated localization of buggy mobile app uis from bug descriptions,

    A. Saha, Y . Song, J. Mahmud, Y . Zhou, K. Moran, and O. Chaparro, “Toward the automated localization of buggy mobile app uis from bug descriptions,” inProceedings of the 33rd ACM SIGSOFT International Symposium on Software Testing and Analysis (ISSTA 2024), 2024, pp. 1249–1261

    2024

  67. [67]

    Translating video recordings of mobile app usages into replayable scenarios,

    C. Bernal-C ´ardenas, N. Cooper, K. Moran, O. Chaparro, A. Marcus, and D. Poshyvanyk, “Translating video recordings of mobile app usages into replayable scenarios,” inProceedings of the ACM/IEEE 42nd international conference on software engineering (ICSE 2020), 2020, pp. 309–321

    2020

  68. [68]

    V2s: A tool for translating video recordings of mobile app usages into replayable scenarios,

    M. Havranek, C. Bernal-C ´ardenas, N. Cooper, O. Chaparro, D. Poshy- vanyk, and K. Moran, “V2s: A tool for translating video recordings of mobile app usages into replayable scenarios,” in2021 IEEE/ACM 43rd International Conference on Software Engineering: Companion Proceedings (ICSE-Companion). IEEE, 2021, pp. 65–68

    2021

  69. [69]

    Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions,

    Z. Liu, C. Chen, J. Wang, M. Chen, B. Wu, X. Che, D. Wang, and Q. Wang, “Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions,” inProceedings of the IEEE/ACM 46th International Conference on Software Engineering (ICSE 2024), 2024, pp. 1–13

    2024

  70. [70]

    Prompting is all you need: Automated android bug replay with large language models,

    S. Feng and C. Chen, “Prompting is all you need: Automated android bug replay with large language models,” inProceedings of the 46th IEEE/ACM International Conference on Software Engineering (ICSE 2024), 2024, pp. 1–13

    2024

  71. [2025]

    Available: https://arxiv.org/abs/2507.05595

    [Online]. Available: https://arxiv.org/abs/2507.05595

    Pith/arXiv arXiv

  72. [2026]

    Available: https://github.com/bpmn-io/bpmn-js

    [Online]. Available: https://github.com/bpmn-io/bpmn-js