Pith. sign in

REVIEW 4 major objections 6 minor 48 references

Exploring the Capabilities of Vision-Language Models to Detect Visual Bugs in HTML5 <canvas> Applications

T0 review · 4 major / 6 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GPT-4o catches canvas app bugs up to 100% of the time with context

desk verdict Useful first dataset and feasibility study for VLM-based visual bug detection in canvas apps, but the headline 100% accuracy overstates what the evidence supports. read the letter →

arxiv 2501.09236 v1 pith:LODS72SE submitted 2025-01-16 cs.SE

Finlay Macklon , Cor-Paul Bezemer This is my paper · ORCID
classification cs.SE
keywords HTML5canvasvisualbugdetectionvision-languagemodelGPT-4oWebGLshaderinjectiontestoraclePixiJSsoftwaretesting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

HTML5 applications draw everything to a bitmap, so standard DOM-based web testing tools cannot see what is on screen, and visual bugs must often be found by hand. This paper asks whether a vision-language model (VLM) can act as an automatic visual test oracle by looking at a screenshot and deciding whether it matches the application's expected behavior. The authors built a dataset of 100 screenshots from 20 PixiJS canvas applications, injected four types of visual bugs with custom WebGL shaders, and prompted GPT-4o with different combinations of context. They report that when the prompt includes the application README, a description of possible visual bug types, and a bug-free screenshot, the VLM detects visual bugs with up to 100% per-application accuracy, and aggregate accuracy rises to 39% from a 26% baseline. The same prompting strategy keeps bug-free accuracy between 95% and 100%, suggesting the approach is usable as a low-cost complement to manual testing.

What carries the argument

The central mechanism is a prompting strategy, called AllContextExceptAssets, that gives the VLM a text description of the application (its README), a taxonomy of what visual bugs look like (Layout, Rendering, Appearance, State), and one bug-free screenshot as a visual reference, then asks whether a second screenshot from the same application contains a bug. The bug-free screenshot is the load-bearing piece: without it, median precision sits around 34–50%, and with it, median precision jumps to 100%. To build the evaluation data, the authors created a just-in-time bug-injection framework that overrides PixiJS' WebGL shaders with four custom shader programs, altering only the rendered bitmap and leaving the canvas object representation untouched. This setup allows controlled, repeatable injection of one bug instance per bug type into each of 20 applications.

What would settle it

Collect N real visual bugs from canvas application issue trackers that include a bug-free 'before' screenshot, run the AllContextExceptAssets prompt on each, and compare recall to the 33–100% range reported here; a chance-level result would show the injected shader bugs were easier than real ones.

Watch

Extended reading notes

Core claim

The paper's central claim is that a pre-trained vision-language model can detect visual bugs in HTML5 <canvas> applications when it is given enough context about what the screen should look like, and that the most important piece of context is a bug-free screenshot from the same application. With the best prompting strategy, AllContextExceptAssets, which adds the README and bug-type descriptions to the bug-free screenshot, GPT-4o achieves up to 100% accuracy on a per-application basis, raises overall accuracy across 100 screenshots to 39% from 26% with no context, and keeps bug-free accuracy between 95% and 100%. State bugs (objects missing or in the wrong state) are detected most reliably, with 33% average recall, while Appearance bugs are the hardest, at 14% average recall. The authors conclude that VLMs are not yet ready to replace manual testing, but they can automatically catch some of the most obvious visual bugs, including in procedurally generated graphics that previous asset-based oracles could not handle.

Load-bearing premise

The results transfer to real canvas development only if the shader-injected visual bugs look like the visual bugs developers actually encounter, and if the single author's manual labels on all 100 screenshots are correct.

Editorial extensions

If this is right

  • Developers of canvas applications can use a VLM plus a known-good screenshot as a lightweight regression-test oracle, catching a useful share of visual bugs without maintaining pixel-perfect snapshot baselines.
  • State bugs are the most promising target for VLM-based detection, while Appearance bugs will still need manual inspection.
  • Generating several VLM responses per screenshot and treating it as buggy if any response flags it (pass@k) improves detection rates, so sampling multiple outputs is a practical mitigation for the model's stochasticity.
  • The approach extends to procedurally generated graphics, which prior asset-based test oracles could not cover, because the VLM compares screenshots to textual and visual descriptions rather than to image assets.
  • Per-application accuracy varies widely, so teams should evaluate VLM usefulness case-by-case before adopting it.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • A natural extension is to run the same prompting strategy on real bug reports from canvas application issue trackers that include before-and-after screenshots; if accuracy drops, the shader-injected dataset is the reason.
  • Since the README ablation shows functional descriptions help and setup instructions hurt, automatically extracting only functionality-oriented sentences from a README could raise accuracy without any model training.
  • Providing canvas object representation (COR) property data, such as positions, sizes, and tint values, alongside the screenshot might specifically improve Appearance-bug detection, because those bugs are hard to see but easy to compute from the object representation.
  • The shader-injection method alters only the bitmap, so bugs caused by logic errors in the canvas object graph are underrepresented, and real deployment bugs may be harder for a VLM that cannot inspect application state.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper studies whether a pre-trained vision-language model (GPT-4o) can detect visual bugs in HTML5 <canvas> applications by prompting it with context about the application's expected functionality. The authors construct a dataset of 100 screenshots from 20 free/open-source PixiJS applications: 20 bug-free screenshots plus 80 screenshots created by injecting four bug types (State, Rendering, Layout, Appearance) through custom WebGL shaders. Five prompting strategies are compared, ranging from no context to combinations of README text, bug-type descriptions, a bug-free screenshot, and image assets. The headline result is that the strategy AllContextExceptAssets (README + bug descriptions + bug-free screenshot) yields up to 100% per-application accuracy, with 95–100% bug-free accuracy, while overall accuracy across all 100 screenshots is 39%. State bugs are detected most reliably and Appearance bugs least reliably. A small ablation study examines which parts of the README help or hurt recall.

Significance. The paper addresses a real and under-studied testing problem, and the core idea—prompting a VLM with a bug-free screenshot instead of maintaining a pixel-level oracle—is plausible and potentially useful for regression testing of <canvas> applications. Strengths include a publicly released dataset and injection framework, the use of real FOSS applications, a transparent comparison of prompting strategies, and honest reporting of per-application variability and threats to validity. If the findings generalize beyond the specific injection protocol, the approach would offer a low-cost complement to manual testing. However, because the bugs are produced by only four global shader transforms and are labeled by a single author, the quantitative results should be read as an upper-bound estimate for this protocol rather than as a general accuracy estimate for naturally occurring canvas bugs.

major comments (4)
  1. [Section 5.3.2 / Table 4; Section 8] Bug-injection realism is load-bearing for the paper's central feasibility claim. Table 4 shows that each of the four bug types is represented by exactly one hand-written WebGL shader applied uniformly to all 20 applications: opacity multiplied by 0.001 for State, a fixed band-scaling factor for Rendering, a scale/position change for Layout, and opacity multiplied by 0.5 for Appearance. Consequently, every injected instance of a type has the same canonical visual signature, independent of application semantics. The paper's own threat-to-validity paragraph acknowledges that 'the causes of our injected visual bugs may not necessarily be the same as real visual bugs,' but the abstract and conclusion state that VLMs can detect visual bugs with relatively high accuracy. Since a naturally occurring canvas bug is not a global shader transformation, the measured per-application and per-type accuracies are protocol-specific upper bounds. Please either add multiple distinct bug instances per type (e.g., several shader variants and semantic-level injections) and show that the accuracy holds, or explicitly restrict the central claim throughout the title, abstract, and conclusion to bugs of the visual character studied here.
  2. [Sections 5.4.1 and 6.2; Section 8] Ground-truth verification rests on a single author. Section 5.4.1 states that the author manually analyzed all 20 bug-free screenshots to ensure no bugs were present and manually verified that all 80 injected bugs were visible and human-detectable; Section 6.2 adds that the same author manually judged whether the VLM descriptions matched the observed bugs. No second annotator, inter-rater agreement statistic, or objective labeling criterion is reported. If any bug-free screenshot contains an unintended artifact (e.g., from timing-dependent rendering) or any injected bug is not clearly visible, all accuracy, precision, and recall figures move, and the ranking of prompting strategies could change. This is a load-bearing threat to the dataset's ground truth. Please add a second independent annotator with agreement statistics, or a pre-defined objective protocol (e.g., pixel-difference verification for the shader effects), and re-report the affected metrics.
  3. [Section 7 (RQ1), Table 6, Figures 4–6] The best strategy is selected on the same data used to report its accuracy. The experiments run all five prompting strategies on the same 100 screenshots, and AllContextExceptAssets is then presented as the best strategy, with its per-application accuracy (up to 100%) and per-bug-type recall reported prominently. There is no held-out split, pre-registration, or correction for multiple comparisons, so the reported advantage of AllContextExceptAssets over the other strategies is subject to selection bias. The 'up to 100%' figure is for a single application chosen post hoc, while Figure 6 shows per-application accuracies as low as 20%. For a feasibility study this is not fatal, but the strategy comparison should be framed as exploratory, and the chosen strategy should be validated on a hold-out set (new applications or new injections) before the paper claims that this prompting strategy is the recommended one.
  4. [Abstract and Section 7 (RQ1)] The headline wording overstates the aggregate results. The abstract says VLMs can be leveraged to detect visual bugs 'with relatively high accuracy' and highlights 'up to 100% per-application accuracy,' but Table 6 shows that the best overall strategy, AllContextExceptAssets, achieves only 39% mean accuracy across the 100 screenshots, with pass@1 at 39% and large standard deviations. Section 7 itself describes accuracy as low as 20% on some repetitions for several applications. The contribution should be reframed as a feasibility result: VLMs can detect some visual bugs in some canvas applications with high bug-free accuracy when a bug-free screenshot is provided, but the overall detection rate is modest. Please either report the aggregate 39% in the abstract or temper the 'relatively high accuracy' language.
minor comments (6)
  1. [Figure 6 caption] The caption says per-application accuracy is 'computed over the 100 screenshots,' but each application contributes only five screenshots (one bug-free plus four bug-injected). The caption should say 'over the five screenshots for each application.'
  2. [Table 5 / Section 6.1.1] For p5aholic-playground, README(Good) has 0 tokens. The paper should state how the empty README block was handled in the prompt (e.g., whether the 'README:' section was omitted or left blank), because an empty template block could itself be a confound in the ablation.
  3. [Appendix (AllContextExceptAssets / AllContext prompts)] The bug-free screenshot is introduced with a human-written, pre-determined response ('This screenshot is free of any visual bugs as defined in the provided set of categories') rather than letting the VLM analyze it. This is a reasonable way to provide context, but it may anchor the model; please report a sensitivity check in which the bug-free screenshot is supplied without the pre-written assertion.
  4. [Section 6 / Figure 4] AllContext is evaluated only on the 11 applications with image assets (55 screenshots), while AllContextExceptAssets is evaluated on all 20 applications (100 screenshots). The distributions in Figure 4 therefore mix different test sets; this should be stated next to the figure so readers do not compare the two strategies directly.
  5. [Section 6.3 (pass@k)] With n=4 repetitions, pass@4 is 1 whenever at least one of the four responses is correct, so pass@4 conveys little information beyond 'did any response succeed.' Consider reporting the proportion of screenshots with at least one correct response separately, or using a larger number of repetitions if pass@k is a central metric.
  6. [Title] The title refers to 'Vision-Language Models' in the plural, but the experiments use a single model (GPT-4o, snapshot gpt-4o-2024-08-06). Consider making the title or abstract explicit that this is a single-model feasibility study.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the study is an empirical benchmark with shader-injected ground truth, not a derivation that reduces to its inputs.

full rationale

This paper reports an empirical evaluation of GPT-4o on a dataset of canvas screenshots; it does not derive predictions from definitions or fit parameters that are then renamed as predictions. Ground-truth labels are assigned by an external injection procedure (Section 5.3.2, Table 4): each of the four shaders deterministically transforms the rendered frame, and the labels (State/Rendering/Layout/Appearance) are stored before any VLM is run. The VLM's outputs are therefore compared to labels that do not depend on the VLM or on the prompting strategy. The context inputs (README, bug descriptions, bug-free screenshot) are genuine inputs rather than fitted values; the paper shows the outcome is not forced by construction, since overall accuracy is only 39% for the best strategy, bug-free accuracy with the identical screenshot as context is a median of 98% rather than 100%, and per-application accuracy ranges from 20% to 100% across repetitions. The self-citations to the authors' prior taxonomy [26] and prior oracle work [25] supply the bug-type vocabulary and a prior empirical basis for the visual effects; they are external to the current VLM experiment and are not used to define the VLM's answer. The paper's own threats-to-validity section concedes that injected bug causes may differ from real bugs and that a single author verified screenshots and VLM descriptions; those are realism and coder-bias concerns, not circularity. No equation is shown to equal its own input, and no fitted parameter is relabeled as a prediction. Accordingly, no circular step is present.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no new physical or conceptual entities. Its central claim rests on the representativeness of the injected bugs, the correctness of manual labeling, and the reliability of the VLM answer-extraction pipeline. The bug-injection severity values are hand-chosen and directly influence how detectable the injected bugs are.

free parameters (2)
  • Bug injection severity = Multipliers 0.001 (State), 0.5 (Appearance), plus unspecified scaling factors for Rendering and Layout
    Hand-chosen in Section 5.3.2 and Table 4 to make injected bugs clearly visible. The difficulty of these injected bugs directly affects measured VLM recall, so they are a free parameter of the experimental setup.
  • GitHub star threshold = 11 stars
    Chosen in Section 5.1.1 to filter toy repositories. This selection criterion shapes the application sample and affects external validity, though not the internal detection mechanism.
assumptions (3)
  • domain assumption Shader-injected visual bugs are representative of real visual bugs in canvas applications.
    Invoked in Section 5.3 and defended in Section 8, where the authors state that the visual effects are similar to real bugs. If synthetic shader bugs differ from real bugs in visibility or appearance, the measured accuracy will not transfer.
  • domain assumption Single-author manual verification of screenshots and VLM descriptions is correct.
    Section 5.4.1 states that one author manually verified bug-free and bug-injected screenshots, and Section 6.2 relies on manual review of VLM descriptions to label true positives and false positives. Section 8 acknowledges coder bias as a threat to internal validity.
  • domain assumption The two-stage GPT-4o pipeline (visual analysis then structured answer extraction) preserves the model's actual judgments.
    Section 6.1 separates visual analysis from JSON answer extraction. Errors introduced by the extraction step would be attributed to bug detection accuracy.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Exploring the Capabilities of Vision-Language Models to Detect Visual Bugs in HTML5 <canvas> Applications." pith.science (2026). https://pith.science/paper/LODS72SE

@misc{pith2026250109236,
  author       = {Pith},
  title        = {Pith review of: Exploring the Capabilities of Vision-Language Models to Detect Visual Bugs in HTML5 <canvas> Applications},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LODS72SE}},
  note         = {Machine review of arXiv:2501.09236}
}
read the original abstract

The HyperText Markup Language 5 (HTML5) <canvas> is useful for creating visual-centric web applications. However, unlike traditional web applications, HTML5 <canvas> applications render objects onto the <canvas> bitmap without representing them in the Document Object Model (DOM). Mismatches between the expected and actual visual output of the <canvas> bitmap are termed visual bugs. Due to the visual-centric nature of <canvas> applications, visual bugs are important to detect because such bugs can render a <canvas> application useless. As we showed in prior work, Asset-Based graphics can provide the ground truth for a visual test oracle. However, many <canvas> applications procedurally generate their graphics. In this paper, we investigate how to detect visual bugs in <canvas> applications that use Procedural graphics as well. In particular, we explore the potential of Vision-Language Models (VLMs) to automatically detect visual bugs. Instead of defining an exact visual test oracle, information about the application's expected functionality (the context) can be provided with the screenshot as input to the VLM. To evaluate this approach, we constructed a dataset containing 80 bug-injected screenshots across four visual bug types (Layout, Rendering, Appearance, and State) plus 20 bug-free screenshots from 20 <canvas> applications. We ran experiments with a state-of-the-art VLM using several combinations of text and image context to describe each application's expected functionality. Our results show that by providing the application README(s), a description of visual bug types, and a bug-free screenshot as context, VLMs can be leveraged to detect visual bugs with up to 100% per-application accuracy.

Figures

Figures reproduced from arXiv: 2501.09236 by the authors.

Figure 1
Figure 1. Two screenshots from a Breakthrough clone ( [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Overview of our dataset construction. 5.1.1 Collecting GitHub Dependents of PixiJS We began our process for collecting FOSS HTML5 <canvas> applications by collecting GitHub repositories that are dependents of PixiJS. Because the PixiJS rendering framework has undergone several major architectural changes since its inception, we chose to focus on the most recent versions of PixiJS (v6 and v7). Both PixiJS v6 and v7 a… view at source ↗
Figure 3
Figure 3. Four bug-free screenshots ((a), (c), (e), (g)) and four bug-injected [PITH_FULL_IMAGE:figures/full_fig_p018_3.png] view at source ↗
Figures from the paper (5 more)
Figure 4
Figure 4. Figure 4: Distributions of accuracy (%) yielded in experiments using VLMs to [PITH_FULL_IMAGE:figures/full_fig_p023_4.png]
Figure 5
Figure 5. Figure 5: Distributions of precision (%) and recall (%) per bug type when using [PITH_FULL_IMAGE:figures/full_fig_p024_5.png]
Figure 6
Figure 6. Figure 6: Distributions of accuracy (%) per-application when using VLMs to [PITH_FULL_IMAGE:figures/full_fig_p026_6.png]
Figure 7
Figure 7. Figure 7: Differences in recall (%) per-application when using VLMs to detect vi [PITH_FULL_IMAGE:figures/full_fig_p027_7.png]
Figure 8
Figure 8. Figure 8: A bug-injected and a bug-free screenshot from the same [PITH_FULL_IMAGE:figures/full_fig_p031_8.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

48 extracted references · 44 canonical work pages

  1. [1]

    Maintenance of Automated Test Suites in Industry: An Empirical Study on Visual GUI Testing,

    E. Al´ egroth, R. Feldt, and P. Kolstr¨ om, “Maintenance of Automated Test Suites in Industry: An Empirical Study on Visual GUI Testing,” Information and Software Technology, vol. 73, pp. 66–80, 2016

    work page 2016

  2. [2]

    How King Uses AI to Test Candy Crush Saga

    A. Andelkovic. “How King Uses AI to Test Candy Crush Saga.” (Dec. 30, 2019), [Online]. Available: https://www.infoq.com/articles/candy- crush-QA-AI-saga/ (visited on 09/19/2024)

    work page 2019

  3. [3]

    About Canvas

    Apple. “About Canvas.” (Sep. 18, 2013), [Online]. Available: https:// developer.apple.com/library/archive/documentation/AudioVideo/ Conceptual/HTML-canvas-guide/Introduction/Introduction.html (visited on 09/29/2021)

    work page 2013

  4. [4]

    A Framework for Automated Testing of JavaScript Web Applications,

    S. Artzi, J. Dolby, S. H. Jensen, A. Møller, and F. Tip, “A Framework for Automated Testing of JavaScript Web Applications,” in Proceedings of the 33rd International Conference on Software Engineering (ICSE) , 2011, pp. 571–580. Detecting Visual Bugs in HTML5 <canvas> applications 35

    work page 2011

  5. [5]

    Mining Questions Asked by Web Developers,

    K. Bajaj, K. Pattabiraman, and A. Mesbah, “Mining Questions Asked by Web Developers,” in Proceedings of the 11th Working Conference on Mining Software Repositories (MSR) , 2014, pp. 112–121

    work page 2014

  6. [6]

    Web Canvas Testing Through Visual Inference,

    M. Bajammal and A. Mesbah, “Web Canvas Testing Through Visual Inference,” in 2018 IEEE 11th International Conference on Software Testing, Verification and Validation (ICST) , IEEE, 2018, pp. 193–203

    work page 2018

  7. [7]

    A Survey on the Use of Computer Vision to Improve Software Engineering Tasks,

    M. Bajammal, A. Stocco, D. Mazinanian, and A. Mesbah, “A Survey on the Use of Computer Vision to Improve Software Engineering Tasks,” IEEE Transactions on Software Engineering , 2020

    work page 2020

  8. [8]

    Graphical User Interface (GUI) Testing: Systematic Mapping and Repository,

    I. Banerjee, B. Nguyen, V. Garousi, and A. Memon, “Graphical User Interface (GUI) Testing: Systematic Mapping and Repository,” Infor- mation and Software Technology, vol. 55, no. 10, pp. 1679–1694, 2013

    work page 2013

Show all 48 references
  1. [9]

    Search Based Path and Input Data Generation for Web Application Testing,

    M. Biagiola, F. Ricca, and P. Tonella, “Search Based Path and Input Data Generation for Web Application Testing,” in Search Based Soft- ware Engineering: 9th International Symposium, SSBSE 2017, Pader- born, Germany, September 9-11, 2017, Proceedings 9 , Springer, 2017, pp. 18–32

    2017

  2. [10]

    Diversity-Based Web Test Generation,

    M. Biagiola, A. Stocco, F. Ricca, and P. Tonella, “Diversity-Based Web Test Generation,” in Proceedings of the 2019 27th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering (FSE) , 2019, pp. 142–153

    2019

  3. [11]

    GUI Testing Using Computer Vision,

    T.-H. Chang, T. Yeh, and R. C. Miller, “GUI Testing Using Computer Vision,” in Proceedings of the SIGCHI Conference on Human Factors in Computing Systems , 2010, pp. 1535–1544

    2010

  4. [12]

    Evaluating Large Language Models Trained on Code,

    M. Chen, J. Tworek, H. Jun, et al., “Evaluating Large Language Models Trained on Code,” arXiv preprint arXiv:2107.03374 , 2021

    2021 arXiv

  5. [13]

    Search- Based Testing for Scratch Programs,

    A. Deiner, C. Fr¨ adrich, G. Fraser, S. Geserer, and N. Zantner, “Search- Based Testing for Scratch Programs,” in Search-Based Software Engi- neering: 12th International Symposium, SSBSE 2020, Bari, Italy, Octo- ber 7–8, 2020, Proceedings 12 , Springer, 2020, pp. 58–72

    2020

  6. [14]

    Web Application Testing: A Systematic Literature Review,

    S. Do˘ gan, A. Betin-Can, and V. Garousi, “Web Application Testing: A Systematic Literature Review,” Journal of Systems and Software , vol. 91, pp. 174–201, 2014

    2014

  7. [15]

    Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat,

    S. Feng, H. Lu, J. Jiang, et al., “Enabling Cost-Effective UI Automation Testing with Retrieval-Based LLMs: A Case Study in WeChat,” arXiv preprint arXiv:2409.07829, 2024

    2024 arXiv

  8. [16]

    Exploring the Dependencies of a Repository

    GitHub. “Exploring the Dependencies of a Repository.” (Sep. 20, 2024), [Online]. Available: https://docs.github.com/en/code- security/ supply- chain- security/understanding- your- software- supply- chain/exploring- the- dependencies- of- a- repository(visited on 09/20/2024)

    2024

  9. [17]

    An Oracle Based on Im- age Comparison for Regression Testing of Web Applications,

    A. Hori, S. Takada, H. Tanno, and M. Oinuma, “An Oracle Based on Im- age Comparison for Regression Testing of Web Applications,” in SEKE, 2015, pp. 639–645

    2015

  10. [18]

    Visual Testing of Graphical User Interfaces: An Exploratory Study Towards Systematic Definitions and 36 Finlay Macklon, Cor-Paul Bezemer Approaches,

    A. Issa, J. Sillito, and V. Garousi, “Visual Testing of Graphical User Interfaces: An Exploratory Study Towards Systematic Definitions and 36 Finlay Macklon, Cor-Paul Bezemer Approaches,” in 2012 14th IEEE International Symposium on Web Sys- tems Evolution (WSE) , IEEE, 2012, ...

    2012

  11. [19]

    A Study of Using Multimodal LLMs for Non-Crash Functional Bug Detection in Android Apps,

    B. Ju, J. Yang, T. Yu, et al. , “A Study of Using Multimodal LLMs for Non-Crash Functional Bug Detection in Android Apps,” arXiv preprint arXiv:2407.19053, 2024

    2024 arXiv

  12. [20]

    The Promises and Perils of Mining GitHub,

    E. Kalliamvakou, G. Gousios, K. Blincoe, L. Singer, D. M. German, and D. Damian, “The Promises and Perils of Mining GitHub,” inProceedings of the 11th working conference on mining software repositories , 2014, pp. 92–101

    2014

  13. [21]

    Visualwebarena: Evaluating multi- modal agents on realistic visual web tasks,

    J. Y. Koh, R. Lo, L. Jang, et al. , “Visualwebarena: Evaluating multi- modal agents on realistic visual web tasks,”arXiv preprint arXiv:2401.13649, 2024

    2024 arXiv

  14. [22]

    Visual vs. DOM-Based Web Locators: An Empirical Study,

    M. Leotta, D. Clerissi, F. Ricca, and P. Tonella, “Visual vs. DOM-Based Web Locators: An Empirical Study,” in Web Engineering: 14th Interna- tional Conference, ICWE 2014, Toulouse, France, July 1-4, 2014. Pro- ceedings 14, Springer, 2014, pp. 322–340

    2014

  15. [23]

    On the Accu- racy, Efficiency, and Reusability of Automated Test Oracles for Android Devices,

    Y.-D. Lin, J. F. Rojas, E. T.-H. Chu, and Y.-C. Lai, “On the Accu- racy, Efficiency, and Reusability of Automated Test Oracles for Android Devices,” IEEE Transactions on Software Engineering , vol. 40, no. 10, pp. 957–970, 2014

    2014

  16. [24]

    Vision-Driven Automated Mobile GUI Testing via Multimodal Large Language Model,

    Z. Liu, C. Li, C. Chen, et al. , “Vision-Driven Automated Mobile GUI Testing via Multimodal Large Language Model,”arXiv preprint arXiv:2407.03037, 2024

    2024 arXiv

  17. [25]

    Automatically Detect- ing Visual Bugs in HTML5 Canvas Games,

    F. Macklon, M. R. Taesiri, M. Viggiato, et al. , “Automatically Detect- ing Visual Bugs in HTML5 Canvas Games,” in Proceedings of the 37th IEEE/ACM International Conference on Automated Software Engineer- ing (ASE) , 2022, pp. 211–221

    2022

  18. [26]

    A Taxonomy of Testable HTML5 Canvas Issues,

    F. Macklon, M. Viggiato, N. Romanova, C. Buzon, D. Paas, and C.-P. Bezemer, “A Taxonomy of Testable HTML5 Canvas Issues,”IEEE Trans- actions on Software Engineering , vol. 49, no. 6, pp. 3647–3659, 2023

    2023

  19. [27]

    Finding HTML Presentation Fail- ures Using Image Comparison Techniques,

    S. Mahajan and W. G. Halfond, “Finding HTML Presentation Fail- ures Using Image Comparison Techniques,” in Proceedings of the 29th ACM/IEEE International Conference on Automated Software Engineer- ing (ASE) , 2014, pp. 91–96

    2014

  20. [28]

    The Fault in Our Stars: Designing Reproducible Large-scale Code Analysis Experiments,

    P. Maj, S. Muroya, K. Siek, L. Di Grazia, and J. Vitek, “The Fault in Our Stars: Designing Reproducible Large-scale Code Analysis Experiments,” in 38th European Conference on Object-Oriented Programming (ECOOP 2024), Schloss Dagstuhl–Leibniz-Zentrum f¨ ur Informatik, 2024

    2024

  21. [29]

    State-Based Testing of AJAX Web Applications,

    A. Marchetto, P. Tonella, and F. Ricca, “State-Based Testing of AJAX Web Applications,” in 2008 1st International Conference on Software Testing, verification, and validation , IEEE, 2008, pp. 121–130

    2008

  22. [30]

    Crawling AJAX by Infer- ring User Interface State Changes,

    A. Mesbah, E. Bozdag, and A. Van Deursen, “Crawling AJAX by Infer- ring User Interface State Changes,” in 2008 Eighth International Con- ference on Web Engineering , IEEE, 2008, pp. 122–134. Detecting Visual Bugs in HTML5 <canvas> applications 37

    2008

  23. [31]

    Invariant-Based Automatic Testing of AJAX User Interfaces,

    A. Mesbah and A. Van Deursen, “Invariant-Based Automatic Testing of AJAX User Interfaces,” in 2009 IEEE 31st International Conference on Software Engineering (ICSE) , IEEE, 2009, pp. 210–220

    2009

  24. [32]

    Handling User Input in HTML5 Canvas-Based Games

    K. Moot. “Handling User Input in HTML5 Canvas-Based Games.” (Jul. 23, 2012), [Online]. Available: https://developer.ibm.com/tutorials/ wa-games (visited on 09/29/2021)

    2012

  25. [33]

    De- tecting and Summarizing GUI Changes in Evolving Mobile Apps,

    K. Moran, C. Watson, J. Hoskins, G. Purnell, and D. Poshyvanyk, “De- tecting and Summarizing GUI Changes in Evolving Mobile Apps,” in Proceedings of the 33rd ACM/IEEE International Conference on Auto- mated Software Engineering (ASE) , 2018, pp. 543–553

    2018

  26. [34]

    Unity Application Testing Automation with Appium and Image Recognition,

    M. Mozgovoy and E. Pyshkin, “Unity Application Testing Automation with Appium and Image Recognition,” in Tools and Methods of Program Analysis: 4th International Conference, TMPA 2017, Moscow, Russia, March 3-4, 2017, Revised Selected Papers 4, Springer, 2018, pp. 139–150

    2017

  27. [35]

    Canvas tutorial

    Mozilla. “Canvas tutorial.” (Aug. 27, 2024), [Online]. Available: https: / / developer . mozilla . org / en - US / docs / Web / API / Canvas _ API / Tutorial (visited on 09/19/2024)

    2024

  28. [36]

    WebGL tutorial

    Mozilla. “WebGL tutorial.” (Jul. 26, 2024), [Online]. Available: https: / / developer . mozilla . org / en - US / docs / Web / API / WebGL _ API / Tutorial (visited on 09/19/2024)

    2024

  29. [37]

    Curating GitHub for Engineered Software Projects,

    N. Munaiah, S. Kroh, C. Cabrey, and M. Nagappan, “Curating GitHub for Engineered Software Projects,”Empirical Software Engineering, vol. 22, pp. 3219–3253, 2017

    2017

  30. [38]

    Web Test Automation: Insights from the Grey Literature,

    F. Ricca and A. Stocco, “Web Test Automation: Insights from the Grey Literature,” in International Conference on Current Trends in Theory and Practice of Informatics , Springer, 2021, pp. 472–485

    2021

  31. [39]

    NaviQAte: Functionality- Guided Web Application Navigation,

    M. Shahbandeh, P. Alian, N. Nashid, and A. Mesbah, “NaviQAte: Functionality- Guided Web Application Navigation,” arXiv preprint arXiv:2409.10741, 2024

    2024 arXiv

  32. [40]

    A Framework of a Hybrid Focused Web Crawler,

    Y. Sun, P. Jin, and L. Yue, “A Framework of a Hybrid Focused Web Crawler,” in 2008 Second International Conference on Future Gener- ation Communication and Networking Symposia , IEEE, vol. 2, 2008, pp. 50–53

    2008

  33. [41]

    VideoGameBunny: Towards Vision Assistants for Video Games,

    M. R. Taesiri and C.-P. Bezemer, “VideoGameBunny: Towards Vision Assistants for Video Games,” in Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV) , 2025

    2025

  34. [42]

    GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?

    M. R. Taesiri, T. Feng, C.-P. Bezemer, and A. Nguyen, “GlitchBench: Can Large Multimodal Models Detect Video Game Glitches?” In Pro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), 2024, pp. 22 444–22 455

    2024

  35. [43]

    Support for Finding Presentation Failures by Using Computer Vision Techniques,

    H. Tanno and Y. Adachi, “Support for Finding Presentation Failures by Using Computer Vision Techniques,” in 2018 IEEE International Conference on Software Testing, Verification and Validation Workshops (ICSTW), IEEE, 2018, pp. 356–363

    2018

  36. [44]

    Prioritizing Natural Lan- guage Test Cases Based on Highly-Used Game Features,

    M. Viggiato, D. Paas, and C.-P. Bezemer, “Prioritizing Natural Lan- guage Test Cases Based on Highly-Used Game Features,” in Proceedings 38 Finlay Macklon, Cor-Paul Bezemer of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softw...

    2023

  37. [45]

    Identifying Similar Test Cases that are Specified in Natural Language,

    M. Viggiato, D. Paas, C. Buzon, and C.-P. Bezemer, “Identifying Similar Test Cases that are Specified in Natural Language,” IEEE Transactions on Software Engineering , vol. 49, no. 3, pp. 1027–1043, 2022

    2022

  38. [46]

    Using Natural Language Processing Techniques to Improve Manual Test Case Descrip- tions,

    M. Viggiato, D. Paas, C. Buzon, and C.-P. Bezemer, “Using Natural Language Processing Techniques to Improve Manual Test Case Descrip- tions,” in Proceedings of the 44th International Conference on Software Engineering: Software Engineering in Practice (ICSE) , 2022, pp. 311– 320

    2022

  39. [47]

    Sikuli: Using GUI Screenshots for Search and Automation,

    T. Yeh, T.-H. Chang, and R. C. Miller, “Sikuli: Using GUI Screenshots for Search and Automation,” in Proceedings of the 22nd annual ACM symposium on User interface software and technology , 2009, pp. 183– 192

    2009

  40. [48]

    Vision-language models for vision tasks: A survey,

    J. Zhang, J. Huang, S. Jin, and S. Lu, “Vision-language models for vision tasks: A survey,” IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024. Detecting Visual Bugs in HTML5 <canvas> applications 39 Appendix Prompts used in experiments NoContext Message sent...

    2024

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.