Pith. sign in

REVIEW 4 major objections 5 minor 93 references

GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read GUIPilot reports 94.5% precision and 99.6% recall when checking mobile app screens against design mock-ups.

desk verdict GUIPilot is a genuinely useful approach, but its evaluation needs a number audit and a proper threshold split before the claims hold. read the letter →

arxiv 2506.07385 v1 pith:FNGSKFEO submitted 2025-06-09 cs.SE

Ruofan Liu , Xiwen Teoh , Yun Lin , Guanjie Chen , Ruofei Ren , Denys Poshyvanyk , Jin Song Dong This is my paper
classification cs.SE
keywords GUItestingdesignmock-upsscreeninconsistencyprocesswidgetalignmentvision-languagemodelmobileapplicationsAndroid
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that a mobile app's implementation can be checked automatically against the designer's mock-ups, catching the application-specific bugs that crash-oriented testing misses. GUIPilot detects two kinds of drift: screen inconsistencies, meaning missing, extra, or semantically changed widgets, and process inconsistencies, meaning transitions specified in the mock-up that do not happen or land on the wrong screen. On a benchmark of 80 Android apps and 160 simulated mock-ups it reports 94.5% precision and 99.6% recall for screen inconsistencies, beating the prior GVT approach by 66.2% and 56.6%, and reports zero errors in detecting process inconsistencies on the 100 retained transitions. In an industrial case study on a trading app with 19 usable design mock-ups, it found nine inconsistencies that the app's own experts confirmed.

What carries the argument

The load-bearing object is the widget-alignment formulation built on the longest common subsequence (LCS). A screen is abstracted into a sequence of widgets sorted by a (y, x) partial order; pairwise similarity is the product of position distance, area ratio, aspect-ratio ratio, and a type-match score; dynamic programming then finds the global maximum-similarity common subsequence, tolerating insertions and deletions. This turns screen matching into an optimization problem instead of a nearest-neighbor search. The second mechanism is the visual prompt for the vision-language model: annotated bounding-box IDs over interactable widgets, a closed action space of seven commands (click, long press, send keys, scroll, swipe, drag and drop, go back), a few-shot example, and iterative feedback when an action produces no transition. Together these let GUIPilot compare screens globally and execute transitions without manual scripting.

What would settle it

Give GUIPilot a held-out set of authentic designer-created mock-ups that were not produced by mutating the apps' own screenshots, label the real inconsistencies by hand, and compare precision and recall; if they fall far below 94.5% and 99.6%, the simulated-mock-up evaluation overstated the tool's transfer.

Watch

Extended reading notes

Core claim

GUIPilot's central claim is that both kinds of design-implementation drift can be reduced to well-defined matching problems. Each screen, whether from a mock-up or from the running app, is converted by a trained object detector into a set of widgets carrying position, size, and type; screen comparison then becomes an optimal alignment of two sequences sorted top-to-bottom and left-to-right, solved as a longest-common-subsequence dynamic program whose similarity score multiplies position, area, aspect-ratio, and type agreement. This global alignment is what fixes the failure mode of the prior GVT approach, in which local nearest-neighbor matching mispairs widgets when a row is inserted or deleted. For transitions, a vision-language model receives the mock-up's natural-language description plus a screenshot with interactable widgets highlighted and numbered, and must emit actions from a fixed action space; executing those actions on the app and comparing the resulting screen with the expected mock-up screen detects missing or wrong transitions. The paper reports that the pipeline finds screen inconsistencies at 94.5% precision and 99.6% recall, catches all 100 mutated process inconsistencies with no errors, and in the industrial case study found nine expert-confirmed bugs.

Load-bearing premise

The evaluation treats mutated screenshots of the very apps being tested as stand-ins for real designer mock-ups, so the reported accuracy may not transfer to genuine design files.

Editorial extensions

If this is right

  • Screen matching no longer breaks when a row of widgets is inserted or deleted, because alignment is global rather than local nearest-neighbor matching.
  • Design-mock-up validation becomes end-to-end automatable: given mock-ups in the paper's meta-model format, GUIPilot can drive the app through the specified flow and compare every reached screen.
  • The pipeline reports violation type (extra, missing, semantic change) rather than only a difference, so testers can file targeted bug reports.
  • The reported per-screen matching time of about 0.001 seconds and transition time of about 0.19 seconds make consistency checks cheap enough to run on every build.
  • If the industrial case study generalizes, the tool can surface subtle defects, such as a missing small widget or a replaced label, that manual inspection of a real trading app missed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Beyond the paper, the same screenshot-to-widget-sequence abstraction could apply to web or desktop UIs, since it relies on bounding boxes and widget classes rather than on Android internals.
  • Beyond the paper, a public dataset of genuine designer-produced mock-ups paired with the corresponding app screens would give a harder transfer test than the mutation-based mock-ups used in the benchmark.
  • Beyond the paper, the action-completion protocol could be reused as an automatic regression-test generator: once a transition description is turned into executable actions, those actions remain useful even when no inconsistency is found.
  • Beyond the paper, moving from an external vision-language API to a locally hosted model would remove network latency and cost as operational dependencies, making continuous-integration deployment more practical.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. GUIPilot proposes an end-to-end approach for detecting inconsistencies between mobile design mock-ups and app implementations. Screen inconsistencies are detected by detecting widgets on both screens, ordering them partially, and solving an LCS-based widget alignment problem; process inconsistencies are detected by translating described screen transitions into executable actions via a visual prompt to a VLM, then comparing the reached screen with the target screen using a similarity threshold. The paper evaluates the approach on 80 apps with 160 simulated mock-ups, reports very high precision and recall for both screen and process inconsistency detection, and presents a case study on an industrial trading app in which nine inconsistencies were confirmed by experts.

Significance. If the reported results held, GUIPilot would fill a real industrial gap: automated validation of GUI implementations against design mock-ups, going beyond crash-oriented GUI testing. The widget-alignment formulation and the VLM-based action-completion prompt are sensible technical contributions, and the industrial case study with expert-confirmed bugs is a valuable piece of evidence. The availability of the code and dataset artifacts is also a strength. However, the central quantitative claims currently rest on internally inconsistent headline numbers and on evaluation setups in which key thresholds were selected on the same data used to report performance, so the empirical contribution needs substantial reworking before the stated significance can be accepted.

major comments (4)
  1. [Abstract / Section 1 / Table 3] The headline screen-consistency numbers are mutually inconsistent. The full-text abstract reports 99.8% precision and 98.6% recall, the Introduction reports 94.5% precision and 99.6% recall, and Table 3 reports per-mutation-type precision values of 0.987, 0.997, 0.998, 0.996, and 1.000. Since a weighted average of values all at least 0.987 cannot equal 0.945, the 94.5% figure is arithmetically impossible if it is an aggregate of Table 3. Please report one consistent set of numbers and explicitly state how the per-type results are aggregated into the headline precision and recall.
  2. [Section 4 (Configuration) / Section 4.2] The process-inconsistency result is circular. The configuration paragraph states that the screen matching threshold epsilon_screen is selected using the 100 mutated design mock-ups introduced in the process consistency experiment, and Section 4.2.2 then reports 100% precision and 100% recall on those same 100 mutated processes. This means the reported zero-error result is a fitted value rather than an evaluation of generalization. Please evaluate the threshold on a held-out set or use cross-validation, or clearly report the result as development-set performance rather than as the achieved precision and recall.
  3. [Section 4 (Configuration) / Section 4.1] The same selection-on-evaluation-data issue applies to the screen-consistency experiment. The hyperparameters alpha and delta in Algorithm 1 are selected by grid search using the screen consistency experimental performance described in Section 4.1, and no separate validation or test split is described for that experiment. Consequently, the RQ1 precision, recall, and Jaccard indices may be over-optimistic estimates of performance on new screens. Please clarify the data split used for hyperparameter selection, or re-run the experiment with proper separation between parameter tuning and evaluation.
  4. [Section 4.1.1 / Section 5.1] The simulated mock-ups are not design mock-ups. In Section 4.1.1, the 'simulated mock-ups' are real application screenshots labeled by experts and then mutated, and the threats-to-validity discussion in Section 5.1 explicitly acknowledges that actual design mock-ups were unavailable. The industrial case study, which did use real mock-ups, required manual navigation information and reports no precision or recall. As a result, the quantitative screen-consistency claims do not yet establish performance on the target input type. Please either add an evaluation on real design mock-ups or explicitly reframe the headline claims as applying to simulated mock-ups, while treating the case study as qualitative evidence only.
minor comments (5)
  1. [Section 4.3.2] The sentence beginning 'which overall achieves a satisfactory detection rate ( As a reference...' has a malformed parenthesis and the comparison to the COCO benchmark is not directly relevant to GUI widget detection; please rephrase or remove it.
  2. [Figure 10] The caption and axes contain the typo 'Trail' instead of 'Trial' in '1 Trail Success Rate' and '2 Trail Success Rate'.
  3. [Section 4.2.2] The text says 'following the workflow in Figure 3.3'; this should refer to Figure 3 or Section 3.3.
  4. [Section 5.1] In the external validity discussion, the sentence 'Any shortcomings can be manually corrected' is vague; please specify which shortcomings require manual correction and how that affects the automated pipeline's claims.
  5. [Section 4.2.1 / Abstract] The abstract says the experiments use 160 design mock-ups, but Section 4.2.1 states that only 100 design mock-ups were retained for the process consistency experiment; please clarify whether the 160 figure refers to the full collected set and state clearly which experiments use which subset.

Circularity Check

2 steps flagged · score 6.0 of 10

Process-consistency '100%' precision/recall is reported on the same 100 mutated mock-ups used to select its screen-matching threshold, and the screen-consistency hyperparameters are grid-searched on the same experiment whose results are then reported as performance.

  1. fitted input called prediction [Section 4, Configuration paragraph; Section 4.2.2 Results]
    "Regarding process inconsistencies, we choose the optimal screen matching threshold (𝜖𝑠𝑐𝑟𝑒𝑒𝑛 in Section 3.3.3) using the 100 mutated design mock-ups introduced in the process consistency experiment (Section 4.2). The threshold is selected to be 0.73, which achieves the highest F1-score. ... Our approach yielded Precision and Recall scores of 100%, confirming that all introduced process inconsistencies were successfully detected."

    The threshold 𝜖𝑠𝑐𝑟𝑒𝑒𝑛 is the decision boundary that determines whether a transition is reported as a process inconsistency. The paper explicitly selects this threshold on the 100 mutated design mock-ups that constitute the RQ2 process-consistency evaluation, and then reports 100% precision and recall on that same set. There is no separate validation set described. The reported perfect scores are therefore the performance of the fitted threshold on its own training data, not an independent estimate of detection accuracy. This is the canonical fitted-input-called-prediction pattern: the parameter was tuned to maximize F1 on the exact cases later counted as true positives and true negatives.

  2. fitted input called prediction [Section 4, Configuration paragraph; Section 4.1.4 Results (Table 3)]
    "The hyperparameters 𝛼 and𝛿 in Algorithm 1 are selected through grid search. We use the screen consistency experimental performance (as introduced in Section 4.1) as the metric to guide the selection process."

    The grid search for α and δ is guided by the performance of the very screen-consistency experiment whose results are later reported in Table 3 as GUIPilot's precision, recall, and Jaccard index. With no held-out validation set described, the reported screen-consistency numbers are post-selection values rather than out-of-sample predictions. The headline claim of 94.5% precision and 99.6% recall is thus partly an artifact of tuning the alignment similarity formula on the same mutated screens used for evaluation. This does not invalidate the algorithmic contribution, but it means the quantitative screen-consistency claim is not independent of the fitted hyperparameters.

full rationale

The core algorithmic derivation is not circular: widget detection, LCS-based widget alignment, and the VLM visual prompt are defined independently of the reported outcomes, and the paper does not rely on a self-citation chain to justify its main technical claims. The baseline comparison to GVT uses externally published configurations, and no load-bearing uniqueness theorem is imported from the authors' prior work. However, the evaluation protocol contains two concrete fit-to-test-set steps. First, the process-inconsistency threshold ε_screen is chosen on the 100 mutated mock-ups that are then used to report the 100% precision/recall result; this makes the perfect process-consistency score a direct consequence of selecting the threshold on the same samples. Second, the screen-consistency hyperparameters α and δ are grid-searched using the same screen-consistency experiment whose results are presented as the method's performance, so the reported precision/recall values are optimistically biased and not independent estimates. These are not merely external-validity concerns; they are cases where reported 'predictions' are evaluated on the data used to fit their controlling parameters. The industrial case study is qualitative and is not used to claim precision/recall, so it does not enter the circularity analysis. Overall, the paper has substantial independent content, but two of its headline quantitative claims reduce, at least in part, to fitted values. Score 6 reflects partial circularity in the reported evaluation, not circularity in the derivation of the algorithm itself.

Assumptions & free parameters 7 free parameters · 6 assumptions · 0 invented entities

The central claims rest on several domain assumptions about mock-up structure and widget semantics, plus a set of fitted thresholds. The process inconsistency result is especially sensitive to epsilon_screen, which was tuned on the evaluation set itself.

free parameters (7)
  • alpha (position scaling factor) = 10
    Scaling factor in sim_pos (Algorithm 1), selected by grid search on the screen consistency experimental performance (Section 4).
  • delta (type mismatch penalty) = 0.5
    Down-weight factor when widget types differ, selected by grid search (Section 4).
  • epsilon_screen (process screen matching threshold) = 0.73
    Threshold for Equation 5, chosen to maximize F1 on the same 100 mutated design mock-ups used for the process consistency evaluation (Section 4, Configuration).
  • epsilon_ed (text edit threshold) = 0.95
    Adopted from GVT [52], used to flag text semantic changes (Section 3.2.3).
  • epsilon_color (RGB color threshold) = 0.05
    Adopted from GVT [52], for top-3 color differences (Section 3.2.3).
  • epsilon_binary (binary color pixel threshold) = 20%
    Adopted from GVT [52], for binary color space differences (Section 3.2.3).
  • Top-3 frequent colors = 3
    Number of colors extracted for color comparison, chosen following GVT (Section 3.2.3).
assumptions (6)
  • domain assumption Seven widget types (TextButton, IconButton, CombinedButton, InputBox, TextView, ImageView, Chart) are sufficient to represent mock-up and implementation screens.
    Section 2 defines the screen model; the widget detector is trained only on these classes, so any widget outside this taxonomy is invisible to the tool.
  • domain assumption Screen layout semantics are captured by widget position, width, height, type, and a top-to-bottom then left-to-right partial order.
    Section 3.2.2 defines the partial order; if real layout semantics violate this reading order, the LCS alignment will produce mismatches.
  • domain assumption The widget detection model (YOLO-v8) trained on app screenshots generalizes to design mock-ups.
    Section 3.2.1 unifies widget extraction for both mock-ups and implementation; no mock-up-specific training data is used, and the evaluation only uses screenshots of the same apps.
  • domain assumption The VLM (GPT-4o) with the visual prompt can reliably convert ambiguous natural-language transition descriptions into executable widget-specific actions.
    Section 3.3 depends entirely on this; the authors report 99/100 success within two trials, so roughly 1% of transitions are not executable by the agent.
  • domain assumption Injected mutations (missing, extra, swapped, text, and color changes on 5% of widgets) faithfully represent real inconsistency distributions.
    Section 4.1.1 bases the evaluation on these mutations, citing [52]; if real bugs differ in structure, the measured precision and recall may not transfer.
  • domain assumption The 100 retained design mock-ups are representative of the original 160 after excluding 60 with version updates or blocked interactions.
    Section 4.2.1 discards 60 processes; if the discarded ones are systematically harder or easier, the process inconsistency results are biased.

how reviews work

0 comments
Cite this review

Pith. "Pith review of GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs." pith.science (2026). https://pith.science/paper/FNGSKFEO

@misc{pith2026250607385,
  author       = {Pith},
  title        = {Pith review of: GUIPilot: A Consistency-based Mobile GUI Testing Approach for Detecting Application-specific Bugs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FNGSKFEO}},
  note         = {Machine review of arXiv:2506.07385}
}
read the original abstract

In this work, we propose GUIPilot, an approach for detecting inconsistencies between the mobile design and their implementations. The mobile design usually consists of design mock-ups that specify (1) the expected screen appearances (e.g., widget layouts, colors, and shapes) and (2) the expected screen behaviors, regarding how one screen can transition into another (e.g., labeled widgets with textual description). Given a design mock-up and the implementation of its application, GUIPilot reports both their screen inconsistencies as well as process inconsistencies. On the one hand, GUIPilot detects the screen inconsistencies by abstracting every screen into a widget container where each widget is represented by its position, width, height, and type. By defining the partial order of widgets and the costs of replacing, inserting, and deleting widgets in a screen, we convert the screen-matching problem into an optimizable widget alignment problem. On the other hand, we translate the specified GUI transition into stepwise actions on the mobile screen (e.g., click, long-press, input text on some widgets). To this end, we propose a visual prompt for the vision-language model to infer widget-specific actions on the screen. By this means, we can validate the presence or absence of expected transitions in the implementation. Our extensive experiments on 80 mobile applications and 160 design mock-ups show that (1) GUIPilot can achieve 94.5% precision and 99.6% recall in detecting screen inconsistencies, outperforming the state-of-the-art approach, such as GVT, by 66.2% and 56.6% respectively, and (2) GUIPilot reports zero errors in detecting process inconsistencies. Furthermore, our industrial case study on applying GUIPilot on a trading mobile application shows that GUIPilot has detected nine application bugs, and all the bugs were confirmed by the original application experts.

Figures

Figures reproduced from arXiv: 2506.07385 by the authors.

Figure 1
Figure 1. A design mock-up on the login process of a trading mobile application, consisting of five screens and [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. A failure case of GVT. Red boxes are widgets that could not be matched. Lines highlight the matches. • Accurate Screen Matching: Technically, GVT matches the widgets by relative position in the screen, which can miss important layout semantics [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Overview of the GUIPilot framework. It consists of three main modules: Mock-up compliance checking (Section 3.1), Screen inconsistency checking (Section 3.2), and Process inconsistency checking (Section 3.3). graph p, we identify and output the inconsistent edges: ∃𝑡, (s tar 𝑡 , ACtar 𝑡 , s tar 𝑡+1 ) ∈ p tar , (s𝑡 , AC𝑡 , s𝑡+1) ∈ p such that sim(s𝑡+1, s tar 𝑡+1 ) < 𝜖𝑠𝑐𝑟𝑒𝑒𝑛 (4) 3 Approach Overview [PITH_FULL_IMAGE:f… view at source ↗
Figures from the paper (8 more)
Figure 4
Figure 4. Figure 4: Meta-model for a design mock-up. As shown in [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: Security buying process, taken from a design mock-up of an industrial trading application. The [PITH_FULL_IMAGE:figures/full_fig_p009_5.png]
Figure 6
Figure 6. Figure 6: Process Execution Workflow. extra widget, missing widget, and semantic change. The first two types, i.e., extra and missing widgets, are identified by comparing the matched sets W𝑚 1 and W𝑚 2 with the initial sets W1 and W2. The last type, i.e., semantic change, is com…
Figure 7
Figure 7. Figure 7: Comparison between GVT and GUIPilot. In each figure, the original screen is displayed on the left, and the mutated screen (after insertion, deletion, or swapping) appears on the right. Red boxes indicate extra or missing widgets. Green boxes denote widgets that are una…
Figure 8
Figure 8. Figure 8: False positive examples (FP) and false negative examples (FN) of [PITH_FULL_IMAGE:figures/full_fig_p015_8.png]
Figure 9
Figure 9. Figure 9: Comparison of Object Detection Architectures. The Y-axis represents the mean average precision, the [PITH_FULL_IMAGE:figures/full_fig_p017_9.png]
Figure 10
Figure 10. Figure 10: VLM Action Completion Performance by UI Layout [PITH_FULL_IMAGE:figures/full_fig_p018_10.png]
Figure 11
Figure 11. Figure 11: Real screen inconsistencies reported by the [PITH_FULL_IMAGE:figures/full_fig_p019_11.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

93 extracted references · 60 canonical work pages

  1. [1]

    [n. d.]. Android UiAutomator2. https://github.com/appium/appium-uiautomator2-driver

  2. [2]

    [n. d.]. Anonymous Website: Case Study. https://sites.google.com/view/guipilot/case-study

  3. [3]

    [n. d.]. Anonymous Website: Failure Examples of Widget Detector. https://sites.google.com/view/guipilot/qualitative- analysis-rq3

  4. [4]

    [n. d.]. Anonymous Website for GUIPilot. https://sites.google.com/view/guipilot/home

  5. [5]

    [n. d.]. Anonymous Website: Qualitative Analysis on RQ1. https://sites.google.com/view/guipilot/qualitative-analysis- rq1

  6. [6]

    [n. d.]. Anonymous Website: Simulation Dataset Examples. https://sites.google.com/view/guipilot/dataset

  7. [7]

    [n. d.]. Axure. https://www.axure.com/

  8. [8]

    [n. d.]. Balsamiq. https://balsamiq.com/

Show all 93 references
  1. [9]

    [n. d.]. Code Repository for GUIPilot. https://github.com/code-philia/GUIPilot

  2. [10]

    [n. d.]. mAP (mean Average Precision) for Object Detection. https://jonathan-hui.medium.com/map-mean-average- precision-for-object-detection-45c121a31173

  3. [11]

    [n. d.]. Sketch. https://www.sketch.com/

  4. [12]

    [n. d.]. Waydroid. https://github.com/waydroid/waydroid

  5. [13]

    Khaled Ahmed, Yingying Wang, Mieszko Lis, and Julia Rubin. 2023. ViaLin: Path-Aware Dynamic Taint Analysis for Android. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1598–1610

    2023

  6. [14]

    Ali S Alotaibi, Paul T Chiou, and William GJ Halfond. 2021. Automated repair of size-based inaccessibility issues in mobile applications. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 730–742

    2021

  7. [15]

    Abdulaziz Alshayban and Sam Malek. 2022. AccessiText: automated detection of text accessibility issues in Android apps. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 984–995

    2022

  8. [16]

    Lingfeng Bao, Jing Li, Zhenchang Xing, Xinyu Wang, and Bo Zhou. 2015. scvRipper: video scraping tool for mod- eling developers’ behavior using interaction data. In2015 IEEE/ACM 37th IEEE International Conference on Software Engineering, Vol. 2. IEEE, 673–676

    2015

  9. [17]

    Carlos Bernal-Cárdenas, Nathan Cooper, Madeleine Havranek, Kevin Moran, Oscar Chaparro, Denys Poshyvanyk, and Andrian Marcus. 2022. Translating video recordings of complex mobile app ui gestures into replayable scenarios.IEEE Transactions on Software Engineering49, 4 (2022), 1782–1803

    2022

  10. [18]

    Priyanka Bose, Dipanjan Das, Saastha Vasan, Sebastiano Mariani, Ilya Grishchenko, Andrea Continella, Antonio Bianchi, Christopher Kruegel, and Giovanni Vigna. 2023. Columbus: Android app testing through systematic callback exploration. In2023 IEEE/ACM 45th International Confer...

    2023

  11. [19]

    Shaoheng Cao, Minxue Pan, Yu Pei, Wenhua Yang, Tian Zhang, Linzhang Wang, and Xuandong Li. 2024. Comprehensive Semantic Repair of Obsolete GUI Test Scripts for Mobile Applications. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  12. [20]

    Chunyang Chen, Sidong Feng, Zhenchang Xing, Linda Liu, Shengdong Zhao, and Jinshui Wang. 2019. Gallery dc: Design search and knowledge discovery through auto-created gui component gallery.Proceedings of the ACM on Human-Computer Interaction3, CSCW (2019), 1–22

    2019

  13. [21]

    Jieshan Chen, Mulong Xie, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, and Guoqiang Li. 2020. Object detection for graphical user interface: Old fashioned or deep learning or a combination?. Inproceedings of the 28th ACM joint meeting on European Software Engineering C...

    2020

  14. [22]

    Morgan Dixon and James Fogarty. 2010. Prefab: implementing advanced behaviors using pixel-based reverse en- gineering of interface structure. InProceedings of the SIGCHI Conference on Human Factors in Computing Systems. 1525–1534

    2010

  15. [23]

    Ultralytics YOLO Docs. [n. d.]. Ultralytics YOLOv8. https://docs.ultralytics.com/models/yolov8/

  16. [24]

    Sidong Feng and Chunyang Chen. 2022. Gifdroid: Automated replay of visual bug reports for android apps. In Proceedings of the 44th International Conference on Software Engineering. 1045–1057

    2022

  17. [25]

    Sidong Feng and Chunyang Chen. 2024. Prompting is all you need: Automated android bug replay with large language models. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13

    2024

  18. [26]

    Sidong Feng, Mulong Xie, and Chunyang Chen. 2023. Efficiency matters: Speeding up automated testing with gui rendering inference. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 906–918

    2023

  19. [27]

    Sidong Feng, Mulong Xie, Yinxing Xue, and Chunyang Chen. 2023. Read It, Don’t Watch It: Captioning Bug Recordings Automatically. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2349–2361. Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISS...

    2023

  20. [28]

    Daniel S Hirschberg. 1975. A linear space algorithm for computing maximal common subsequences.Commun. ACM 18, 6 (1975), 341–343

    1975

  21. [29]

    2023.𝜔Test: WebView-Oriented Testing for Android Applica- tions

    Jiajun Hu, Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2023.𝜔Test: WebView-Oriented Testing for Android Applica- tions. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 992–1004

    2023

  22. [30]

    Yongxiang Hu, Xuan Wang, Yingchuan Wang, Yu Zhang, Shiyu Guo, Chaoyi Chen, Xin Wang, and Yangfan Zhou

  23. [31]

    Huaxun Huang, Ming Wen, Lili Wei, Yepang Liu, and Shing-Chi Cheung. 2021. Characterizing and detecting configu- ration compatibility issues in android apps. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 517–528

    2021

  24. [32]

    Huaxun Huang, Chi Xu, Ming Wen, Yepang Liu, and Shing-Chi Cheung. 2023. ConfFix: Repairing Configuration Compatibility Issues in Android Apps. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 514–525

    2023

  25. [33]

    Yuchao Huang, Junjie Wang, Zhe Liu, Song Wang, Chunyang Chen, Mingyang Li, and Qing Wang. 2023. Context-aware bug reproduction for mobile apps. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 2336–2348

    2023

  26. [34]

    Yuchao Huang, Junjie Wang, Zhe Liu, Yawen Wang, Song Wang, Chunyang Chen, Yuanzhe Hu, and Qing Wang. 2024. Crashtranslator: Automatically reproducing mobile application crashes directly from stack trace. InProceedings of the 46th IEEE/ACM International Conference on Software E...

    2024

  27. [35]

    Market IntelliX. [n. d.]. Global GUI Testing Tool Industry Research and Trends Analysis Report. https://www. marketintellix.com/sample-request/global-gui-testing-tool-industry-265845

  28. [36]

    Arun Krishna Vajjala, SM Hasan Mansur, Justin Jose, and Kevin Moran. 2024. MotorEase: Automated detection of motor impairment accessibility issues in mobile app UIs. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  29. [37]

    Nazar Kvartalnyi. [n. d.]. Application Development Life Cycle Explained: From Concept to Launch. https://inoxoft. com/blog/stages-of-app-development/

  30. [38]

    Yuanhong Lan, Yifei Lu, Zhong Li, Minxue Pan, Wenhua Yang, Tian Zhang, and Xuandong Li. 2024. Deeply Reinforcing Android GUI Testing with Deep Reinforcement Learning. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13

    2024

  31. [39]

    Yun Lin, Ruofan Liu, Dinil Mon Divakaran, Jun Yang Ng, Qing Zhou Chan, Yiwen Lu, Yuxuan Si, Fan Zhang, and Jin Song Dong. 2021. Phishpedia: A hybrid deep learning based approach to visually identify phishing webpages. In 30th USENIX Security Symposium (USENIX Security 21). 3793–3810

    2021

  32. [40]

    Yuliang Liu, Zhang Li, Mingxin Huang, Biao Yang, Wenwen Yu, Chunyuan Li, Xu-Cheng Yin, Cheng-Lin Liu, Lianwen Jin, and Xiang Bai. 2024. OCRBench: on the hidden mystery of OCR in large multimodal models.Science China Information Sciences67, 12 (2024), 220102

    2024

  33. [41]

    Zhe Liu, Chunyang Chen, Junjie Wang, Xing Che, Yuekai Huang, Jun Hu, and Qing Wang. 2023. Fill in the blank: Context-aware automated text input generation for mobile gui testing. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 1355–1367

    2023

  34. [42]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Xing Che, Dandan Wang, and Qing Wang. 2024. Make llm a testing expert: Bringing human-like interaction to mobile gui testing via functionality-aware decisions. In Proceedings of the IEEE/ACM 46th International Confer...

    2024

  35. [43]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Yuekai Huang, Jun Hu, and Qing Wang. 2024. Unblind Text Inputs: Predicting Hint-text of Text Input in Mobile Apps via LLM. InProceedings of the CHI Conference on Human Factors in Computing Systems. 1–20

    2024

  36. [44]

    Zhe Liu, Chunyang Chen, Junjie Wang, Mengzhuo Chen, Boyu Wu, Zhilin Tian, Yuekai Huang, Jun Hu, and Qing Wang. 2024. Testing the limits: Unusual text inputs generation for mobile app crash detection with large language model. InProceedings of the IEEE/ACM 46th International Co...

    2024

  37. [45]

    Zhe Liu, Chunyang Chen, Junjie Wang, Yuekai Huang, Jun Hu, and Qing Wang. 2022. Nighthawk: Fully automated localizing ui display issues via visual understanding.IEEE Transactions on Software Engineering49, 1 (2022), 403–418

    2022

  38. [46]

    Zhe Liu, Cheng Li, Chunyang Chen, Junjie Wang, Boyu Wu, Yawen Wang, Jun Hu, and Qing Wang. 2024. Vision-driven Automated Mobile GUI Testing via Multimodal Large Language Model.arXiv preprint arXiv:2407.03037(2024)

    2024 arXiv

  39. [47]

    Enze Ma, Shan Huang, Weigang He, Ting Su, Jue Wang, Huiyu Liu, Geguang Pu, and Zhendong Su. 2023. Automata- Based Trace Analysis for Aiding Diagnosing GUI Testing Tools for Android. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on t...

    2023

  40. [48]

    Junayed Mahmud, Nadeeshan De Silva, Safwat Ali Khan, Seyed Hooman Mostafavi, SM Hasan Mansur, Oscar Chaparro, Andrian Marcus, and Kevin Moran. 2024. On Using GUI Interaction Data to Improve Text Retrieval-based Bug Localization. InProceedings of the 46th IEEE/ACM International...

    2024

  41. [49]

    Leonardo Mariani, Ali Mohebbi, Mauro Pezzè, and Valerio Terragni. 2021. Semantic matching of gui events for test reuse: are we there yet?. InProceedings of the 30th ACM SIGSOFT International Symposium on Software Testing and Analysis. 177–190

    2021

  42. [50]

    Nariman Mirzaei, Joshua Garcia, Hamid Bagheri, Alireza Sadeghi, and Sam Malek. 2016. Reducing combinatorics in GUI testing of android applications. InProceedings of the 38th international conference on software engineering. 559–570

    2016

  43. [51]

    Kevin Moran, Carlos Bernal-Cárdenas, Michael Curcio, Richard Bonett, and Denys Poshyvanyk. 2018. Machine learning-based prototyping of graphical user interfaces for mobile apps.IEEE Transactions on Software Engineering46, 2 (2018), 196–221

    2018

  44. [52]

    Kevin Moran, Boyang Li, Carlos Bernal-Cárdenas, Dan Jelf, and Denys Poshyvanyk. 2018. Automated reporting of GUI design violations for mobile apps. InProceedings of the 40th International Conference on Software Engineering. 165–175

    2018

  45. [53]

    Kevin Moran, Mario Linares-Vasquez, Carlos Bernal-Cardenas, Christopher Vendome, and Denys Poshyvanyk. 2016. Automatically Discovering, Reporting and Reproducing Android Application Crashes . In2016 IEEE International Conference on Software Testing, Verification and Validation...

    2016 doi

  46. [54]

    Tuan Anh Nguyen and Christoph Csallner. 2015. Reverse engineering mobile application user interfaces with remaui (t). In2015 30th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 248–259

    2015

  47. [55]

    Ju Qian, Yingwei Ma, Chenghao Lin, and Lin Chen. 2022. Accelerating OCR-based widget localization for test automation of GUI applications. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–13

    2022

  48. [56]

    Ju Qian, Zhengyu Shang, Shuoyan Yan, Yan Wang, and Lin Chen. 2020. Roscript: a visual script driven truly non- intrusive robotic testing system for touch screen applications. InProceedings of the ACM/IEEE 42nd International Conference on Software Engineering. 297–308

    2020

  49. [57]

    Dezhi Ran, Hao Wang, Wenyu Wang, and Tao Xie. 2023. Badge: prioritizing UI events with hierarchical multi-armed bandits for automated UI testing. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 894–905

    2023

  50. [58]

    Dillon Reis, Jordan Kupec, Jacqueline Hong, and Ahmad Daoudi. 2023. Real-time flying object detection with YOLOv8. arXiv preprint arXiv:2305.09972(2023)

    2023 arXiv

  51. [59]

    Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. 2016. Faster R-CNN: Towards real-time object detection with region proposal networks.IEEE transactions on pattern analysis and machine intelligence39, 6 (2016), 1137–1149

    2016

  52. [60]

    Grand View Research. [n. d.]. Mobile Application Market Size & Trends. https://www.grandviewresearch.com/industry- analysis/mobile-application-market

  53. [61]

    T-YLPG Ross and GKHP Dollár. 2017. Focal loss for dense object detection. Inproceedings of the IEEE conference on computer vision and pattern recognition. 2980–2988

    2017

  54. [62]

    Jonathan A Saddler and Myra B Cohen. 2017. EventFlowSlicer: a tool for generating realistic goal-driven GUI tests.. In ASE. 955–960

    2017

  55. [63]

    Wei Song, Xiangxing Qian, and Jeff Huang. 2017. EHBDroid: Beyond GUI testing for Android applications. In2017 32nd IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 27–37

    2017

  56. [64]

    Yiren Song, Danze Chen, and Mike Zheng Shou. 2025. LayerTracer: Cognitive-Aligned Layered SVG Synthesis via Diffusion Transformer.arXiv preprint arXiv:2502.01105(2025)

    2025 arXiv

  57. [65]

    Ting Su, Guozhu Meng, Yuting Chen, Ke Wu, Weiming Yang, Yao Yao, Geguang Pu, Yang Liu, and Zhendong Su

  58. [66]

    Ting Su, Yichen Yan, Jue Wang, Jingling Sun, Yiheng Xiong, Geguang Pu, Ke Wang, and Zhendong Su. 2021. Fully automated functional fuzzing of Android apps for detecting non-crashing logic bugs.Proceedings of the ACM on Programming Languages5, OOPSLA (2021), 1–31

    2021

  59. [67]

    Yuhui Su, Chunyang Chen, Junjie Wang, Zhe Liu, Dandan Wang, Shoubin Li, and Qing Wang. 2022. The metamorphosis: Automatic detection of scaling issues for mobile apps. InProceedings of the 37th IEEE/ACM International Conference on Automated Software Engineering. 1–12

    2022

  60. [68]

    Jingling Sun, Ting Su, Jiayi Jiang, Jue Wang, Geguang Pu, and Zhendong Su. 2023. Property-Based Fuzzing for Finding Data Manipulation Errors in Android Apps. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations of Softwa...

    2023

  61. [69]

    Saghar Talebipour, Yixue Zhao, Luka Dojcilović, Chenggang Li, and Nenad Medvidović. 2021. Ui test migration across mobile platforms. In2021 36th IEEE/ACM International Conference on Automated Software Engineering (ASE). IEEE, 756–767

    2021

  62. [70]

    UX Tools. 2022. Basic Prototyping. https://uxtools.co/survey/2022/basic-prototyping

    2022

  63. [71]

    Wagner and Michael J

    Robert A. Wagner and Michael J. Fischer. 1974. The string-to-string correction problem.Journal of the ACM (JACM) 21, 1 (1974), 168–173. Proc. ACM Softw. Eng., Vol. 2, No. ISSTA, Article ISSTA034. Publication date: July 2025. ISSTA034:24 Ruofan Liu, Xiwen Teoh, Yun Lin, Guanjie...

    1974

  64. [72]

    Ao Wang, Hui Chen, Lihao Liu, Kai Chen, Zijia Lin, Jungong Han, and Guiguang Ding. 2024. Yolov10: Real-time end-to-end object detection.arXiv preprint arXiv:2405.14458(2024)

    2024 arXiv

  65. [73]

    Chien-Yao Wang, I-Hau Yeh, and Hong-Yuan Mark Liao. 2024. Yolov9: Learning what you want to learn using programmable gradient information. InEuropean conference on computer vision. Springer, 1–21

    2024

  66. [74]

    Jue Wang, Yanyan Jiang, Ting Su, Shaohua Li, Chang Xu, Jian Lu, and Zhendong Su. 2022. Detecting non-crashing functional bugs in Android apps via deep-state differential analysis. InProceedings of the 30th ACM Joint European Software Engineering Conference and Symposium on the...

    2022

  67. [75]

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. 2024. Autodroid: Llm-powered task automation in android. InProceedings of the 30th Annual International Conference on Mobile Computing and Networking...

    2024

  68. [76]

    Mulong Xie, Sidong Feng, Zhenchang Xing, Jieshan Chen, and Chunyang Chen. 2020. UIED: a hybrid tool for GUI element detection. InProceedings of the 28th ACM Joint Meeting on European Software Engineering Conference and Symposium on the Foundations of Software Engineering. 1655–1659

    2020

  69. [77]

    Yiheng Xiong, Mengqian Xu, Ting Su, Jingling Sun, Jue Wang, He Wen, Geguang Pu, Jifeng He, and Zhendong Su

  70. [78]

    Yanfu Yan, Nathan Cooper, Oscar Chaparro, Kevin Moran, and Denys Poshyvanyk. 2024. Semantic GUI Scene Learning and Video Alignment for Detecting Duplicate Video-based Bug Reports. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  71. [79]

    Bo Yang, Zhenchang Xing, Xin Xia, Chunyang Chen, Deheng Ye, and Shanping Li. 2021. Don’t do that! hunting down visual design smells in complex uis against design guidelines. In2021 IEEE/ACM 43rd international conference on software engineering (ICSE). IEEE, 761–772

    2021

  72. [80]

    Sen Yang, Sen Chen, Lingling Fan, Sihan Xu, Zhanwei Hui, and Song Huang. 2023. Compatibility issue detection for Android apps based on path-sensitive semantic analysis. In2023 IEEE/ACM 45th International Conference on Software Engineering (ICSE). IEEE, 257–269

    2023

  73. [81]

    Jiaming Ye, Ke Chen, Xiaofei Xie, Lei Ma, Ruochen Huang, Yingfeng Chen, Yinxing Xue, and Jianjun Zhao. 2021. An empirical study of GUI widget detection for industrial mobile games. InProceedings of the 29th ACM Joint Meeting on European Software Engineering Conference and Symp...

    2021

  74. [82]

    Tom Yeh, Tsung-Hsiang Chang, and Robert C Miller. 2009. Sikuli: using GUI screenshots for search and automation. In Proceedings of the 22nd annual ACM symposium on User interface software and technology. 183–192

    2009

  75. [83]

    Shengcheng Yu, Chunrong Fang, Mingzhe Du, Yuchen Ling, Zhenyu Chen, and Zhendong Su. 2024. Practical Non- Intrusive GUI Exploration Testing with Visual-based Robotic Arms. InProceedings of the IEEE/ACM 46th International Conference on Software Engineering. 1–13

    2024

  76. [84]

    Yuxin Zhang, Sen Chen, Lingling Fan, Chunyang Chen, and Xiaohong Li. 2023. Automated and Context-Aware Repair of Color-Related Accessibility Issues for Android Apps. InProceedings of the 31st ACM Joint European Software Engineering Conference and Symposium on the Foundations o...

    2023

  77. [85]

    Yakun Zhang, Wenjie Zhang, Dezhi Ran, Qihao Zhu, Chengfeng Dou, Dan Hao, Tao Xie, and Lu Zhang. 2024. Learning- based Widget Matching for Migrating GUI Test Cases. InProceedings of the 46th IEEE/ACM International Conference on Software Engineering. 1–13

    2024

  78. [86]

    Dehai Zhao, Zhenchang Xing, Chunyang Chen, Xiwei Xu, Liming Zhu, Guoqiang Li, and Jinshui Wang. 2020. Seenomaly: Vision-based linting of gui animation effects against design-don’t guidelines. InProceedings of the ACM/IEEE 42nd international conference on software engineering. ...

    2020

  79. [87]

    Yanjie Zhao, Li Li, Kui Liu, and John Grundy. 2022. Towards automatically repairing compatibility issues in published Android apps. InProceedings of the 44th International Conference on Software Engineering. 2142–2153

    2022

  80. [88]

    Yu Zhao, Ting Su, Yang Liu, Wei Zheng, Xiaoxue Wu, Ramakanth Kavuluru, William GJ Halfond, and Tingting Yu

  81. [89]

    Yixue Zhao, Saghar Talebipour, Kesina Baral, Hyojae Park, Leon Yee, Safwat Ali Khan, Yuriy Brun, Nenad Medvidović, and Kevin Moran. 2022. Avgust: automating usage-based test generation from videos of app executions. InProceedings of the 30th ACM Joint European Software Enginee...

    2022

  82. [2017]

    InProceedings of the 2017 11th joint meeting on foundations of software engineering

    Guided, stochastic model-based GUI testing of Android apps. InProceedings of the 2017 11th joint meeting on foundations of software engineering. 245–256

    2017

  83. [2022]

    Recdroid+: Automated end-to-end crash reproduction from bug reports for android apps.ACM Transactions on Software Engineering and Methodology (TOSEM)31, 3 (2022), 1–33

    2022

  84. [2023]

    InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis

    An empirical study of functional bugs in android apps. InProceedings of the 32nd ACM SIGSOFT International Symposium on Software Testing and Analysis. 1319–1331

  85. [2024]

    AUITestAgent: Automatic Requirements Oriented GUI Function Testing.arXiv preprint arXiv:2407.09018(2024)

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.