{"id":"f9a69549-a52d-4647-935a-0b827bf97c03","arxiv_id":"2506.07385","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"GUIPilot compares mobile app screens and workflows against design mock-ups, using widget sequence alignment and a vision-language model to detect layout and transition inconsistencies.","lead":"This paper presents GUIPilot, a tool that checks whether a mobile app's screens and navigation flows match its design mock-ups, catching layout, text, color, and transition errors. It reports high accuracy on 80 apps and found nine confirmed bugs in an industrial trading app.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Simulated mock-ups are mutated app screenshots rather than real design mock-ups, so the headline screen-consistency numbers may not transfer to the target deployment domain.","rationale":"The reader's weakest_assumption identifies exactly the same external-validity concern: simulated mock-ups are mutated app screenshots, and the paper itself concedes that real industrial mock-ups needed manual corrections. My stress-test pass confirms that all quantitative claims in the abstract (94.5% precision, 99.6% recall, 66.2%/56.6% improvement over GVT, zero process-inconsistency errors) derive from experiments built on these simulated mock-ups. The paper does not overclaim internal consistency: Algorithm 1 is clearly specified, the LCS formulation is sound, and the runtime claims are plausible. The RQ3-1 widget detection numbers (mAP 0.515 overall, 0.317 for charts) are honestly reported and actually strengthen the external-validity concern: the detector is the first stage for both mock-up and implementation screens, and its lowest-performing classes are exactly the ones (charts, input boxes) where prototype-tool mock-ups tend to differ most from rendered apps. The paper's internal arithmetic inconsistency (abstract says 94.5%/99.6%, Table 3 implies roughly 99.6%/99.2% overall) and the threshold-tuning-on-test-data issue for process inconsistency are real but secondary; the process-inconsistency 100% is additionally based on the same 100 transitions used for VLM evaluation, and the paper reports 99/100 VLM success, so the zero-error claim is fragile. The industrial case study with nine confirmed bugs is independent evidence that the pipeline can work end-to-end on real mock-ups, but it is a small qualitative sample (19 mock-ups, one app) and cannot support the precision/recall magnitudes in the abstract. Therefore the reader's CONDITIONAL verdict is appropriate, and my concern does not move the verdict; it sharpens the condition: the quantitative screen-consistency claims should be re-estimated on real industrial mock-ups or explicitly scoped to screenshots that match the detector's training distribution.","tokens_in":22582,"tokens_out":1865,"duration_ms":19050,"concrete_test":"Re-run RQ1 (Section 4.1) with the 19 industrial mock-ups from the case study (Section 5) as the mock-up side, using the same 80-app implementation screens and the same GUIPilot pipeline, and report precision/recall against expert-labeled ground truth. If the industrial mock-ups cannot be processed without manual corrections, report the number and nature of the corrections and recompute the metrics on the automatically-handled subset. A drop in precision/recall below the abstract's 94.5%/99.6% would confirm that the simulation-based evaluation overstates transfer to real mock-ups.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim—94.5% precision and 99.6% recall for screen inconsistency detection—rests entirely on Section 4.1.1, where 'simulated mock-ups' are generated by mutating the apps' own screenshots. The paper's own Threat to Validity (Section 5.1) admits that real mock-ups were unavailable and that 'any shortcomings can be manually corrected.' Meanwhile the industrial case study found that real mock-ups required manual navigation information (Section 5.1, 'Call for Good Practices'), and the industrial evaluation is qualitative with no precision/recall reported. The widget detector (Table 4) has class-wise mAP as low as 0.317 for charts and 0.446 for input boxes, and the widget-alignment similarity is computed on coordinates, size, aspect ratio, and type from that detector. If real mock-ups differ in rendering style, widget typography, chart graphics, or layout conventions (as prototype tools such as Sketch produce vector-style mock-ups rather than rendered Android screens), the trained detector and the coordinate-based similarity are the components most likely to degrade. Because the reported screen-consistency numbers are computed on mutated screenshots whose widget types, text, and colors come from the same distribution as the training data, they are not evidence of performance on real mock-ups. This is not an internal inconsistency in Algorithm 1; it is an external-validity gap at the exact point where the quantitative claim lives.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"GUIPilot proposes an end-to-end approach for detecting inconsistencies between mobile design mock-ups and app implementations. Screen inconsistencies are detected by detecting widgets on both screens, ordering them partially, and solving an LCS-based widget alignment problem; process inconsistencies are detected by translating described screen transitions into executable actions via a visual prompt to a VLM, then comparing the reached screen with the target screen using a similarity threshold. The paper evaluates the approach on 80 apps with 160 simulated mock-ups, reports very high precision and recall for both screen and process inconsistency detection, and presents a case study on an industrial trading app in which nine inconsistencies were confirmed by experts.","tokens_in":22885,"tokens_out":4916,"duration_ms":59738,"significance":"If the reported results held, GUIPilot would fill a real industrial gap: automated validation of GUI implementations against design mock-ups, going beyond crash-oriented GUI testing. The widget-alignment formulation and the VLM-based action-completion prompt are sensible technical contributions, and the industrial case study with expert-confirmed bugs is a valuable piece of evidence. The availability of the code and dataset artifacts is also a strength. However, the central quantitative claims currently rest on internally inconsistent headline numbers and on evaluation setups in which key thresholds were selected on the same data used to report performance, so the empirical contribution needs substantial reworking before the stated significance can be accepted.","major_comments":[{"comment":"The headline screen-consistency numbers are mutually inconsistent. The full-text abstract reports 99.8% precision and 98.6% recall, the Introduction reports 94.5% precision and 99.6% recall, and Table 3 reports per-mutation-type precision values of 0.987, 0.997, 0.998, 0.996, and 1.000. Since a weighted average of values all at least 0.987 cannot equal 0.945, the 94.5% figure is arithmetically impossible if it is an aggregate of Table 3. Please report one consistent set of numbers and explicitly state how the per-type results are aggregated into the headline precision and recall.","section":"Abstract / Section 1 / Table 3"},{"comment":"The process-inconsistency result is circular. The configuration paragraph states that the screen matching threshold epsilon_screen is selected using the 100 mutated design mock-ups introduced in the process consistency experiment, and Section 4.2.2 then reports 100% precision and 100% recall on those same 100 mutated processes. This means the reported zero-error result is a fitted value rather than an evaluation of generalization. Please evaluate the threshold on a held-out set or use cross-validation, or clearly report the result as development-set performance rather than as the achieved precision and recall.","section":"Section 4 (Configuration) / Section 4.2"},{"comment":"The same selection-on-evaluation-data issue applies to the screen-consistency experiment. The hyperparameters alpha and delta in Algorithm 1 are selected by grid search using the screen consistency experimental performance described in Section 4.1, and no separate validation or test split is described for that experiment. Consequently, the RQ1 precision, recall, and Jaccard indices may be over-optimistic estimates of performance on new screens. Please clarify the data split used for hyperparameter selection, or re-run the experiment with proper separation between parameter tuning and evaluation.","section":"Section 4 (Configuration) / Section 4.1"},{"comment":"The simulated mock-ups are not design mock-ups. In Section 4.1.1, the 'simulated mock-ups' are real application screenshots labeled by experts and then mutated, and the threats-to-validity discussion in Section 5.1 explicitly acknowledges that actual design mock-ups were unavailable. The industrial case study, which did use real mock-ups, required manual navigation information and reports no precision or recall. As a result, the quantitative screen-consistency claims do not yet establish performance on the target input type. Please either add an evaluation on real design mock-ups or explicitly reframe the headline claims as applying to simulated mock-ups, while treating the case study as qualitative evidence only.","section":"Section 4.1.1 / Section 5.1"}],"minor_comments":[{"comment":"The sentence beginning 'which overall achieves a satisfactory detection rate ( As a reference...' has a malformed parenthesis and the comparison to the COCO benchmark is not directly relevant to GUI widget detection; please rephrase or remove it.","section":"Section 4.3.2"},{"comment":"The caption and axes contain the typo 'Trail' instead of 'Trial' in '1 Trail Success Rate' and '2 Trail Success Rate'.","section":"Figure 10"},{"comment":"The text says 'following the workflow in Figure 3.3'; this should refer to Figure 3 or Section 3.3.","section":"Section 4.2.2"},{"comment":"In the external validity discussion, the sentence 'Any shortcomings can be manually corrected' is vague; please specify which shortcomings require manual correction and how that affects the automated pipeline's claims.","section":"Section 5.1"},{"comment":"The abstract says the experiments use 160 design mock-ups, but Section 4.2.1 states that only 100 design mock-ups were retained for the process consistency experiment; please clarify whether the 160 figure refers to the full collected set and state clearly which experiments use which subset.","section":"Section 4.2.1 / Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a real problem and contains interesting components, but the published version cannot contain contradictory headline precision/recall numbers. The process-consistency 100% claim is currently an artifact of threshold selection on the same data, and the screen-consistency hyperparameter selection appears to have the same issue. I would ask the authors to re-run the evaluations with proper data separation and to either obtain real mock-ups or substantially weaken the external claims. If those points are addressed, the paper could be acceptable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Core idea is sound, but the evaluation has three fixable problems. What's actually new: GUIPilot does global widget alignment via LCS, which genuinely improves on GVT's nearest-neighbor matching for layout shifts, and it uses a VLM visual prompt to turn mock-up transition descriptions into executable actions. That's an end-to-end mock-up validation tool, which is a real industrial need. Credit where due: code and data are released, and the industrial case study found nine bugs confirmed by domain experts. That is real evidence the approach works in practice.\n\nThe soft spots are all in the evaluation. First, the headline numbers disagree: the abstract states 94.5% precision and 99.6% recall, the introduction states 99.8% and 98.6%, and the per-type precision in Table 3 (all above 0.987) makes 94.5% arithmetically impossible. This has to be corrected. Second, the 100% precision and recall for process inconsistency is obtained with the screen-matching threshold epsilon_screen selected on the same 100 mutated mock-ups used for testing. That turns the perfect score into a fitted value. Third, the simulated mock-ups are the apps' own screenshots with mutations injected; they are not real design mock-ups. The paper acknowledges this in the threats to validity, and the stress-test note is fair. The industrial case study partially offsets the concern—it is qualitative, and real mock-ups needed manual navigation info—but it shows the tool can work on realistic artifacts.\n\nThe widget detector's low mAP on charts (0.317) and input boxes (0.446) is worth a mention, but the paper argues class confusion is symmetric across the two screens, which is plausible. Overall, the approach is well motivated, the writing is clear, and the authors are upfront about limitations. The fixes are mechanical: correct the numbers, use a validation set for threshold selection, and tone down what the simulated-mock-up results can claim.\n\nThis paper deserves peer review. The core contribution passes a first read, and the evaluation issues are repairable. Send it out, but condition acceptance on the number audit and a proper train/validation split.","headline":"GUIPilot is a genuinely useful approach, but its evaluation needs a number audit and a proper threshold split before the claims hold.","tokens_in":23442,"tokens_out":3881,"would_cite":true,"duration_ms":42642,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"GUIPilot reports 94.5% precision and 99.6% recall when checking mobile app screens against design mock-ups.","keywords":["GUI testing","design mock-ups","screen inconsistency","process inconsistency","widget alignment","vision-language model","mobile applications","Android"],"falsifier":"Give GUIPilot a held-out set of authentic designer-created mock-ups that were not produced by mutating the apps' own screenshots, label the real inconsistencies by hand, and compare precision and recall; if they fall far below 94.5% and 99.6%, the simulated-mock-up evaluation overstated the tool's transfer.","tokens_in":22380,"feed_emoji":"📱","tokens_out":8093,"duration_ms":82535,"temperature":0.7,"pith_summary":"This paper tries to establish that a mobile app's implementation can be checked automatically against the designer's mock-ups, catching the application-specific bugs that crash-oriented testing misses. GUIPilot detects two kinds of drift: screen inconsistencies, meaning missing, extra, or semantically changed widgets, and process inconsistencies, meaning transitions specified in the mock-up that do not happen or land on the wrong screen. On a benchmark of 80 Android apps and 160 simulated mock-ups it reports 94.5% precision and 99.6% recall for screen inconsistencies, beating the prior GVT approach by 66.2% and 56.6%, and reports zero errors in detecting process inconsistencies on the 100 retained transitions. In an industrial case study on a trading app with 19 usable design mock-ups, it found nine inconsistencies that the app's own experts confirmed.","feed_headline":"GUI checker finds app bugs by matching screens to design mock-ups","feed_subtitle":"GUIPilot also validates screen transitions, and found nine confirmed bugs in a real trading app.","key_machinery":"The load-bearing object is the widget-alignment formulation built on the longest common subsequence (LCS). A screen is abstracted into a sequence of widgets sorted by a (y, x) partial order; pairwise similarity is the product of position distance, area ratio, aspect-ratio ratio, and a type-match score; dynamic programming then finds the global maximum-similarity common subsequence, tolerating insertions and deletions. This turns screen matching into an optimization problem instead of a nearest-neighbor search. The second mechanism is the visual prompt for the vision-language model: annotated bounding-box IDs over interactable widgets, a closed action space of seven commands (click, long press, send keys, scroll, swipe, drag and drop, go back), a few-shot example, and iterative feedback when an action produces no transition. Together these let GUIPilot compare screens globally and execute transitions without manual scripting.","core_discovery":"GUIPilot's central claim is that both kinds of design-implementation drift can be reduced to well-defined matching problems. Each screen, whether from a mock-up or from the running app, is converted by a trained object detector into a set of widgets carrying position, size, and type; screen comparison then becomes an optimal alignment of two sequences sorted top-to-bottom and left-to-right, solved as a longest-common-subsequence dynamic program whose similarity score multiplies position, area, aspect-ratio, and type agreement. This global alignment is what fixes the failure mode of the prior GVT approach, in which local nearest-neighbor matching mispairs widgets when a row is inserted or deleted. For transitions, a vision-language model receives the mock-up's natural-language description plus a screenshot with interactable widgets highlighted and numbered, and must emit actions from a fixed action space; executing those actions on the app and comparing the resulting screen with the expected mock-up screen detects missing or wrong transitions. The paper reports that the pipeline finds screen inconsistencies at 94.5% precision and 99.6% recall, catches all 100 mutated process inconsistencies with no errors, and in the industrial case study found nine expert-confirmed bugs.","pith_inferences":["Beyond the paper, the same screenshot-to-widget-sequence abstraction could apply to web or desktop UIs, since it relies on bounding boxes and widget classes rather than on Android internals.","Beyond the paper, a public dataset of genuine designer-produced mock-ups paired with the corresponding app screens would give a harder transfer test than the mutation-based mock-ups used in the benchmark.","Beyond the paper, the action-completion protocol could be reused as an automatic regression-test generator: once a transition description is turned into executable actions, those actions remain useful even when no inconsistency is found.","Beyond the paper, moving from an external vision-language API to a locally hosted model would remove network latency and cost as operational dependencies, making continuous-integration deployment more practical."],"forward_implications":["Screen matching no longer breaks when a row of widgets is inserted or deleted, because alignment is global rather than local nearest-neighbor matching.","Design-mock-up validation becomes end-to-end automatable: given mock-ups in the paper's meta-model format, GUIPilot can drive the app through the specified flow and compare every reached screen.","The pipeline reports violation type (extra, missing, semantic change) rather than only a difference, so testers can file targeted bug reports.","The reported per-screen matching time of about 0.001 seconds and transition time of about 0.19 seconds make consistency checks cheap enough to run on every build.","If the industrial case study generalizes, the tool can surface subtle defects, such as a missing small widget or a replaced label, that manual inspection of a real trading app missed."],"supporting_citations":[{"why":"The GVT approach that serves as the baseline; supplies the widget-similarity metric, the mutation recipe for injecting inconsistencies, and the taxonomy GUIPilot extends.","marker":"[52]"},{"why":"The YOLOv8 object detection architecture trained as the widget detector that converts both mock-up and implementation screens into typed bounding boxes.","marker":"[58]"},{"why":"The longest-common-subsequence algorithm that the global widget alignment is formulated as.","marker":"[28]"},{"why":"The dynamic programming approach to string-to-string correction that underlies the LCS-based matching optimization.","marker":"[71]"},{"why":"The UiAutomator2 driver used to execute the VLM-inferred action chains on the running app during process inconsistency checks.","marker":"[1]"},{"why":"The Waydroid virtual-device environment used to run the reproducible process consistency experiments.","marker":"[12]"}],"fun_headline_variants":["Screen-to-mock-up matching exposes app bugs","GUIPilot finds app bugs via GUI consistency checks","Design vs implementation: GUIPilot catches mismatches","App testing tool spots screen and transition flaws","GUIPilot: nine confirmed bugs in a trading app"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation treats mutated screenshots of the very apps being tested as stand-ins for real designer mock-ups, so the reported accuracy may not transfer to genuine design files.","fun_headline_variants_meta":{"raw":{"variants":["Screen-to-mock-up matching exposes app bugs","GUIPilot finds app bugs via GUI consistency checks","Design vs implementation: GUIPilot catches mismatches","App testing tool spots screen and transition flaws","GUIPilot: nine confirmed bugs in a trading app"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1867,"prompt_tokens":1138,"completion_tokens":729,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":754,"completion_tokens_details":{"reasoning_tokens":655}},"tokens_in":754,"tokens_out":729,"duration_ms":8820,"temperature":1.0,"reasoning_tokens":655,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:35:51.857738+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Give GUIPilot a held-out set of authentic designer-created mock-ups that were not produced by mutating the apps' own screenshots, label the real inconsistencies by hand, and compare precision and recall; if they fall far below 94.5% and 99.6%, the simulated-mock-up evaluation overstated the tool's transfer.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The GVT approach that serves as the baseline; supplies the widget-similarity metric, the mutation recipe for injecting inconsistencies, and the taxonomy GUIPilot extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The longest-common-subsequence algorithm that the global widget alignment is formulated as."},{"cited_title":"Wagner and Michael J","cited_arxiv_id":null,"evidence_quote":"The dynamic programming approach to string-to-string correction that underlies the LCS-based matching optimization."}],"review_version":1}