{"id":"700f6bdc-0a55-4c69-b034-bcc36db6a95c","arxiv_id":"2508.01337","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"MobileGUIPerf detects on-screen touch indicators and frame-level visual changes in screencasts to measure GUI response and finish times, reaching 0.96 precision, 0.93 recall, and 50 ms and 100 ms timing accuracy on 2,458 annotated interactions.","lead":"This paper presents MobileGUIPerf, a tool that measures how quickly Android apps visually respond to taps and swipes by analyzing screen recordings. It is designed for large-scale industrial app testing, where it can flag user-perceived delays that system-level performance tools miss.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Response-time accuracy hinges on an undisclosed 'offset' in keyframe detection, while the paper's definition drifts from 'first visual change' to 'substantial transition'; the headline timing numbers may be in-sample-tuned to the authors' own annotations.","rationale":"This is a solid experience report: the released 2,458-interaction benchmark is a real contribution, the use of the V2S dataset is reasonable, and the deployment narrative is plausible. The reader's weakest assumption (dependence on the Android Show taps overlay) is a genuine scope boundary, but it is stated honestly and matches the deployment setting; it limits generality rather than threatening the measured numbers. The undisclosed 'offset' is more significant because it sits inside the measured quantity. Section IV.C is explicit that an offset is introduced, but the paper never defines it; Section V provides no cross-validation or sensitivity analysis; and the ground truth was produced by the same three authors who designed the tool, with disagreements resolved by consensus. The definitional drift from 'first visible change' to 'substantial transition' further blurs what is being measured. If the offset was chosen to match the authors' annotation consensus, the 95%/89% accuracy figures would be expected by construction. Therefore, the verdict should remain CONDITIONAL, with disclosure of the offset and a sensitivity analysis as necessary conditions for accepting the timing claim. The interaction-detection precision/recall results are somewhat less affected, but they should be re-run with the released code.","tokens_in":15938,"tokens_out":5694,"duration_ms":79401,"concrete_test":"Release the exact offset parameter used in Section IV.C and re-run the RQ2 evaluation (Tables III and IV) with the offset shifted by -2, -1, +1, and +2 frames, holding all other steps fixed. If the fraction of interactions within 3 frames for response time (95%) or within 6 frames for finish time (89%) changes by more than about 5 percentage points across this range, the reported timing accuracy is not robust to the undisclosed parameter.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The timing claim (response/finish errors within 50/100 ms for 89-95% of interactions) is the quantitative core of the paper. Algorithm 1 Step 4 locates the response frame by applying SSIM and Isolation Forest to a frame-difference sequence, but Section IV.C states that 'we introduce an offset in the Isolation Forest algorithm to refine the position of the response frame.' The paper gives no value, formula, or selection procedure for this offset, and says it encodes the belief that users care about 'substantial visual transitions' rather than 'minor effects such as a button dimming.' This conflicts with the paper's own definition of response time as the time to the 'first visible GUI frame update' (Table I). If the three annotators who built the ground truth applied the same substantial-transition intuition, then high agreement between the tool and the ground truth measures agreement with a shared heuristic, not accuracy against the stated definition. If the offset was tuned on the same 2,458 interactions used for evaluation, then the 95%-within-3-frames and 89%-within-6-frames results are at least partly in-sample. The paper reports no inter-annotator reliability, so we cannot separate genuine measurement accuracy from annotation consensus. This is load-bearing because the central claim is precisely that MobileGUIPerf can replace manual frame-by-frame inspection.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"MobileGUIPerf is a black-box pipeline that uses Android's Show taps overlay to segment screencasts into user interactions and uses SSIM frame differences with Isolation Forest to locate response and finish frames. The paper evaluates interaction detection (0.96 precision, 0.93 recall) and timing accuracy (95% of response times within 3 frames, 89% of finish times within 6 frames) on 2,458 manually annotated interactions drawn from the V2S dataset, and reports successful deployment in an industrial testing pipeline. The central claim is that screencasts alone can provide user-perceived responsiveness measurements at scale, without access to source code or system instrumentation.","tokens_in":16224,"tokens_out":6223,"duration_ms":80862,"significance":"The paper addresses a real gap: system-level metrics miss user-perceived delays, and frame-by-frame manual inspection does not scale. If the headline numbers are credible, MobileGUIPerf is a useful practical contribution, and the released annotated dataset is a valuable resource for future research. I credit the public benchmark, the black-box design, the explicit RQ structure, and the evidence of real deployment. However, the quantitative core is not yet fully supported: the response-frame definition drifts from 'first visible change' to 'substantial transition' through an undisclosed offset, several parameters are selected on the same benchmark used for evaluation, and the timing numbers are computed only on already-detected interactions. These issues are load-bearing for the paper's claim that MobileGUIPerf can replace manual frame-by-frame inspection, but they are fixable with additional analysis, so I recommend major revision rather than rejection.","major_comments":[{"comment":"The operational definition used to locate the response frame is not aligned with the stated definition of response time. Table I defines response time as the duration to the 'first visible GUI frame update', but Section IV.C says that an 'offset' is introduced in the Isolation Forest algorithm to reflect that users associate responsiveness with 'substantial visual transitions' rather than 'minor effects such as a button dimming'. The paper never states the value, formula, or selection procedure for this offset. If the annotators used the same substantial-transition intuition when labeling the ground truth, then the reported agreement measures consistency with that heuristic, not accuracy against the stated definition. If the offset was tuned on the same 2,458 interactions, the RQ2 numbers are partly in-sample. Please disclose the offset and its selection procedure, report timing accuracy with and without it, and either align Table I with the operational criterion or justify the deviation.","section":"Section IV.C and Table I"},{"comment":"The timing-accuracy results in RQ2 are computed only for 'successfully identified interactions'. The abstract's headline claim ('for over 89% of interactions') therefore excludes interactions that are missed by the detector (7% recall loss in RQ1) or affected by segmentation errors, which the paper itself identifies as a cause of finish-time errors larger than six frames. RQ4 provides an end-to-end threshold classification, but it does not report an end-to-end timing-error distribution. Please report timing accuracy over the full set of ground-truth interactions, treating missed interactions as errors or separately reporting detection-conditioned and unconditional error rates.","section":"Section V, RQ2 (Tables III and IV)"},{"comment":"The ground truth is created by three annotators who independently label frames and then reach consensus, but no inter-annotator agreement metric (e.g., Cohen's or Fleiss' kappa, or per-frame agreement) is reported. Without this, it is difficult to separate measurement accuracy from shared annotator interpretation of when a 'response' starts. Please report initial agreement before the consensus discussion, and ideally validate a sample of annotations with an independent annotator who did not build the tool.","section":"Section V.A"},{"comment":"Several parameters appear to be selected on the same benchmark that is later used for evaluation: the 10-pixel tap/swipe threshold is 'empirically selected' against a subset of annotated interactions, and the Isolation Forest parameters, including the response-frame offset, are not reported. If these parameters are tuned on the same data used for RQ1-RQ3, the reported precision, recall, and timing errors are in-sample estimates. Please use a held-out validation split or cross-validation, and report a sensitivity analysis showing that the results are stable over a reasonable range of parameter values.","section":"Section IV.B.2 and Section IV.C"}],"minor_comments":[{"comment":"The abstract says finish time is measured within 100 ms error for 'over 89% of interactions', but Table IV(b) reports only 88% within 100 ms; please make the numbers consistent.","section":"Abstract and Table IV"},{"comment":"The method depends on the Android Show taps overlay being present in every screencast, and the evaluation only uses V2S recordings that already have the overlay. Please state explicitly that recordings without the overlay are out of scope, and describe how the production pipeline verifies that the overlay is enabled before relying on the interaction detection step.","section":"Section IV.B.1"},{"comment":"Recording overhead is reported as single average deltas (16 ms response, 51 ms finish) from only three apps and ten interactions per app, with no variance or confidence intervals; please report per-interaction distributions so the overhead claim is assessable.","section":"Section V, RQ5"},{"comment":"Table V is difficult to read because the response-time and finish-time accuracy blocks appear to repeat the same column headers, and the finish-time columns are labeled inconsistently; please reformat the table so the metric for each block is clear.","section":"Table V"},{"comment":"The text around Figure 5 contains garbled characters in the submitted PDF; please verify that the figure caption and axis labels render correctly in the final version.","section":"Figure 5"},{"comment":"The claim of being 'the first black-box technique' for measuring GUI responsiveness from screencasts is strong; please soften it or provide a more systematic comparison with prior screencast-analysis work such as AdaT and GIFDroid.","section":"Contributions"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid practical contribution and a valuable released dataset, but the undisclosed Isolation Forest offset and the selective RQ2 evaluation are the main risks to the headline claims. I believe these can be addressed with additional analysis, so major revision is appropriate rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nQuick take: MobileGUIPerf is a credible industrial tool paper with a genuinely useful new annotated dataset, but the timing accuracy claims are weaker than they look. The offset in the Isolation Forest keyframe refinement is never specified, and the thresholds (10-pixel tap/swipe split, offset) are tuned on the same data used for evaluation. That's enough to make the 95%/89% numbers provisional, not invalid.\n\nWhat's new: the pipeline combining tap-indicator detection (Faster R-CNN) with SSIM and Isolation Forest to locate response/finish frames is a sensible combination, and the 2,458-interaction benchmark from 64 apps is a real contribution. The paper also reports per-category accuracy and an alerting evaluation with practical thresholds, which is more than many tool papers do. The industrial deployment story adds credibility that the tool actually works at scale.\n\nWhere it's soft: the stress-test concern lands. Section IV.C says 'we introduce an offset in the Isolation Forest algorithm' but never gives the value, formula, or selection procedure. That's a load-bearing detail because it encodes the belief that users care about 'substantial visual transitions' rather than 'minor effects such as a button dimming' — which drifts from the stated definition of response time as 'first visible GUI frame update.' If the ground truth annotators applied the same intuitive bias, high agreement measures consensus with a shared heuristic, not accuracy against the definition. The paper reports no inter-annotator agreement, so you can't separate the two. The abstract also claims 'over 89%' for both metrics, but Table IV shows 88% of finish times within 100 ms. Minor, but sloppy.\n\nThe lack of quantitative baselines (GUIWatcher, AdaT, or even a simple frame-diff heuristic) is a missed opportunity; you can't tell if the Isolation Forest machinery actually beats a cheaper method. The dependency on the Show taps overlay is honestly stated but limits generality.\n\nWho it's for: researchers and practitioners in mobile GUI testing and performance analysis. The dataset alone justifies a look. I'd send it to review — it deserves a serious referee — but I'd push hard for the offset to be disclosed, the tuning procedure clarified, and inter-annotator reliability reported.\n\nRecommendation: engage with it, but treat the timing numbers as upper bounds until the offset is on the table.","headline":"A practical black-box screencast analyzer with a genuinely useful benchmark, but the headline timing numbers rest on an undisclosed tuning offset and in-sample thresholds; worth reviewing, not worth trusting as-is.","tokens_in":16748,"tokens_out":2469,"would_cite":true,"duration_ms":28398,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"MobileGUIPerf claims that GUI responsiveness can be measured black-box from screencasts by detecting Android's 'Show taps' indicator and timing visual frame changes.","keywords":["GUI responsiveness","mobile app testing","screencast analysis","computer vision","performance measurement","Android Show taps","user-perceived performance","SSIM"],"falsifier":"Take a set of V2S-style videos, programmatically inpaint or remove the 'Show taps' circles, and run MobileGUIPerf against the same ground-truth annotations: if interaction detection precision and recall stay near 0.9, the assumed reliance on the overlay is wrong, while a collapse would confirm that the overlay is load-bearing.","tokens_in":15753,"feed_emoji":"⏱️","tokens_out":6126,"duration_ms":70136,"temperature":0.7,"pith_summary":"MobileGUIPerf sets out to prove that user-perceived GUI responsiveness can be measured directly from screen recordings, with no access to app source code or system internals. It detects each tap or swipe through the Android 'Show taps' overlay, then times the app's reaction by watching frame-to-frame visual change. On a manually annotated benchmark of 2,458 interactions from 64 popular Android apps, the paper reports 0.96 precision and 0.93 recall for detecting interactions, response-time measurement within 50 ms for 95% of interactions, and finish-time within six frames for 89%. The payoff is practical: automated test pipelines can flag unresponsive interactions at scale, and the tool is already deployed in an industrial pipeline analyzing thousands of screencasts daily.","feed_headline":"Screencast tool times app lag within 50 ms for 95% of taps","feed_subtitle":"Reads Android's 'Show taps' circles and frame changes to flag unresponsive interactions at industrial scale.","key_machinery":"The central object is the Android 'Show taps' overlay, a semi-transparent circle Android renders at each touch point; it is the visual synchronization marker that lets every user action be located in the video. Around it, the pipeline uses two detection techniques: Faster R-CNN to spot the overlay in each frame, and SSIM-based frame differencing followed by Isolation Forest to identify which frame is the first visual response and which frame is the stable end. The response time is the timestamp difference from the segment's first frame to the response frame, and the finish time is the difference to the finish frame.","core_discovery":"The paper's central discovery is that a screencast alone contains enough timing information to quantify GUI responsiveness: the 'Show taps' overlay marks exactly when and where each user action occurs, and the pixels show when the interface first changes and when it settles. MobileGUIPerf operationalizes this by running a Faster R-CNN detector to find the overlay, grouping detections into interaction segments, classifying each as tap or swipe from the indicator's movement, and computing SSIM between consecutive frames. It then treats the first and last frames that deviate strongly from the local visual flow as the response and finish frames, using Isolation Forest to find those outliers, with an offset calibrated to practitioner notions of a noticeable transition. The resulting per-interaction response and finish times match manual annotation closely enough that the authors claim the method works as an automatic performance-alerting system, flagging interactions that exceed 100 ms response or 1000 ms finish thresholds with over 96% precision and recall for response time.","pith_inferences":["A natural extension is to use the same segmentation to compute per-app responsiveness distributions over a test suite, turning individual measurements into regression alarms when a release shifts the distribution.","The reliance on 'Show taps' suggests the approach could generalize to any platform or recorder that renders a persistent visual cursor; iOS without such an overlay would need a different interaction marker.","The frame-level granularity of about 16.7 ms at 60 fps means the reported 50 ms response accuracy is near the theoretical floor for this input; sub-frame interpolation would be needed to tighten it.","If the overlay's visibility is the true bottleneck, an industry deployment could inject a synthetic overlay at recording time, keeping the black-box spirit without waiting for Android's debug option to be enabled."],"forward_implications":["Automated GUI test pipelines can add user-perceived responsiveness checks without modifying the app under test, since the input is an ordinary screencast.","Interactions that exceed the 100 ms response or 1000 ms finish thresholds can be flagged automatically at scale: the paper reports 96.8% precision and 96.3% recall on response-time alerting and 88.1% precision and 90.6% recall on finish-time alerting.","The processing cost, about 9 seconds for a 5-second 60 fps interaction, and the recording overhead, about 16 ms response and 51 ms finish on a Pixel 7, are small enough for daily large-scale use.","Because accuracy holds across 32 app categories but degrades for animation-heavy apps, the technique is a general screening tool rather than a per-app-calibrated profiler."],"supporting_citations":[{"why":"Supplies the V2S screencast dataset and the pre-trained Faster R-CNN tap-indicator detector that MobileGUIPerf reuses for interaction segmentation.","marker":"[32]"},{"why":"Defines SSIM, the frame-similarity measure used to detect visual changes and locate response and finish frames.","marker":"[15]"},{"why":"Describes Isolation Forest, the anomaly detector used to flag outlier frames as visual transitions.","marker":"[40]"},{"why":"Presents Faster R-CNN, the object-detection architecture behind the tap-indicator model.","marker":"[39]"},{"why":"Establishes the roughly 100 ms perception threshold that motivates why response-time accuracy within 50 ms is sufficient and why the two metrics matter.","marker":"[17]"},{"why":"Provides scrcpy, the recording tool whose low overhead is measured in RQ5 and used for capture.","marker":"[36]"}],"fun_headline_variants":["Screencast-only lag detection: 96% precision on slow taps","Tap overlay + SSIM = 50ms-accurate GUI delay timing","From screencasts to response times: 0.96 precision on delays","Flagging unresponsive UI from video alone: 96% precision","GUI lag measurement from screencasts: 50ms error on 89% of taps"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method works only when every screencast shows the Android 'Show taps' circle over every touch, and only when the object detector never confuses a UI element for that circle; without the visible overlay, the pipeline cannot segment the video into interactions, so the reported timing accuracy does not apply.","fun_headline_variants_meta":{"raw":{"variants":["Screencast-only lag detection: 96% precision on slow taps","Tap overlay + SSIM = 50ms-accurate GUI delay timing","From screencasts to response times: 0.96 precision on delays","Flagging unresponsive UI from video alone: 96% precision","GUI lag measurement from screencasts: 50ms error on 89% of taps"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000264,"raw_usage":{"total_tokens":1630,"prompt_tokens":999,"completion_tokens":631,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":615,"completion_tokens_details":{"reasoning_tokens":529}},"tokens_in":615,"tokens_out":631,"duration_ms":7314,"temperature":1.0,"reasoning_tokens":529,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T05:40:22.756990+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a set of V2S-style videos, programmatically inpaint or remove the 'Show taps' circles, and run MobileGUIPerf against the same ground-truth annotations: if interaction detection precision and recall stay near 0.9, the assumed reliance on the overlay is wrong, while a collapse would confirm that the overlay is load-bearing.","supporting_citations":[{"cited_title":"Translating video recordings of mobile app usages into replayable scenarios,","cited_arxiv_id":null,"evidence_quote":"Supplies the V2S screencast dataset and the pre-trained Faster R-CNN tap-indicator detector that MobileGUIPerf reuses for interaction segmentation."},{"cited_title":"Developers","cited_arxiv_id":null,"evidence_quote":"Presents Faster R-CNN, the object-detection architecture behind the tap-indicator model."},{"cited_title":"Need for non-visual feedback with long response times in mobile hci,","cited_arxiv_id":null,"evidence_quote":"Establishes the roughly 100 ms perception threshold that motivates why response-time accuracy within 50 ms is sufficient and why the two metrics matter."},{"cited_title":"scrcpy: Display and control your android device","cited_arxiv_id":null,"evidence_quote":"Provides scrcpy, the recording tool whose low overhead is measured in RQ5 and used for capture."}],"review_version":1}