{"id":"629a0f5a-9d42-41ed-840b-655daf4cc1e1","arxiv_id":"2412.02933","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"PopSweeper detects app-blocking pop-ups in mobile GUI screenshots with 93.5% recall and localizes their close buttons, but its real-time resolution claim is validated only in simulation.","lead":"PopSweeper is a tool that watches mobile app screenshots during automated testing, flags pop-up ads or alerts that block the interface, and returns the coordinates of the close button so the test can click it away. The authors built a labeled dataset of 832 blocking pop-ups from RICO and popular apps and report 91.7% precision and 93.5% recall on pop-up classification, but their end-to-end evidence comes from replayed recordings rather than live test runs.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 87.1% end-to-end resolution claim is not actually demonstrated: RQ3 replays recordings and compares predicted close-button coordinates to labels, but never clicks a returned coordinate in a live automated test.","rationale":"The reader's weakest assumption is the same one I identified: RQ3's end-to-end evaluation is a simulated replay that never exercises the actual click-and-dismiss loop in a live app. That assumption is load-bearing because the paper's headline claim is that PopSweeper resolves app-blocking pop-ups to keep automated GUI tests running, and the only evidence for that is a coordinate-matching exercise on recorded frames. If a real test runner's tap at those coordinates frequently fails to dismiss the pop-up—because of animation, timing, coordinate scaling, or view hierarchy mismatches—the practical value of the tool would be lower than the 87.1% figure suggests. The paper's own discussion of error types concedes that some returned coordinates are invalid and clicking them can cause unintended behavior, which heightens the need for a live test. Other issues, such as the inconsistent app counts, the 7K vs 72K screenshot typo, the 92.4% vs 93.5% recall discrepancy, and the placeholder 'John Doe' references, are real manuscript-quality problems but do not by themselves invalidate the central technical approach. My proposed concrete test—a live integration with a real test runner that actually taps the returned coordinates and verifies the pop-up is dismissed—would settle whether the replay results transfer. Until that test is run, the paper should remain CONDITIONAL rather than ACCEPT, which is exactly the reader's verdict, so no change is needed.","tokens_in":21707,"tokens_out":4205,"duration_ms":43229,"concrete_test":"Run a live integration experiment: instrument the same or a comparable set of Android apps (e.g., the 25 recorded apps from RQ3's second experiment) with an automated GUI test runner such as Appium or UIAutomator. For each app, execute a scripted usage session for 60 seconds; when PopSweeper returns close-button coordinates, perform a real tap at those coordinates and then verify, via the runner's UI hierarchy or a post-tap screenshot, that the pop-up is gone and the intended underlying action proceeds. Count the fraction of apps in which the test completes without manual intervention. If this fraction is materially below 87.1%, the replay-based evaluation overstates real-world resolution; if it is close to 87.1%, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central practical claim—PopSweeper 'successfully resolved blockages in 87.1% of apps' (abstract, Section 4.4)—rests on end-to-end experiments that are explicitly replay-based, not live GUI tests. The first RQ3 experiment feeds RICO GIF frames through the pipeline; the second 'automated end-to-end evaluation' (Table 4) is described in the same section as 'simulated': the authors replayed 60-second recordings of 51 apps, sent frames to PopSweeper every 100 ms, and compared returned coordinates with manual ground-truth labels. No tap is ever dispatched to an app, no UI hierarchy is inspected to verify that a pop-up disappeared, and no automated test script continues past the pop-up. Therefore the reported 87.1% app-level resolution rate is at best an upper bound on coordinate agreement, not a measured resolution of a real interruption. Live conditions can break this: pop-up entrance animations can shift a close button between screenshot and tap, the test runner may capture its own scaled coordinates, a tap can be intercepted by an overlapping view, or the returned coordinate may lie on a non-clickable region. The paper itself acknowledges this in Section 5.2: 'the end-to-end app testing scenarios in RQ3 may not fully capture the real-world behavior of pop-ups.' There are also smaller consistency problems—the app-level denominator appears as both 154 and 155 unique apps in Section 4.4, and the abstract/conclusion report slightly different precision (91.7% vs 91.8%), with Section 5.3 citing a 92.4% recall—but the missing live click loop is the load-bearing gap because the headline contribution is specifically to automated GUI testing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"PopSweeper is a computer-vision pipeline for detecting and resolving app-blocking pop-ups during automated mobile GUI testing. It combines histogram-based frame differencing, a two-stage ResNet50+MobileNetV2 classifier, and a YOLO-World close-button detector, and returns close-button coordinates to a test script. The paper contributes a manually labeled dataset of 832 app-blocking pop-ups from the RICO dataset and 87 top-ranked apps, an empirical prevalence study, and an evaluation reporting 91.7% precision and 93.5% recall for classification, 93.9% BoxAP and 89.2% recall for close-button detection, and an app-level resolution rate of 87.1%. The two RQ3 experiments are both offline replays: recorded frames are passed through the pipeline and the returned close-button coordinates are compared with manual ground-truth labels; no tap is dispatched to a live app and no post-tap state is verified.","tokens_in":21998,"tokens_out":6118,"duration_ms":57376,"significance":"The dataset and code release (Section 8), the breadth of the manual labeling effort, and the comparison against several reasonable baselines (ResNet50, MobileNetV2, VGG19, custom CNN, CLIP, Faster R-CNN, YOLOv8) are genuine strengths. The second RQ3 experiment on 51 additional apps provides some out-of-distribution evidence. If the central claim were demonstrated in a live test runner, PopSweeper would be a useful complement to existing automated GUI testing workflows, with an attractively low per-frame overhead. The main weakness is that the headline 'resolved blockages in 87.1% of apps' claim is currently supported only by coordinate agreement on replayed recordings, not by actual resolution of pop-ups in a running test; this gap is acknowledged in Section 5.2 and is the key obstacle to accepting the paper's strongest claim.","major_comments":[{"comment":"The central 'end-to-end resolution' claim is not demonstrated by the experiments as described. In both RQ3 experiments, recorded frames are replayed and the returned close-button coordinates are compared with manual ground-truth labels; no tap is dispatched, no UI hierarchy or screenshot is checked after a tap to verify that the pop-up disappeared, and no test script continues past the pop-up. The reported 87.1% app-level rate and the Table 4 precision/recall values therefore measure offline coordinate agreement, not successful resolution of live app-blocking pop-ups. Pop-up entrance animations, coordinate scaling under a real runner, overlapping views, and returned coordinates on non-clickable regions can all break the transfer. The paper's own Section 5.2 concedes that the RQ3 scenarios 'may not fully capture the real-world behavior of pop-ups.' The stress-test concern lands: either a live test-runner integration must be added, or the abstract and conclusions must be reworded to state that the result is close-button localization agreement rather than demonstrated resolution.","section":"Section 4.4, Table 4"},{"comment":"There is a numeric inconsistency in the key result. The first RQ3 experiment states that the test set contains pop-up screenshots from 154 unique apps, but the app-level result is reported as '135 out of 155 apps (87.1%).' If the denominator is 154, the rate is 87.7%; if it is 155, the earlier sentence should say 155. The manuscript must correct this and state the exact denominator for the headline 87.1% figure, since the contradiction undermines reproducibility of the abstract's main quantitative claim.","section":"Section 4.4, app-level results"},{"comment":"The end-to-end metrics in Equations (10) and (11) are under-specified. The text says the authors 'compare the returned coordinates with the ground truth,' but it does not define a correctness criterion: no IoU threshold, no pixel-distance tolerance, and no requirement that the predicted point lie inside the ground-truth close-button box are given. Without this operational definition, the precision/recall values in Table 4 and the app-level resolution rate cannot be independently reproduced or compared with future detection-based approaches.","section":"Section 4.4, Evaluation Metrics"},{"comment":"Manual annotation quality is not quantified. The paper relies on manually labeled pop-up regions and close-button boxes both for training and for ground-truth evaluation, and Section 5.2 acknowledges that 'the reliance on manual labeling ... may introduce subjectivity,' but no inter-annotator agreement (e.g., Cohen's kappa), no independent second-pass verification, and no detailed annotation protocol are reported. Because all reported metrics inherit the label quality, the paper should report at least a sampled agreement study or otherwise characterize label reliability.","section":"Section 5.2 (construct validity)"}],"minor_comments":[{"comment":"The abstract reports 91.7% precision while the conclusion reports 91.8%, and Table 2 shows 0.917; these numbers should be unified.","section":"Abstract vs. Conclusion"},{"comment":"Section 5.2 says PopSweeper achieved 'a high recall of 92.4%' in identifying pop-ups, which does not match the 93.5% recall reported in Table 2 and the abstract; Section 5.3 repeats the 92.4% figure. This inconsistency should be corrected.","section":"Section 5.2 and Section 5.3"},{"comment":"The conclusion says 'we manually reviewed over 7K screenshots from the RICO dataset,' but the dataset described throughout the paper is 72,218 screenshots; '7K' appears to be a typo.","section":"Conclusion, Section 7"},{"comment":"References [14] and [15] list 'John Doe and Jane Smith' and are not verifiable, and [1] is 'Anonymous' before the abstract. Placeholder or unverifiable citations are not acceptable in a journal submission; all bibliography entries should be real, complete, and traceable.","section":"References [14], [15], [1]"},{"comment":"Section 3.2 contains 'We leverages a two-stage classification pipeline,' which is a subject-verb agreement error; a proofread pass would improve readability.","section":"Section 3.2, sentence-level clarity"},{"comment":"Section 4.1 says 'we ultimately selected approximately 1,000 unique apps from the Rico dataset,' but the RQ3 test set is described as containing 154 unique apps; the relationship between the full curated set and the RQ3 test subset should be stated explicitly.","section":"Section 4.1 vs. Section 4.4"}],"recommendation":"major_revision","confidential_remarks":"The placeholder references [14] and [15] should be verified and corrected before further consideration; this is an integrity issue independent of the technical evaluation. The paper fits the journal's scope, and I see no novelty-disclosure problem. The key technical risk is the live-integration gap around the 87.1% resolution claim; a live integration with even a subset of apps (e.g., through UIAutomator or Appium) would substantially increase confidence in the headline result, whereas the current replay-based comparison supports only a coordinate-agreement claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper. It defines a genuinely useful task—screenshot-based detection of app-blocking pop-ups for automated mobile GUI testing—and ships a manually labeled dataset of 832 pop-ups plus prevalence figures (8% of RICO apps, 42% of top apps) that the community will likely reuse. But the headline claim, that PopSweeper resolved blockages in 87.1% of apps, is not actually demonstrated. RQ3 replays 60-second recordings and compares predicted close-button coordinates with ground-truth labels; no tap is dispatched, no UI hierarchy is inspected, and no test script continues past the pop-up. The paper admits this in Section 5.2, yet the abstract and conclusion still say 'resolved.' That is the load-bearing gap, because the whole pitch is real-time unblocking during live tests.\n\nWhat is solid: the classification and detection RQs are internally coherent. The two-stage ResNet50/MobileNetV2 stacking beats the baselines on F1 (0.926 vs 0.905 for the best baseline), and YOLO-World is competitive with Faster R-CNN and YOLOv8 on BoxAP and recall. The 60ms per-frame overhead and 0.0048s similarity check are measured and plausible. The prevalence study is a real contribution on its own.\n\nSoft spots, in proportion: the missing live loop is major; the rest is minor. The paper quotes 154 unique apps in one place and 155 in another, reports precision as 91.7% vs 91.8% across abstract and conclusion, and Section 5.3 mentions a 92.4% recall that does not match the 93.5% elsewhere. It also has placeholder 'John Doe' references [14][15], which should have been caught before submission. Annotation quality is described but not measured; no inter-rater agreement is reported.\n\nWho is this for: software testing researchers, especially people doing large-scale app testing. As a dataset-plus-task paper, it deserves serious review. As a tool paper, it needs a live integration experiment, or the claims need to be re-scoped to coordinate suggestion rather than resolution. I would send it to review with a clear request for major revision and an artifact check.","headline":"Useful dataset and task definition, but the marquee end-to-end number is a replay-based upper bound, not a demonstrated live resolution.","tokens_in":22585,"tokens_out":3082,"would_cite":true,"duration_ms":28391,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A screenshot-only pipeline can detect app-blocking pop-ups during mobile GUI tests and return close-button coordinates, resolving blockages in 87.1% of apps in end-to-end evaluation.","keywords":["mobile GUI testing","in-app advertisements","app-blocking pop-ups","close button detection","computer vision","screenshot classification","Android app testing","record-and-replay testing"],"falsifier":"Run PopSweeper inside a real automated GUI test runner on a fresh set of 50 popular apps, letting the test script click the returned coordinates as soon as they arrive. If the app-level blockage-resolution rate falls substantially below the claimed 87.1%, or if clicks frequently land at the wrong moment because a pop-up is still animating or has already auto-closed, the central claim is not supported. A cleaner direct check is to count how often a click at the returned coordinates actually dismisses the pop-up (verified by screenshot before and after) in live runs versus the replay-based estimate.","tokens_in":21499,"feed_emoji":"📱","tokens_out":6967,"duration_ms":58426,"temperature":0.7,"pith_summary":"PopSweeper is a computer-vision pipeline that watches the screen during an automated mobile GUI test, recognizes when an advertisement or system alert is blocking the app, and supplies the coordinates of the close button so the test script can tap it and continue. The paper argues this is a real and unmet need: in a manual review of over 72,000 GUI screenshots plus 87 popular apps from two app stores, it found app-blocking pop-ups in 8.3% of the open-source apps and 42% of popular apps. On its collected dataset of 832 labeled pop-ups, the two-stage classifier reaches 91.7% precision and 93.5% recall, the close-button detector reaches 93.9% BoxAP and 89.2% recall, and the end-to-end replay evaluation resolves all blockages in 87.1% of apps. The authors' conclusion is that automated GUI testing can proceed with minimal manual intervention, at roughly 60 milliseconds of added processing per selected frame. If true, this would remove a common cause of flaky or stalled mobile test runs.","feed_headline":"PopSweeper clears app-blocking pop-ups in 87.1% of apps","feed_subtitle":"Computer-vision pipeline detects ads and alerts and returns close-button coordinates, keeping automated GUI tests running.","key_machinery":"The load-bearing mechanism is a three-stage real-time pipeline. First, a 100 ms sampler plus RGB histogram similarity (threshold 0.8) sends only visually changed frames onward, cutting the frame volume by roughly an order of magnitude. Second, a two-stage classifier stacks ResNet50 as primary detector and MobileNetV2 as a verification stage, with custom binary-classification heads, so that a frame is treated as a pop-up only when both models agree; this stacking is what lifts precision from about 0.87 (ResNet50 alone) to 0.917 while keeping recall near 0.935. Third, YOLO-World, fine-tuned on 832 manually labeled pop-up screenshots, outputs a 640x640 bounding box (x1,y1,x2,y2) for the close button. The pipeline's output is not a label but an actionable coordinate that a test script can click, which is what turns detection into resolution.","core_discovery":"An app-blocking pop-up can be treated as a visual anomaly during GUI testing rather than a special case that the test script must predict. PopSweeper establishes that a lightweight screenshot-only pipeline can detect these pop-ups and localize their close buttons with enough accuracy and speed to be inserted into an existing test workflow: sample frames every 100 ms, drop near-duplicates by histogram similarity, classify remaining frames with a two-stage ResNet50/MobileNetV2 stack, and, when a pop-up is found, run YOLO-World to return a clickable bounding box. The central empirical claim is the combination of classification precision 91.7%, recall 93.5%; close-button detection BoxAP 93.9%, recall 89.2%; and an end-to-end app-level resolution rate of 87.1% across 155 apps, at an average processing cost of roughly 60 ms per selected frame. The authors claim this is the first screenshot-based system that both detects app-blocking pop-ups and provides the resolution action, and they position it as a complement to existing exploration and LLM-driven GUI testing agents rather than a replacement.","pith_inferences":["Beyond the paper, the same pipeline could be dropped in front of LLM-driven GUI agents, which the paper notes currently analyze screenshots without pop-up awareness, letting agents receive a cleaned frame before choosing their next action.","Beyond the paper, the histogram gate may miss pop-ups that appear and vanish within one 100 ms sampling window or that barely change the global RGB distribution, so a change-detection baseline that also tracks local regions would be a natural testable extension.","Beyond the paper, the lower performance on fullscreen text-based and Chinese close buttons suggests a concrete next experiment: retrain the detector with a balanced multilingual close-button corpus and measure whether the 37.5% recall on fullscreen top-app pop-ups recovers.","Beyond the paper, the public dataset of 832 labeled pop-up screenshots could support a benchmark for open-vocabulary pop-up detection, since YOLO-World is an open-vocabulary detector and the paper only uses it in a fixed fine-tuned mode."],"forward_implications":["Automated GUI test scripts can be extended with a few lines: send each 100 ms frame to PopSweeper, and if coordinates come back, tap there and continue; no per-app pop-up modeling is needed.","Large-scale comparative and regression testing, where one team runs many apps across devices, would no longer stall at ads and system alerts that appear unpredictably.","The reported error analysis says that misclassifying a pop-up as app content behaves like not using the tool, while the rare bad-coordinate clicks within a pop-up can still redirect, so the tool degrades gracefully rather than crashing the run.","Because the system works on screenshots alone, it applies to any GUI test setup that can produce frames, without instrumenting the app or needing layout or XML access."],"supporting_citations":[{"why":"Supplies the 72,218 GUI screenshots from roughly 9K apps that were manually reviewed and used as the main source of pop-up and app-content training data.","marker":"[12]"},{"why":"YOLO-World is the model fine-tuned as the close-button detector; the paper's BoxAP and recall numbers are measured on its output.","marker":"[8]"},{"why":"ResNet50 is the primary stage of the two-stage classifier whose stacking yields the reported precision and recall.","marker":"[27]"},{"why":"MobileNetV2 is the secondary verification stage that filters false positives in the classifier.","marker":"[50]"},{"why":"CLIP serves as the zero-shot vision-model baseline whose low precision motivates the supervised two-stage design.","marker":"[45]"},{"why":"Faster R-CNN is the object-detection baseline compared for close-button localization.","marker":"[46]"},{"why":"YOLOv8 is the object-detection baseline compared for close-button localization, showing equal recall but slightly lower mAP than YOLO-World.","marker":"[56]"}],"fun_headline_variants":["PopSweeper clears 87.1% of app-blocking pop-ups automatically","PopSweeper: real-time pop-up detection and close for GUI tests","PopSweeper spots app pop-ups in 60ms and clicks close","PopSweeper uses CV to resolve app pop-ups with 91.7% precision","PopSweeper keeps GUI tests running by auto-dismissing pop-ups"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the 60-second recorded replays used for end-to-end evaluation behave like a live test: if pop-up timing, animations, dynamically loaded content, or close-button hit behavior differs under a real test runner, the reported 87.1% resolution rate may not transfer, and the metrics also depend on the undocumented accuracy of the manual annotations.","fun_headline_variants_meta":{"raw":{"variants":["PopSweeper clears 87.1% of app-blocking pop-ups automatically","PopSweeper: real-time pop-up detection and close for GUI tests","PopSweeper spots app pop-ups in 60ms and clicks close","PopSweeper uses CV to resolve app pop-ups with 91.7% precision","PopSweeper keeps GUI tests running by auto-dismissing pop-ups"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000271,"raw_usage":{"total_tokens":1694,"prompt_tokens":1073,"completion_tokens":621,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":689,"completion_tokens_details":{"reasoning_tokens":515}},"tokens_in":689,"tokens_out":621,"duration_ms":45019,"temperature":1.0,"reasoning_tokens":515,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:56:27.826004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PopSweeper inside a real automated GUI test runner on a fresh set of 50 popular apps, letting the test script click the returned coordinates as soon as they arrive. If the app-level blockage-resolution rate falls substantially below the claimed 87.1%, or if clicks frequently land at the wrong moment because a pop-up is still animating or has already auto-closed, the central claim is not supported. A cleaner direct check is to count how often a click at the returned coordinates actually dismisses the pop-up (verified by screenshot before and after) in live runs versus the replay-based estimate.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the 72,218 GUI screenshots from roughly 9K apps that were manually reviewed and used as the main source of pop-up and app-content training data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"YOLO-World is the model fine-tuned as the close-button detector; the paper's BoxAP and recall numbers are measured on its output."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ResNet50 is the primary stage of the two-stage classifier whose stacking yields the reported precision and recall."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MobileNetV2 is the secondary verification stage that filters false positives in the classifier."}],"review_version":1}