{"id":"6c4af044-a5e5-4dbe-a788-eb9add873e2c","arxiv_id":"2412.08581","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A multi-agent LLM system with a scenario knowledge graph can automate soap opera testing on Android apps, finding real bugs but with more false positives than manual testing.","lead":"Researchers built a system of three AI agents, guided by a knowledge graph of past bug reports, that automatically executes scenario-based exploratory tests (soap opera testing) on Android apps. The system finds real bugs but still lags behind human testers, with many false alarms.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Automated true positives in Table 1 are author-adjudicated while manual bugs are developer-validated; the two columns are not commensurable, so the feasibility/gap claim rests on unvalidated ground truth.","rationale":"The paper is a feasibility study and is transparent about limitations; the formative study and the multi-agent design are reasonable, and the authors did obtain developer feedback on 34 reports. The reader's CONDITIONAL verdict is sensible. My strongest concern is narrower than the reader's: not whether the SKG/screenshots are sufficient, but whether the headline numbers that define \"feasibility\" and \"gap\" are measured with commensurable ground truth. Manual bug counts are developer-validated; automated bug counts are author-validated. Since the final claim is explicitly comparative (\"significant gap compared to manual execution\"), the comparison must hold under the same validation standard. The current evidence (34/68 submitted, mixed maintainer responses) is suggestive but not decisive. This does not require rejection; it reinforces a conditional acceptance pending independent validation. I also note the paper includes a replication-package URL in the Data Availability section, so the reader's claim of no available package should be checked rather than assumed.","tokens_in":21033,"tokens_out":4582,"duration_ms":51551,"concrete_test":"Have the authors (or an independent party) submit all 68 author-confirmed automated bugs to Firefox/WordPress/AntennaPod issue trackers, preserving the same report quality as the 34 already submitted, and compute the developer-confirmation rate for automated versus manual bugs. If the confirmed rate is materially lower than the author-adjudicated rate (e.g., below the manual rows' confirmation rate), then Table 1's true-positive counts and the magnitude of the claimed gap must be revised. A cheaper alternative: blind independent testers reproduce all 120 candidate detections on devices; count only those they and the maintainers confirm.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central feasibility claim is supported by Table 1, which reports 32/21/15 automated true positives at roughly 50-66% accuracy, and by the contrast with manual soap opera testing. But the two sides of that table are not measured the same way. Manual rows (Sections 2.1.1 and 2.2) count bugs that were reported to project trackers and received developer status (fixed/confirmed/assigned/wontfix); automated rows count bugs that two authors reproduced and judged valid (Section 4.1.1). Of the 68 automated \"true positives\", only 34 were submitted to projects, with 3 fixed, 13 confirmed, 6 assigned priority/severity, and 1 duplicate; the remaining 34 were never developer-validated. Since author adjudication is the ground truth for automated rows, the reported true-positive counts and the \"gap compared to manual execution\" could be substantially overstated if author judgments are more lenient than maintainers'. This is not a minor measurement issue: the paper's own false-positive analysis (Section 4.2.2) shows that a large share of author-called bugs (20/52) are enhancement suggestions maintainers might reject. The conclusion that automated testing is \"promising but with a significant gap\" therefore depends on a developer-validation step that was only partially performed.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper investigates whether soap opera testing (scenario-based exploratory testing) can be automated. The authors first conduct a formative study with six practitioners, deriving two insights (observant discovery of unexpected behaviors; creative expansion of exploration boundaries) and two automation challenges (test auto-play and real-time bug detection). They then build a multi-agent LLM system with Planner, Player, and Detector agents augmented by a scenario knowledge graph (SKG) built from each app's bug reports and issues. The system is evaluated on Firefox, WordPress, and AntennaPod, using 30 soap opera tests from the formative study. The reported results are 19/30 tests fully executed, 68 author-confirmed bug reports across the three apps with accuracy between 0.516 and 0.656, and an ablation showing that both step and oracle knowledge improve performance. The paper concludes that automated soap opera testing is feasible but still lags manual execution, and outlines a research roadmap.","tokens_in":21304,"tokens_out":6027,"duration_ms":57864,"significance":"The paper's main value is a systematic early feasibility study and a concrete architecture (RAG over SKG plus three LLM agents) that others can build on; the replication package and the detailed failure analysis (four causes of false positives) are useful contributions. If the results are taken at face value, they provide the first evidence that LLM-driven soap opera testing can execute complex NL scenarios and surface developer-confirmed bugs, and the roadmap (neural-symbolic synergy, human-AI co-learning, integration with code-level knowledge) is plausible and grounded in the observed failure modes. However, the central quantitative comparison to manual testing is not yet on equal footing, and the circularity of oracle-based detection needs a concrete overlap analysis before the 'significant gap' conclusion can be accepted.","major_comments":[{"comment":"The headline comparison in Table 1 is between two differently measured quantities. The manual rows count bugs that were reported to project trackers and received a developer status (Section 2.2 lists 5 fixed, 28 confirmed, 4 assigned, 5 wontfix), while the automated rows count bugs that two authors reproduced and adjudicated as valid (Section 4.1.1). Of the 68 automated true positives, only 34 were submitted to the projects, and of those only 23 have a reported developer status (3 fixed, 13 confirmed, 6 assigned, 1 duplicate; Section 4.2); the other 34 automated true positives were never seen by developers. Because the abstract and Section 1 use this table to support the claim that automated testing shows 'a significant gap compared to manual execution,' the claim currently conflates a measurement difference with a capability difference. Please developer-validate all automated detections (or a clearly specified random sample) and re-present Table 1 with matched validation standards, or explicitly reframe the automated rows as author-adjudicated and soften the gap claim.","section":"Table 1; Sections 2.2, 4.1.1, 4.2"},{"comment":"Even under author adjudication, the two columns of Table 1 are not comparable because a large share of automated true positives are enhancement suggestions rather than defect reports. Section 4.2.1 reports that automated testing produced 19/32 (Firefox), 15/21 (WordPress), and 11/15 (AntennaPod) enhancements, whereas manual testers found only 3/27, 1/14, and 2/16 enhancements. Section 4.2.2 also classifies 20 of 52 false positives as 'unreasonable or unnecessary improvement suggestions,' which shows that the author-adjudication boundary for what counts as a valid finding is more permissive than the manual bug counts suggest. Please stratify Table 1 by bug type (defect vs. enhancement) and, for the 34 submitted automated bugs, report how many were enhancements and whether the developer responses treated them as valid.","section":"Sections 4.2.1 and 4.2.2"},{"comment":"There is a circularity risk in the evaluation: the SKG is built from historical bug reports and issues (Section 3.2.1), and the Detector identifies bugs by retrieving oracles from that same SKG (Section 3.5.2). A detected 'true positive' can therefore be a reproduction of a known, previously reported issue rather than evidence of unexpected-behavior discovery. The ablation in Table 2 shows that oracle knowledge improves detection, but it does not measure how many automated true positives correspond to oracles already present in the SKG. Please add an overlap analysis that maps each detected bug to the historical reports used to construct the SKG (e.g., by keyword or step matching) and reports the number of detections that do not appear in the knowledge base. The paper's own examples of branch-exploration discoveries (e.g., Bug1912905 in Section 4.3) show that genuinely novel detections occur, so this is a correctness-risk concern about the magnitude of the feasibility claim, not an assertion that the approach is vacuous.","section":"Sections 3.2.1, 3.5.2, and Table 2"}],"minor_comments":[{"comment":"Please state the status of the 11 submitted reports that are not covered by the 3 fixed, 13 confirmed, 6 assigned, and 1 duplicate counts (e.g., pending or unanswered).","section":"Sections 1 and 4.2"},{"comment":"There is a typo: 'challendges' should be 'challenges'.","section":"Section 7 (Related Work)"},{"comment":"Please report the full Likert distribution for sub-step generation and provide the rubric used by the two authors; currently only the extremes (72.0% rated 5, 10.0% rated 1 or 2) are given.","section":"Section 4.3"},{"comment":"The chunk size (800 tokens) and embedding dimension (256) are stated as defaults; a short sensitivity analysis would strengthen the claim that the RAG configuration is not responsible for the observed improvements.","section":"Section 3.2.2"},{"comment":"The random sample of 50 Planner-Player execution pairs is not described in enough detail (e.g., stratification by app, test, and step); please include the sampling procedure and the number of disagreements resolved by discussion.","section":"Section 4.1.2"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is a feasibility study and roadmap rather than a mature framework evaluation, and the replication package plus the failure analysis are genuine assets. The main obstacle to publication is the non-comparable validation of the automated vs. manual bug counts; I believe this is fixable in revision by re-analyzing the data and recalibrating the claims. The paper's topic and empirical style fit a software engineering journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Good quick read if you care about LLM-driven GUI testing. The genuinely new thing is end-to-end automation of soap opera testing. Prior work, including the authors' own SYSKG, generated soap opera tests but required manual execution; this paper closes the loop with three LLM agents (Planner, Player, Detector) plus a scenario knowledge graph built from bug reports. The formative study with six practitioners is a nice touch and yields two sensible insights: stay observant during execution, and expand exploration creatively. The ablation shows both step and oracle knowledge contribute, and the bug reports actually submitted to Firefox, WordPress, and AntennaPod got developer responses: three fixed, thirteen confirmed, six assigned priorities. That is real evidence, not just a demo.\n\nThe soft spots are real but not disqualifying. The headline comparison to manual testing in Table 1 is not apples-to-apples: manual counts are developer-validated, automated counts are author-adjudicated, with only 34 of the 68 'true positives' submitted to projects. The stress-test note is right that the true-positive numbers could be inflated by lenient author judgment. The paper's own false-positive analysis shows 20 of 52 are enhancement suggestions maintainers might reject, so the gap statement should be read with that caveat. The evaluation is small (three apps, thirty tests) and element location accuracy is 70%, which the authors report honestly. There is also partial circularity: the Detector uses oracles from the same bug reports that feed the SKG, so some detections are retrieving known issues. But the system also finds real bugs off the known path (e.g., Bug1912905 from an accidental click), so the concern is real but not central.\n\nWho should read it: anyone building LLM agents for GUI testing or exploratory testing automation. It is a feasibility study, not a controlled comparison, so treat the numbers as existence proofs rather than benchmarks. The replication package link is present in the paper, though whether it is populated is unknown. I would send this to peer review: the area is hot, the system is non-trivial, and the limitations are clearly stated. A good referee will ask for a tighter evaluation and a re-analysis of the manual-versus-automated ground-truth asymmetry, but that is what revisions are for.","headline":"A credible feasibility study that automates soap opera testing with LLM agents; the evaluation overclaims a bit due to author-adjudicated ground truth, but the contribution and transparency warrant peer review.","tokens_in":21820,"tokens_out":2999,"would_cite":true,"duration_ms":29721,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Automated soap opera testing driven by LLMs and scenario knowledge is feasible and finds real bugs, but it trails human testers in exploring scenario boundaries and in avoiding false positives.","keywords":["soap opera testing","exploratory testing","large language models","multi-agent systems","scenario knowledge graph","mobile GUI testing","retrieval-augmented generation","bug detection"],"falsifier":"In a fresh replication, replace each app's scenario knowledge graph with an equal-sized set of unrelated text chunks and keep everything else fixed; if full-test execution rates and developer-confirmed bug counts do not fall substantially relative to the full-knowledge run, the paper's central claim that scenario knowledge drives the automation would be contradicted.","tokens_in":20860,"feed_emoji":"🧪","tokens_out":9576,"duration_ms":91278,"temperature":0.7,"pith_summary":"The paper asks whether soap opera testing—scenario-based exploratory testing in which a tester follows a complex user story and watches for unexpected behavior—can be automated with large language models. After a formative study of six professional testers, the authors build a multi-agent system in which a Planner, a Player, and a Detector cooperate to execute natural-language test steps against Android apps and flag anomalies in real time. On 30 soap opera tests across Firefox, WordPress, and AntennaPod, the system fully executed 19 tests and produced 68 bugs verified by the authors, among them bugs later fixed or confirmed by the projects' developers. False positives made up roughly 40-50 percent of automated findings, and automated and manual testers shared only two identical bugs. The authors conclude that automation is feasible but not yet at human level, and they lay out a research agenda centered on neural-plus-symbolic reasoning, human-AI co-learning, and connecting test scenarios to code-level root causes.","feed_headline":"LLM agents found 68 real app bugs in automated 'soap opera' tests","feed_subtitle":"A three-agent LLM system completed 19 of 30 scenarios on Android apps; its 34 bug reports drew 23 developer responses.","key_machinery":"The load-bearing machinery is a Scenario Knowledge Graph (SKG) built from each app's bug reports and issues, where normalized step descriptions carry alternative phrasings and link to expected-behavior oracles. Retrieval-augmented generation injects relevant SKG steps and oracles into the prompts of three multimodal, multi-turn agents: Planner generates an actionable sub-step plan for the current GUI screenshot; Player translates each sub-step into a structured UI action (tap, long-tap, double-tap, input, scroll, home, enter, landscape, portrait), targeting locations by numbered labels overlaid on a 100-pixel grid and executing through a command-line device tool; Detector retrieves oracles matching the executed instruction, asks the LLM to add creative-thinking oracles, and compares screenshots before and after execution to report violations as structured bug reports. The grid-labeling scheme and the oracle-retrieval loop are what let natural-language scenarios become executable actions with real-time anomaly checks.","core_discovery":"The central discovery is that scenario knowledge—not just a generic multimodal LLM—is what makes automated soap opera testing work, and that the approach is real but incomplete. The authors construct a Scenario Knowledge Graph (SKG) from public bug reports and issues, with normalized steps and associated expected-behavior oracles, and feed it to three role-specialized LLM agents through retrieval-augmented generation. In their evaluation, the Planner identified the next test step correctly 92.0 percent of the time, the Player parsed steps into UI instructions correctly 84.0 percent of the time and located elements 70.0 percent of the time, and the system fully executed 19 of 30 soap opera tests. The Detector found 68 true-positive bugs across the three apps, but also 52 false positives, with per-app accuracy of 0.516, 0.656, and 0.577. Ablation runs show that removing oracle knowledge or step knowledge from the SKG lowers true-positive counts and accuracy on every app, and that without step knowledge the number of fully executed tests drops from 19 to 11. The paper reads these results as evidence of feasibility plus a clear gap: automated testing over-generates UI-enhancement suggestions and under-explores the input and branch boundaries that human testers reach through creative thinking.","pith_inferences":["A testable extension is to add an execution-history check to the Detector: before accepting an oracle-triggered bug, verify that the oracle's own action sequence was actually performed, which would directly address the paper's reported 'unexecuted execution plans' hallucinations.","If the low overlap between automated and manual findings holds beyond three apps, then automated soap opera testing is best positioned as a complement to human exploration rather than a replacement, and a workflow that alternates the two should be measured for unique-bug yield.","The paper's 70 percent element-location accuracy points to a cheap improvement route: ground the Player in the app's accessibility hierarchy or widget coordinates instead of 100-pixel grid labels, which would also remove screenshot-timing false positives caused by delayed rendering.","The authors' observation that oracle text can suggest new branches implies a concrete experiment: convert actionable oracle sentences into executable sub-steps and measure whether true-positive rate rises or false-positive rate falls relative to the current Detector-only use of oracles."],"forward_implications":["If the results hold, a maintainer can run soap opera tests automatically on Android apps that have a history of bug reports, harvesting genuine bug reports without a human executing each scenario.","The ablation data imply that building and curating the scenario knowledge graph is the primary engineering investment; a generic LLM without app-specific steps and oracles loses true-positive detections and leaves more tests unfinished.","The false-positive analysis implies that a practical deployment needs a triage step, because roughly one in two automated findings is not a real defect, and many real findings are enhancement suggestions rather than functional bugs.","Because only two bugs overlapped between automated and manual runs, the paper's view is that automated soap opera testing finds a different slice of the bug space, not merely a cheaper version of manual testing.","The roadmap's human-AI co-learning direction implies an immediate middle-ground use: humans filter automated reports while AI-generated unusual actions seed new manual exploration, a pattern already observed when the authors found 16 additional bugs by pursuing branches the automated run opened."],"supporting_citations":[{"why":"Defines soap opera testing, the scenario-exploration practice this paper automates.","marker":"[6]"},{"why":"Supplies the system-knowledge-graph construction pipeline that the SKG adapts.","marker":"[32]"},{"why":"Establishes combining LLMs with a knowledge graph for exploratory testing, which the multi-agent system extends.","marker":"[34]"},{"why":"Provides retrieval-augmented generation, the mechanism that injects SKG steps and oracles into agent prompts.","marker":"[21]"},{"why":"Provides chain-of-thought prompting, which structures the Planner's step-by-step plan generation.","marker":"[39]"},{"why":"Supports the insight that exploratory testers follow a scenario while pursuing side effects, formalized as creative thinking.","marker":"[41]"},{"why":"Supplies the candidate Android app pool from which the three test apps were selected.","marker":"[43]"}],"fun_headline_variants":["LLM test agents find 68 bugs, but scenario knowledge is key","Scenario graph fuels LLM testers to find 68 real bugs","Automated soap opera testing: 68 bugs, 52 false alarms","LLM agents find real bugs but miss creative test boundaries","Scenario knowledge is key for LLM soap opera tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The system assumes that historical bug reports and issue trackers contain enough accurate, up-to-date step instructions and expected behaviors to steer both execution and bug detection; a project lacking such records, or with stale ones, would weaken the automation.","fun_headline_variants_meta":{"raw":{"variants":["LLM test agents find 68 bugs, but scenario knowledge is key","Scenario graph fuels LLM testers to find 68 real bugs","Automated soap opera testing: 68 bugs, 52 false alarms","LLM agents find real bugs but miss creative test boundaries","Scenario knowledge is key for LLM soap opera tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000797,"raw_usage":{"total_tokens":3563,"prompt_tokens":1056,"completion_tokens":2507,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":2419}},"tokens_in":672,"tokens_out":2507,"duration_ms":17745,"temperature":1.0,"reasoning_tokens":2419,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T17:43:20.410445+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"In a fresh replication, replace each app's scenario knowledge graph with an equal-sized set of unrelated text chunks and keep everything else fixed; if full-test execution rates and developer-confirmed bug counts do not fall substantially relative to the full-knowledge run, the paper's central claim that scenario knowledge drives the automation would be contradicted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines soap opera testing, the scenario-exploration practice this paper automates."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the system-knowledge-graph construction pipeline that the SKG adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes combining LLMs with a knowledge graph for exploratory testing, which the multi-agent system extends."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the insight that exploratory testers follow a scenario while pursuing side effects, formalized as creative thinking."}],"review_version":1}