{"id":"7b4b9dd0-00dd-475b-9377-2df2f256a72e","arxiv_id":"2506.00520","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A multi-agent LLM system layered on an existing crawler raises code coverage on six web applications and surfaces 445 unique faults on 20 real-world sites.","lead":"Temac combines a conventional web crawler with LLM agents that summarize explored pages, infer untested features, and execute targeted test tasks. The authors report higher code coverage than four existing tools on six web apps and 445 unique faults on 20 real-world sites.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claim of improvement within a one-hour budget is not yet supported: knowledge-base construction time is explicitly excluded from Temac's budget, so Temac runs longer than 60 minutes wall-clock while baselines are capped at 60.","rationale":"The reader's weakest_assumption focuses on whether Summarizer descriptions and coverage reports are accurate enough for Reviser and Navigator to direct execution. That is a real mechanism-level risk, but the ablations in RQ3 and RQ4 indirectly support those components, and the consistent final coverage gains suggest the pipeline works at least on the six studied subjects. The more load-bearing threat to the central quantitative claim is the budget accounting: the paper explicitly excludes knowledge-base construction time from the evaluation, yet claims a one-hour time budget. Because Temac's coverage curves are still rising when the budget ends, extra wall-clock time is a concrete alternative explanation for part of the improvement. This is not a fatal flaw: if the excluded construction time is small or genuinely parallelizable, an equal-wall-clock comparison will likely preserve the qualitative conclusion. The central mechanism and ablations are otherwise credible, and the reported 12.5%–60.3% range is consistent with recomputation against the four baseline averages. I therefore keep the reader's CONDITIONAL verdict rather than moving to rejection, but add the time-budget check as an explicit condition for accepting the one-hour improvement claim.","tokens_in":18449,"tokens_out":9766,"duration_ms":95451,"concrete_test":"Instrument Temac to log wall-clock durations for the Summarizer, Reviser, and any other knowledge-base construction steps. Then run Temac under a hard 60-minute wall-clock cap that includes construction (for example, by reducing the 30-minute execution phase by the measured construction time), and run the strongest baseline, WebRLED, for 60 + T minutes, where T is the measured construction time. If Temac still beats WebRLED by roughly the same relative margin and the coverage trends at 60 minutes are similar, the one-hour-budget claim is preserved; if the margin shrinks materially or WebRLED's extra T minutes close the gap, the reported improvement overstates Temac's advantage. Also report per-phase durations to assess whether the claimed parallelization is feasible in practice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section IV states 'We exclude the time cost for constructing the knowledge base in our evaluation,' and Section III-B places Summarizer and Reviser sequentially after the 30-minute exploration phase. Section V-A sets 'a one-hour time budget' for all experiments, so Temac's actual wall-clock time is 60 minutes plus the duration of the knowledge-base construction phase, while Crawljax, FragGen, WebExplor, and WebRLED are limited to exactly 60 minutes of testing. The reported 12.5%–60.3% improvement is therefore not a like-for-like comparison under the stated budget. This matters because Section V-B reports that Temac's coverage curves for 4gaBoards, Gadael, and Parabank 'are not yet stagnated,' so additional wall-clock time is a plausible alternative explanation for at least part of the observed gain. The Introduction's central claim of improvement 'in a one-hour time budget' is directly affected. The paper's suggestion that knowledge-base construction can be parallelized does not apply to the implementation actually evaluated, which runs the phases sequentially. Measuring this excluded duration is necessary before accepting the quantitative headline.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Temac, a multi-agent LLM-based testing system layered on top of a non-LLM automated web GUI testing (AWGT) tool. Temac operates in three phases: (i) exploration, where an existing AWGT approach (WebRLED) runs for 30 minutes and collects states, actions, screenshots, and coverage reports; (ii) knowledge-base construction, where a Summarizer agent converts screenshots and the state transition graph into natural-language descriptions and a Reviser agent infers not-covered functionalities; and (iii) task execution, where a Navigator agent selects a key state from the graph and a planner-actor-decoupled Executor agent performs the inferred tasks. The evaluation compares Temac against Crawljax, FragGen, WebExplor, and WebRLED on six open-source web applications, reporting line-coverage improvements, and applies a variant (Temac-noCR) to 20 popular real-world sites, reporting 445 unique faults. The paper claims Temac is the first LLM-enhanced AWGT approach targeting code coverage and that its results demonstrate effectiveness and general applicability.","tokens_in":18681,"tokens_out":4030,"duration_ms":36047,"significance":"If the reported results hold, Temac would be a useful and practical contribution: it combines the speed and breadth of non-LLM exploration with the semantic and planning capabilities of LLM agents, and it does so with a modular architecture that could accommodate different underlying AWGT tools, MLLMs, and actors. The paper also provides ablations (Temac-LLM, Temac-RL, Temac-noSTG, Temac-noCR) that give evidence for the contribution of each component, and the consistency of the coverage improvement across all six applications is encouraging. However, the headline quantitative claims contain a reporting error, and the evaluation's time-budget fairness is currently questionable because knowledge-base construction time is excluded from Temac's one-hour budget. These issues are load-bearing and need to be addressed before the central claims can be accepted as stated.","major_comments":[{"comment":"The claim of \"average improvement ranging from 12.5% to 60.3%\" is not supported by the reported numbers as a range. The value 12.5% is the average relative improvement of Temac over WebRLED computed from Table IV (44.18% to 49.71%, i.e., (49.71−44.18)/44.18 ≈ 12.5%), while 60.3% is the absolute line coverage achieved by Temac on the Timeoff application in Table III. The manuscript never reports per-application relative improvements, so no range of improvements of the form stated in the abstract, introduction, and conclusion exists. Please compute and report per-application relative improvements using the stated formula, or rephrase the summary statistic to distinguish average relative improvement from per-application absolute coverage.","section":"V-B, Table III, Abstract"},{"comment":"The claimed one-hour time budget comparison is unfair because Temac's knowledge-base construction time is explicitly excluded from the evaluation, and the implementation executes this phase sequentially after the 30-minute exploration phase. Thus Temac's actual wall-clock time is 60 minutes plus the duration of the Summarizer, Reviser, Navigator, and Executor setup and knowledge-base construction, while the four baselines are capped at exactly 60 minutes. Since Section V-B states that Temac's coverage curves for 4gaBoards, Gadael, and Parabank are \"not yet stagnated,\" the additional wall-clock time is a plausible partial explanation for at least some of the observed coverage gain. Please either include the knowledge-base construction time in the one-hour budget, or run a control that grants the strongest baseline the same additional time, and report the time breakdown.","section":"IV and V-A"},{"comment":"The abstract states that \"Temac\" reveals 445 unique faults in the top 20 real-world web applications, but RQ5 explicitly uses the Temac-noCR variant, not the full Temac system, because coverage reports are unavailable for real-world sites. This is an overstatement of the evaluated system: the 445 faults were produced by a configuration that omits the coverage-report component whose contribution is a central part of the paper's contribution. Moreover, RQ5 does not run any baseline (e.g., WebRLED alone) on the same 20 applications, so the reader cannot assess how many of these faults are specifically attributable to Temac's multi-agent layer. Please qualify the fault-detection claim to refer to Temac-noCR and add a baseline comparison, or discuss this as an exploratory finding rather than as evidence for the full Temac approach.","section":"V-F and Abstract"}],"minor_comments":[{"comment":"The legend in Figure 6 uses \"WebExplore\" while the text and tables consistently use \"WebExplor\"; please unify the naming.","section":"V-A / Figure 6"},{"comment":"The paper reports three repetitions per experiment but provides no variance information, confidence intervals, or statistical tests, despite acknowledging that AWGT approaches exhibit randomness; please add at least the per-application variance or standard deviation.","section":"V-A"},{"comment":"The application named \"4gaBoards\" in the text and tables appears as \"4ga Boards\" in Table II; please make the naming consistent.","section":"V-A / Table II"},{"comment":"The list of the \"top 20\" Alexa applications and the date of the ranking used are not provided, which hinders reproducibility of the RQ5 fault-detection results; please include the list and the ranking date.","section":"V-F"},{"comment":"The estimated cost of \"approximately $1.7 per one-hour testing process\" does not appear to account for the UI-TARS actor served on an A100 GPU or for any additional API calls beyond the four agents; please clarify what cost components are included in this estimate.","section":"VII-B"},{"comment":"The provided open-source repository link points to a Google Drive folder labeled \"Seeker\" rather than \"Temac\"; please verify that the link and label match the project name.","section":"IV / Reference [29]"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take on Temac.\n\nThe genuinely new thing is the pipeline: run a conventional AWGT crawler for 30 minutes, summarize the collected state graph and screenshots into a textual knowledge base, use low-coverage files to infer unfinished tasks, then have a Navigator pick a key state and an Executor run the task along the shortest path. That is a sensible division of labor, and the ablation table shows both the state-transition graph and the coverage report contribute. The paper is also honest about what it excludes. So the architecture is worth taking seriously.\n\nThe soft spots are mostly in the framing of the numbers. The stress-test note is correct: Section IV excludes knowledge-base construction time from the evaluation, and the phases run sequentially. So Temac's actual wall-clock run is longer than 60 minutes, while the baselines are capped at 60. The one-hour budget claim is therefore not a like-for-like comparison. The fact that Temac's coverage curves for three apps are still rising makes the excluded time a plausible alternative explanation for part of the gain. This needs to be measured and reported before the quantitative headline is credible.\n\nSecond, the abstract and RQ1 say \"improvement ranging from 12.5% to 60.3%,\" but Table III does not support that as a per-application range. What the data actually show is an average improvement of 12.5% over the strongest baseline (WebRLED) and 60.3% over the weakest (Crawljax). Those are two different averages, not a range of per-app numbers. The per-application improvements over WebRLED run from roughly 8% to 36%. The claim as written is misleading.\n\nMinor points: three runs without significance tests is standard in this niche but weak; RQ5 uses Temac-noCR, not the full system, and counts console errors as faults, so the 445-fault headline is softer than it looks. Also, reference [29] is labeled \"Open Source Repository of Seeker,\" which looks like a leftover from another paper. The artifact may still be fine, but the reference should be checked.\n\nOverall, the mechanism is plausible and the ablations support the core design. The paper deserves a serious referee, but the authors need to fix the budget accounting and the misreported range before publication. I would give it a conditional accept with those as required revisions.","headline":"Useful multi-agent architecture and mostly consistent empirical evidence, but the one-hour budget claim is not yet like-for-like and the headline improvement range is misreported.","tokens_in":19209,"tokens_out":3259,"would_cite":true,"duration_ms":28711,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Temac couples a fast web GUI crawler with four LLM agents that infer and execute untested functionalities, reporting 12.5–60.3% average coverage gains over state-of-the-art baselines and 445 unique faults in 20 real-world sites.","keywords":["web GUI testing","large language models","multi-agent collaboration","code coverage","state transition graph","LLM agents","coverage-guided testing"],"falsifier":"A concrete check: on the same six applications, run Temac with the Navigator's chosen starting states replaced by randomly selected states from the transition graph, keeping the Summarizer, Reviser, and Executor unchanged. If average code coverage does not drop below the reported 49.7%, then state-transition-guided navigation is not what drives the improvement. A second check would replace the Summarizer's descriptions with generic page titles; if coverage remains unchanged, the semantic summarization step is not load-bearing.","tokens_in":18256,"feed_emoji":"🔍","tokens_out":6510,"duration_ms":53454,"temperature":0.7,"pith_summary":"This paper tries to establish that automated web GUI testing gets substantially better when a fast, non-LLM crawler and a team of LLM agents work in sequence rather than either working alone. After the crawler has explored the application for half an hour, Temac converts what the crawler saw into a concise textual knowledge base, uses an LLM to infer which complex functionalities are still untested, and then sends specialized LLM agents to navigate to and execute those functionalities. The payoff, if the paper is right, is a testing approach that keeps the broad coverage of classical crawlers while gaining the ability to complete long, meaningful action sequences that those crawlers abandon midway. The paper reports average code-coverage improvements of 12.5% to 60.3% over four state-of-the-art baselines on six open-source applications, and 445 unique faults found in 20 real-world web applications.","feed_headline":"Multi-agent LLM testing lifts web GUI code coverage up to 60%","feed_subtitle":"A crawler explores first; LLM agents then target uncovered features, finding 445 real faults.","key_machinery":"The load-bearing mechanism is a four-part textual knowledge base plus four specialized LLM agents that consume it. The knowledge base stores natural-language descriptions of each GUI state (generated by Summarizer from screenshots), a simplified state transition graph with self-loops removed, the 50 source files with the lowest coverage, and application-specific knowledge such as login credentials. Reviser turns this knowledge base into ten targeted testing tasks; Navigator selects the single most relevant state per task and supplies the shortest path from the home state; Executor decouples planning from acting, with a planner choosing the next action and a grounding model (UI-TARS) locating and performing it. The state-transition-guided navigation is the part that injects application-specific domain knowledge into the LLM, addressing the known low success rate of LLM web agents.","core_discovery":"The central discovery is that the state transition graph and coverage reports produced by an automated web GUI tester can be repurposed as domain knowledge that makes LLM agents reliable enough to test uncovered complex functionalities. Temac runs an existing AWGT approach (in the implementation, WebRLED) for a fixed period; then the Summarizer agent translates each GUI state from a screenshot into a one-line natural-language description, the Reviser agent combines those descriptions with the 50 lowest-coverage source files to produce ten concrete testing tasks, the Navigator agent picks the most relevant state for each task and extracts the shortest path from the home state, and the Executor agent—split into a planner and a visual grounding actor—carries out the task along that path. The paper reports that this three-phase pipeline outperforms all four baselines on all six subjects, with average coverage rising from roughly 44% for the strongest baseline to 49.7% overall, and it attributes the gain to the complementarity of breadth (crawler) and depth (LLM agents).","pith_inferences":["An untested corollary is that the same crawl-then-summarize-then-execute loop could transfer to mobile GUI testing, where state graphs are smaller and LLM grounding success rates are higher, so the knowledge-base benefit might be even larger there.","The paper uses low-coverage files as a proxy for missing GUI workflows; a direct test would be whether each Reviser-generated task, after execution, actually raises coverage in the specific files it was derived from, and if not, the coverage-report guidance is heuristic rather than causal.","Since the 445 real-world faults were found with the coverage-report component disabled, the full knowledge base might find additional faults on live sites; this is a measurable prediction, not something the paper itself claims.","The one-hour, roughly $1.7 cost estimate with GPT-4o suggests Temac is cheap enough for continuous testing, so a natural extension is to measure coverage-per-dollar against the baselines rather than coverage alone."],"forward_implications":["Full Temac beats both of its ablations: on average 12.5% higher code coverage than the crawler alone (WebRLED) and 22.9% higher than the LLM agents alone (Temac-LLM).","Both knowledge-base components contribute: removing the state transition graph from task execution lowers average coverage by 3.5%, and removing the coverage report lowers it by 5.4%.","The coverage curves for several applications have not plateaued at the one-hour budget, so the paper expects extended budgets of 6 to 12 hours to reveal further gains.","Because Temac is modular, the same pipeline works with a different AWGT approach, state abstraction, MLLM, or actor model, so the reported gains are not tied to WebRLED or GPT-4o specifically.","Using the no-coverage-report variant on the top 20 real-world sites, Temac found 445 unique faults, with 78.65% of them traced to the applications themselves."],"supporting_citations":[{"why":"WebRLED is the AWGT approach used in the exploration phase and the strongest baseline; Temac extends it to save screenshots, HTML, and action traces.","marker":"[13]"},{"why":"WebEmbed supplies the state abstraction used to construct the state transition graph that later feeds the knowledge base.","marker":"[5]"},{"why":"SeeAct contributes the planner-actor decoupling design that Temac's Executor agent is built on.","marker":"[27]"},{"why":"UI-TARS serves as the actor that locates and performs the actions chosen by the planner.","marker":"[39]"},{"why":"Mind2Web is cited as evidence that LLM web agents have low task-execution success rates, motivating the knowledge-assisted design.","marker":"[24]"},{"why":"WebArena is used in the Executor implementation and cited as another benchmark showing low success rates for LLM agents.","marker":"[25]"},{"why":"Crawljax is a model-based baseline whose coverage Temac must beat.","marker":"[3]"},{"why":"WebExplor is an RL-based baseline and related state-of-the-art approach used for comparison.","marker":"[8]"}],"fun_headline_variants":["LLM agents target web gaps, lifting code coverage 60%","How crawlers plus LLM agents uncovered 445 web faults","Knowledge from testing guides LLM agents to deeper web states","Automated web testing gets a 60% coverage boost with agents","Temac: GUI testing that learns from its own explorer"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The approach rests on whether the Summarizer's natural-language page descriptions and the coverage report are accurate enough for the Reviser to infer genuinely missing functionalities, and for the Navigator to choose a starting state whose shortest path from the home state is a workable route for the Executor.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents target web gaps, lifting code coverage 60%","How crawlers plus LLM agents uncovered 445 web faults","Knowledge from testing guides LLM agents to deeper web states","Automated web testing gets a 60% coverage boost with agents","Temac: GUI testing that learns from its own explorer"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000145,"raw_usage":{"total_tokens":1245,"prompt_tokens":1079,"completion_tokens":166,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":695,"completion_tokens_details":{"reasoning_tokens":79}},"tokens_in":695,"tokens_out":166,"duration_ms":2803,"temperature":1.0,"reasoning_tokens":79,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:03:38.772149+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check: on the same six applications, run Temac with the Navigator's chosen starting states replaced by randomly selected states from the transition graph, keeping the Summarizer, Reviser, and Executor unchanged. If average code coverage does not drop below the reported 49.7%, then state-transition-guided navigation is not what drives the improvement. A second check would replace the Summarizer's descriptions with generic page titles; if coverage remains unchanged, the semantic summarization step is not load-bearing.","supporting_citations":[{"cited_title":"GPT-4V(ision) is a gen- eralist web agent, if grounded,","cited_arxiv_id":null,"evidence_quote":"SeeAct contributes the planner-actor decoupling design that Temac's Executor agent is built on."},{"cited_title":"Mind2Web: Towards a generalist agent for the web,","cited_arxiv_id":null,"evidence_quote":"Mind2Web is cited as evidence that LLM web agents have low task-execution success rates, motivating the knowledge-assisted design."},{"cited_title":"Crawling Ajax-based web applications through dynamic analysis of user interface state changes,","cited_arxiv_id":null,"evidence_quote":"Crawljax is a model-based baseline whose coverage Temac must beat."},{"cited_title":"Automatic web testing using curiosity-driven reinforcement learning,","cited_arxiv_id":null,"evidence_quote":"WebExplor is an RL-based baseline and related state-of-the-art approach used for comparison."}],"review_version":1}