{"id":"658415a2-0160-447b-855d-b19818045efe","arxiv_id":"2504.15474","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A multi-agent LLM framework automates multi-user interactive feature testing on TikTok, achieving 75% task success, 85.9% action similarity, and 26 detected bugs in an industrial trial.","lead":"This paper presents a multi-agent system in which separate LLM-driven agents, one per virtual phone, work together to test multi-user features in TikTok such as live streaming and calls. It reports a 75% success rate on 24 such tasks and says the approach saved developers 87% of the time needed to write test scripts.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 75% success claim rests on an untested assumption that every task description carries explicit 'UserN' labels and starts with User1; real-world descriptions are not shown to satisfy this.","rationale":"The reader's weakest assumption identifies the same load-bearing limitation: the pipeline's input handling is brittle and the evaluation tasks are pre-formatted to satisfy it. This is the strongest concern because it attacks the scope of the central claim directly: if real task descriptions do not reliably contain 'UserN' labels and a User1-first structure, the system cannot even allocate devices or assign subtasks, regardless of LLM capability. The paper's own words in Section III-B1—'we only determine the first action' and 'the initial action would invariably originate from User 1'—flag this as an assumption based on a pilot study, yet no data from that pilot are provided. The RQ3 integration on 3,318 tasks is the natural place to validate the assumption, but the paper does not report how many descriptions were actually parseable by the regex rules. This is an external-validity and correctness-risk issue, not a disagreement with current consensus. I would not reject the paper: the 24-task results may be valid for the structured input format, and the industrial deployment suggests the format may be common internally. However, the central claim should be scoped to explicitly formatted task descriptions, or evidence of coverage on real-world descriptions should be added. Since the reader already recommended CONDITIONAL, this stress-test does not change the verdict.","tokens_in":15448,"tokens_out":6960,"duration_ms":63475,"concrete_test":"Take a random sample (e.g., 300) of the 3,318 industrial task descriptions used in RQ3, apply the published regex and rule-based segmentation from Sections III-A2 and III-B1, and count how many have explicit UserN labels for every participant and start with User1. If the parseable fraction is below roughly 90%, the headline results are conditional on a formatting convention not shown to hold for real inputs. A complementary check would be to re-run the 24-task evaluation on free-form paraphrases that preserve meaning but omit UserN labels; if success drops materially, the method does not handle high-level natural-language descriptions as claimed.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim is that a multi-agent LLM pipeline turns a high-level multi-user task description into a correct multi-device interaction sequence. The first two pipeline stages are not LLM-based: device allocation uses a regex for 'User[1-9]' (Section III-A2), task assignment uses rule-based pattern recognition around 'UserX', and Section III-B1 states that 'the initial action would invariably originate from User 1'. Every evaluation task in Table I is pre-formatted as an explicit <User1: ...; User2: ...> sequence, so these rules are satisfied by construction. The paper gives no evidence that the real-world task descriptions it cites (7,870 descriptions averaging 20.64 words; 3,318 tasks in RQ3) carry that structure. If a description is free-form, omits user labels, or begins with a different user, the allocator and segmenter fail before any LLM reasoning occurs. The 75% success rate and 85.9% action similarity are therefore conditional on an input-formatting assumption that is asserted but not demonstrated; this narrows the scope of the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a multi-agent LLM framework for automating tests of multi-user interactive features in the TikTok app. The approach comprises two main phases: a virtual device farm that allocates devices based on the number of users extracted from a task description, and a multi-agent automation phase where each device is controlled by an LLM-based agent that navigates the GUI and uses explicit actions such as tap, input, back, switch user, and end task. The authors evaluate the approach on 24 curated multi-user interactive tasks, reporting a 75% success rate, 85.9% action similarity, and 87% time savings compared with professional developers. They also report integrating the system into TikTok's Fastbot-based testing platform, where it processed 3,318 task descriptions and helped identify 26 bugs.","tokens_in":15647,"tokens_out":5474,"duration_ms":50723,"significance":"If the reported results withstand scrutiny, the paper addresses a genuine and under-served problem: multi-user interactive features constitute a substantial portion of real-world app testing tasks, and existing automated app testing tools are largely single-user oriented. The industrial deployment and the 26 reported bugs are a meaningful practical signal, and the failure analysis in Section IV-A is honest and identifies concrete causes (timing, cross-app interaction, ambiguous GUI). The paper does not ship a replication package or detailed per-task data, so the evidence is currently preliminary. The central contribution is an empirical system, not a formal derivation, so the main risks are external validity and evaluation rigor rather than internal circularity.","major_comments":[{"comment":"The first two stages of the pipeline are rule-based and depend on the task description containing explicit 'UserN' labels. Device allocation uses a regex pattern 'User[1-9]', task assignment uses pattern recognition around 'UserX', and Section III-B1 states that the initial action would invariably originate from User1. Every task in Table I is pre-formatted as an explicit 'User1: ...; User2: ...' sequence, so the rules are satisfied by construction. The paper does not report how many of the 7,870 real-world task descriptions discussed in Section II-B actually contain explicit user labels, whether they start with User1, or whether they use single-digit user numbers. If descriptions are free-form, omit labels, or begin with a different user, the allocator and segmenter fail before any LLM inference occurs. The RQ1 and RQ3 claims are therefore conditional on an input-formatting assumption that is asserted but not demonstrated. Please report the prevalence of the required format in the real corpus and evaluate on non-preformatted descriptions.","section":"Section III-A2 and III-B1, Table I"},{"comment":"The statement that the approach 'significantly outperforms' the baselines is not supported by any statistical analysis. Table II reports mean success rates and mean action similarities over three runs, with no per-task outcome matrix, no standard deviations or ranges, and no significance test. Because the success metric is binary over 24 tasks, three runs produce very limited evidence for the 75% and 85.9% figures. In addition, Monkey and Humanoid are random exploration tools not designed to execute a given task, and the paper does not compute action similarity for them, so the 30.3% improvement comparison applies only to AdbGPT. Please provide run-level data, confidence intervals or exact per-task results, and appropriate statistical tests, and clearly separate task-driven baselines from random-exploration baselines when interpreting the comparison.","section":"Section IV-A, Table II"},{"comment":"The 87% time-saving claim rests on an underspecified comparison. The paper reports an average of 2.46 minutes for the automated approach and 20.02 minutes for two developers who wrote scripts for the 24 tasks, but it does not report the distribution of per-task times, the number of paired observations used in the Mann-Whitney U test, or whether the automated time includes environment setup, LLM inference retries, and manual verification of the generated actions. Because the developers' manual scripting and the automated approach consume different inputs and workflows, the comparison is meaningful, but the missing variance and setup-cost details make the magnitude of the claimed savings difficult to assess.","section":"Section IV-B, RQ2"},{"comment":"The practicality claim of detecting 26 bugs is not quantified rigorously. The paper states that 3,318 task descriptions were run, that the completion rate was 70.09%, and that failing instances were manually inspected to identify 26 bugs, but it gives no baseline, no inter-rater agreement for bug identification, no reproduction or severity information, and no false-positive count. It is also unclear whether these bugs would have been discovered by the existing Fastbot-based process without the multi-agent automation layer. Please provide a breakdown of the 26 bugs by type, the number of failure cases examined, how bugs were confirmed, and a comparison with the prior testing process.","section":"Section IV-C, RQ3"}],"minor_comments":[{"comment":"The abstract says 'aiding in the detection of 26 multi-user interactive bugs,' while the conclusion says 'improving 26 bug detection'; the latter appears to be a wording error and should be made consistent.","section":"Abstract and Conclusion"},{"comment":"The sentence describing the curated 24 tasks states that each task is structured as '<User1: ...; User2: ...>', but the preceding prevalence analysis of 7,870 descriptions does not clarify whether those original descriptions already followed this format; this is directly related to Major Comment 1 and should be clarified in the text.","section":"Section II-B"},{"comment":"The phrase 'we assign separate LLM instances of each agent' should read 'to each agent'; this is a grammatical typo.","section":"Section III-C"},{"comment":"The paper does not release the 24 task descriptions, ground-truth traces, or the exact prompts used; making these artifacts available would substantially improve the reproducibility of the evaluation.","section":"Section IV-A"},{"comment":"The claim that the tasks were 'freshly identified and labeled by human annotators specifically for this study, mitigating the potential bias for data leakage' is only partially convincing, since a model like GPT-4 may still have knowledge of common TikTok GUI patterns; a more direct check would be to test on task descriptions that are semantically similar but use different wording.","section":"Section IV-A"}],"recommendation":"major_revision","confidential_remarks":"The main risk is external validity: the headline effectiveness and practicality figures are built on a curated task format and a single-app deployment, with no released artifacts and limited statistical reporting. The industrial deployment is a genuine strength, but the RQ3 evidence needs more detail before the 26-bug claim can be taken at face value. I do not see citation-pattern concerns or a mismatch with the journal's scope."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The headline is that this is a genuinely useful industrial application of LLM agents to a real gap: multi-user interactive feature testing. The paper shows a multi-agent pipeline where each user gets an LLM-driven agent on a separate device, with a simple device-allocation and turn-taking protocol. On 24 curated TikTok tasks it hits 75% task success and 85.9% action similarity, and the time-savings figure (87% vs. two developers) is plausible given the scale of the difference. The failure analysis is honest—timing, cross-app, and ambiguous GUI causes—and the integration with Fastbot over 3,318 tasks, with 26 bugs manually identified, is real evidence of practical value even if the bug count is not a systematic benchmark.\n\nThe main soft spot is the input-formatting assumption. The first two pipeline stages are not LLM-based: device allocation uses regex User[1-9], task assignment uses rule-based matching around UserX, and the paper asserts the initial action always comes from User 1. All 24 evaluation tasks are pre-formatted as explicit User1/User2 sequences, so the approach satisfies the assumption by construction. The paper does not show that the 7,870 real-world descriptions (or the 3,318 in RQ3) actually carry that structure. If they do, fine; if they are free-form, the allocator and segmenter fail before any LLM reasoning happens. This needs to be explicit, and ideally validated on the industrial dataset.\n\nOther concerns are smaller. The evaluation is 24 tasks with three runs and means only; no code or data is shipped, so the exact prompt and allocation logic cannot be independently reproduced. The random baselines (Monkey, Humanoid) are not designed for task automation, so their near-zero success is unsurprising; AdbGPT is the only meaningful comparison and it is far below on success rate. The 26 bugs are manually identified from failed runs, which is honest but not an automated detection claim.\n\nThe central claim holds up within the stated scope: if the task description is formatted as UserN-labeled segments, the system works on TikTok and likely on similar social apps. That scope limitation is the main thing to fix. This paper is for researchers and practitioners working on LLM-based UI testing and multi-device test automation. A serious referee should engage with it; I would recommend major revision to address the input-format question, release artifacts, and clarify the evaluation's scope.","headline":"A useful industrial multi-agent LLM system for multi-user app testing, but its success rate depends on an input-format assumption that the paper has not yet validated.","tokens_in":16176,"tokens_out":2757,"would_cite":true,"duration_ms":24610,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A team of LLM agents, each playing a user on its own virtual phone, can automate multi-user interactive feature tests in TikTok from a plain-language task description.","keywords":["multi-agent LLM","multi-user interactive features","Android app testing","virtual device farm","GUI testing","task automation","LLM-driven agents","TikTok"],"falsifier":"Take the same 24 tasks, strip the explicit 'UserN' labels, and re-run device allocation and task assignment on the paraphrased free-form text; if success rate falls well below 75%, the rule-based labeling assumption is confirmed as the load-bearing dependency.","tokens_in":15260,"feed_emoji":"🤖","tokens_out":6228,"duration_ms":50686,"temperature":0.7,"pith_summary":"The paper proposes that the hard part of testing multi-user interactive features—like live streams and voice calls—can be handed to a team of LLM-driven agents, one per virtual device, each impersonating a user. Given a high-level task description, the system allocates devices, splits the task per user, and lets the agents click through the GUI in a coordinated order. On 24 TikTok tasks the approach completes 75% of them and matches 85.9% of the reference action sequence, beating three baselines and saving developers 87% of the time they would spend writing scripts. Integrated into TikTok's production testing flow for one month, it processed 3,318 task descriptions and helped surface 26 real multi-user interaction bugs.","feed_headline":"Multi-agent AI testers pass 75% of TikTok interactive tests","feed_subtitle":"Each virtual user runs its own LLM agent, cutting multi-device test scripting time by 87 percent.","key_machinery":"The load-bearing mechanism is the inter-agent handoff loop. Each LLM agent perceives its device screen as a simplified XML view hierarchy—keeping only resource id, class, clickable, text, and content-desc—and chooses from a five-action vocabulary: tap, input, back, switch user, and end task. When an agent's assigned user step is complete, it emits a [switch][user] signal to hand control to the next device's agent; the end-task signal stops the run. The virtual device farm supplies the required number of emulators, determined by a regex that counts distinct 'UserN' labels in the task description.","core_discovery":"The central claim is that a natural-language multi-user task description can be turned into an executable, synchronized multi-device test without any hand-written script. Each user in the description is mapped to an LLM agent on its own emulated Android device; agents step through the GUI one action at a time, and when one finishes its sub-task it sends a switch-user signal so the next agent acts. The only ordering rule is that User 1 always moves first—the rest of the interleaving emerges from the agents' mutual interaction. The authors report that this design, rather than generic random exploration or single-agent LLM bug replay, is what lets the system reach 75% task success and 85.9% action similarity.","pith_inferences":["If the user-label regex is the only device-counting signal, then the approach's reach depends on task descriptions being written with explicit 'UserN' markers; inferring roles from free-form text is an obvious next step the paper does not address.","The reported failure causes—sub-second timing constraints, cross-app jumps, and ambiguous GUI widgets—suggest that pairing the LLM with a low-latency local model or adding a tiny rule-based fast path for time-critical taps could lift the 75% ceiling.","The [switch] handoff is a generic coordination primitive; the same design could test multiplayer games, collaborative editing, or account-sync scenarios that also require ordered actions across devices."],"forward_implications":["Writing a multi-device automation script drops from about 20 minutes per task to 2.46 minutes with the automated pipeline, an 87% time saving for developers.","The single rule that User 1 acts first, with later order emerging from agent handoffs, removes the need to pre-plan the full interaction sequence in the script.","Because the agents read the same XML accessibility data any Android app exposes, the approach can in principle be pointed at other social apps without changing the agent design.","Running the agents on emulated devices in a virtual farm, rather than physical phones, lowers the cost of multi-device testing and makes it practical to run in CI/CD loops."],"supporting_citations":[{"why":"Supplies the task-driven LLM baseline that the approach outperforms; it uses prompting strategies like in-context learning and chain-of-thought.","marker":"[30]"},{"why":"Supplies the random-gesture baseline; the paper runs it on the same tasks for up to two hours per task.","marker":"[10]"},{"why":"Supplies the neural-network-trained simulation baseline; the paper runs it on the same multi-device setup.","marker":"[31]"},{"why":"Provides the prior load-testing evidence that 395 ARM servers can support the 5,918 virtual devices in the farm.","marker":"[15]"},{"why":"Supplies the LCS-based action-similarity metric used to compute the 85.9% match score.","marker":"[2]"}],"fun_headline_variants":["LLM agents simulate users to auto-test TikTok multi-user features","Virtual device farm with LLM agents tests TikTok multi-user tasks","Multi-agent LLM testing cuts TikTok multi-user test scripting by 87%","No scripts: LLM agents turn task descriptions into TikTok multi-user tests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The whole pipeline assumes every task description explicitly labels the users as 'User 1, User 2, ...' and that the first action always belongs to User 1, yet the paper gives no evidence that real-world descriptions, which average about 20 words, reliably follow that format.","fun_headline_variants_meta":{"raw":{"variants":["LLM agents simulate users to auto-test TikTok multi-user features","Virtual device farm with LLM agents tests TikTok multi-user tasks","Multi-agent LLM testing cuts TikTok multi-user test scripting by 87%","No scripts: LLM agents turn task descriptions into TikTok multi-user tests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000714,"raw_usage":{"total_tokens":3182,"prompt_tokens":890,"completion_tokens":2292,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":2216}},"tokens_in":506,"tokens_out":2292,"duration_ms":15678,"temperature":1.0,"reasoning_tokens":2216,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:25:22.687851+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the same 24 tasks, strip the explicit 'UserN' labels, and re-run device allocation and task assignment on the paraphrased free-form text; if success rate falls well below 75%, the rule-based labeling assumption is confirmed as the load-bearing dependency.","supporting_citations":[{"cited_title":"Ui/application exerciser monkey,","cited_arxiv_id":null,"evidence_quote":"Supplies the random-gesture baseline; the paper runs it on the same tasks for up to two hours per task."},{"cited_title":"Humanoid: A deep learning- based approach to automated black-box android app testing,","cited_arxiv_id":null,"evidence_quote":"Supplies the neural-network-trained simulation baseline; the paper runs it on the same multi-device setup."},{"cited_title":"Virtual device farms for mobile app testing at scale: A pursuit for fidelity, efficiency, and accessibility,","cited_arxiv_id":null,"evidence_quote":"Provides the prior load-testing evidence that 395 ARM servers can support the 5,918 virtual devices in the farm."},{"cited_title":"Gifdroid: Automated replay of visual bug reports for android apps,","cited_arxiv_id":null,"evidence_quote":"Supplies the LCS-based action-similarity metric used to compute the 85.9% match score."}],"review_version":1}