{"id":"f8576b86-895e-4e62-bf95-efac607e48d9","arxiv_id":"2504.20896","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"An LLM-based agent with screen refinement, structured prompting, back-button recovery, and LoRA distillation executes natural-language Android test cases, reporting 73% success across 390 cases on 10 apps.","lead":"LELANTE is a framework that uses a large language model to read a natural-language Android test case and perform the clicks and text entries on the device, without a pre-written test script. Across 390 test cases on 10 apps it reports a 73% execution success rate, but that headline number depends on human-rewritten test descriptions, human verification of the final screen, and a thinly documented comparison baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"73% success rate depends on human-rewritten test descriptions (65% on originals), making the headline claim conditional.","rationale":"The reader's weakest-assumption analysis identifies the same load-bearing concern: the 73% headline rate is contingent on human rewriting of test descriptions per Appendix A, and Table 2 shows the unmodified rate is 65%. This is the most direct threat to the central claim because the paper's framing promises execution of natural-language descriptions 'without requiring pre-written scripts' and without mentioning the human preprocessing step. The paper itself flags this limitation in Section 6, which supports treating it as a real concern rather than a manufactured one. I also considered the human-validation oracle in Section 3.2 and the weak AutoDroid baseline, but those are secondary: manual validation is acknowledged as a limitation and does not change the execution-success metric as much as the description-rewriting step, and the AutoDroid comparison is not the core claim. The distilled-model gap (LLaMA fine-tuned: 65% vs GPT-4o: 73%) is noteworthy but the abstract's 73% explicitly refers to LELANTE with a foundational LLM, so it is not inconsistent with the paper's own numbers. Given that the concern is concrete, empirically quantified in the paper, and addressable by releasing artifacts and rerunning on unmodified descriptions, the appropriate verdict remains CONDITIONAL. I recommend no change to the reader's verdict.","tokens_in":10121,"tokens_out":3114,"duration_ms":36314,"concrete_test":"Release the paired original and human-modified test-case descriptions for all 390 cases, and additionally run LELANTE on a held-out set of natural-language test cases written by testers who have not seen Appendix A. Compare success rates on unmodified descriptions against the reported 73% using McNemar's test on the paired original/modified results. If unmodified descriptions underperform by more than 8 percentage points or the difference is not statistically significant, the headline must be scoped as applying only after human de-ambiguization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is the 73% test execution success rate on 390 natural-language test cases, reported in the abstract and Section 1 as evidence that LELANTE can execute test case descriptions 'directly' without manual script writing. However, Table 2 shows this rate is achieved only after human testers rewrite the descriptions according to the guidelines in Appendix A, which require adding explicit final confirmation steps, specifying a single method when multiple exist, and providing UI-locator hints. The original, unmodified test cases reach only 65%. Section 6 explicitly concedes that 'reliance on manual test case modifications suggests limitations in natural language understanding.' Thus the load-bearing assumption is that the input test descriptions have already been de-ambiguated by a human following a style guide. If the claim is read as 'LELANTE executes natural-language test case descriptions as written', that assumption fails: the real success rate on as-written descriptions is 65%, a statistically and practically meaningful 8-point drop. This does not invalidate the framework, but it changes the scope of the claim from fully automated execution of arbitrary descriptions to execution of carefully formatted descriptions, and the abstract's omission of this precondition is misleading. A secondary, related weakness is that the headline 73% uses GPT-4o, while the distilled LLaMA-3.1 8B fine-tuned model achieves only 65%; the scalable version does not reproduce the headline result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces LELANTE, a framework that uses large language models to execute Android test cases from natural-language descriptions without pre-written scripts. The system refines GUI XML into a compact textual representation, prompts an LLM with a structured chain-of-thought template, executes selected actions via Appium, and supports backtracking-based error recovery. It also distills GPT-4o traces into a fine-tuned LLaMA-3.1 8B model for cheaper inference. The evaluation uses 390 test cases across 10 Android apps and reports a 73% test execution success rate, 78% error recovery, and 11.8 seconds per step for GPT-4o, compared against AutoDroid; Table 2 further reports that success drops to 65% on original, unmodified test-case descriptions. The paper concludes that LLM-driven execution is promising but notes limitations in natural-language understanding, lack of autonomous outcome verification, and slow backtracking.","tokens_in":10488,"tokens_out":3008,"duration_ms":33965,"significance":"If the reported results are robust, LELANTE would be a useful step toward reducing manual effort in Android UI test execution: it compares against an existing LLM-based baseline (AutoDroid), provides an explicit error-recovery mechanism, and evaluates a distilled open-source model, which addresses a realistic deployment concern. The authors also deserve credit for disclosing in Section 6 and Table 2 that performance depends on human rewriting of descriptions, and for publishing the Appendix A guidelines that make the evaluated input format concrete. However, the headline success rate is conditional on those rewritten descriptions, the statistical evidence is thin (no error bars, repeated runs, or per-app breakdowns), and the success-validation protocol is underspecified. These issues do not invalidate the framework, but they do change the scope of the central claim and need to be addressed before the paper can be accepted.","major_comments":[{"comment":"The abstract and Section 1 report a 73% test execution success rate without qualification, but Table 2 and Section 6 show that this rate applies only after human testers rewrite descriptions according to the Appendix A guidelines, while original unmodified test cases achieve 65%. Because the paper's central claim is that LELANTE executes natural-language test descriptions as written, this is a load-bearing qualification: the as-written success rate is 8 percentage points lower, and the abstract's unqualified statement materially overstates the result. Please report both figures in the abstract, state how many of the 390 test cases were rewritten and by whom, and frame the contribution as executing descriptions formatted according to the stated guidelines.","section":"Abstract, Section 1, Table 2, Section 6"},{"comment":"Table 1 reports success, recovery, and timing comparisons (e.g., LELANTE vs. AutoDroid: 73% vs. 71.3% success) without confidence intervals, repeated runs, or per-application breakdowns. The 1.7-percentage-point success advantage over AutoDroid is well within the range that could arise from test-set or run-to-run variation, and Figure 3 shows that success is strongly dependent on test-case length. Without per-app results, run counts, and a statistical test or uncertainty quantification, RQ1 and the comparative claim against AutoDroid are not established at the claimed precision. Please add per-app and per-length results and report uncertainty or significance.","section":"Table 1, Section 4"},{"comment":"The paper states that a human tester manually validates whether the final application state matches the intended outcome, and defines error recovery as the rate at which LELANTE 'autonomously detects mistakes, navigates backward, and chooses the correct alternative step.' The manuscript does not specify the validation protocol, who performs it, or whether the 'correct alternative step' judgment is independently verified. Since the two headline metrics depend on this assessment, please provide a precise operational criterion for success and for correct recovery, and report inter-rater agreement or at least a detailed rubric.","section":"Section 3.3, Section 4"}],"minor_comments":[{"comment":"The abstract contains a grammatical issue: 'iteratively generate action plans' should be 'iteratively generates action plans.'","section":"Abstract"},{"comment":"The prompt template contains apparent spacing artifacts (e.g., 'goa l_action_ plan', 'p a s t _ a c t i o n s _ s u m m a r y'). This is likely a typesetting issue, but the prompt should be reproduced cleanly so readers can copy it and verify the evaluation.","section":"Appendix B"},{"comment":"The right-hand histogram of success rate versus test-case length would benefit from explicit axis labels and error bars or counts per bin, since the text draws a monotonicity conclusion from it.","section":"Figure 3"},{"comment":"The definition of 'test case length' as 'minimum number of test steps needed to complete a test case' is ambiguous: it is unclear whether this is the human-annotated minimal length or the length of the executed path, and the two can differ substantially when backtracking occurs.","section":"Section 4"},{"comment":"The distillation section says the fine-tuning data came from 30 applications not in the final evaluation set, but the paper never states whether the 10 evaluation applications overlap with those 30; please clarify the relationship between the training and evaluation app sets.","section":"Section 3.4, Section 4"}],"recommendation":"major_revision","confidential_remarks":"The paper's core idea is plausible and the evaluation is a reasonable first step, but the abstract's unqualified 73% claim in light of Table 2's 65% figure is the kind of issue that reviewers and readers will treat as an honesty problem even if unintentional. The missing statistical grounding and validation protocol are fixable with additional analysis or a more careful framing. I would support resubmission after the authors revise the claims, add uncertainty quantification or per-app data, and clarify the success-judgment process."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a sensible systems paper with a real evaluation, but the headline 73% success rate does not describe the system as it would run on ordinary test descriptions. On the original, unmodified descriptions the success rate is 65%; the 73% only appears after humans rewrite descriptions following the Appendix A style guide. The abstract and introduction omit that precondition, and that is the main thing to know before citing the numbers.\n\nCredit where due. The framework is clearly described and the combination is new: screen XML refinement, structured chain-of-thought prompting, explicit back-button error recovery, and LoRA distillation of GPT-4o traces into LLaMA-3.1 8B. The evaluation covers 390 test cases across 10 apps, which is reasonable scale for this area, and the 78% error-recovery rate is a genuinely useful result. I also appreciate that Section 6 admits the manual-modification dependence and the need for human validation of final states. That honesty matters.\n\nSoft spots, in roughly increasing severity. First, the headline claim: 'executes natural-language test descriptions' is load-bearing. The system works well on descriptions that have been de-ambiguated by a human; on as-written descriptions it is 8 points worse. That is not fatal—the framework is still useful—but it changes the claim from fully automated to assisted execution. Second, the distilled model, which is the scalability story, achieves 65%—the same as the unmodified-input rate. So the practical cheap deployment does not reproduce the headline advantage. Third, the evaluation lacks error bars, repeated runs, per-app breakdowns, and any detail on how AutoDroid was configured, so the claimed 1.7% edge over AutoDroid is not substantiated. Fourth, no code or data is provided, which makes replication impossible for a paper that asks us to trust a measured success rate. The human validation of outcomes is a limitation the authors themselves flag, so I won't belabor it.\n\nThe paper deserves a serious referee. It is a coherent, well-written systems contribution to LLM-based mobile testing, with honest self-assessment. But it needs revision: report both success rates in the abstract, add statistical detail, and release the artifact. If those are addressed, I'd cite it; as is, I'd be cautious.","headline":"A useful LLM-based Android test execution framework, but the 73% headline is conditional on human-rewritten test descriptions—65% on original wording—and the distilled model matches only that lower bar.","tokens_in":10936,"tokens_out":1984,"would_cite":false,"duration_ms":20894,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An LLM-driven framework executes natural-language Android test case descriptions directly as UI actions, reaching a 73% test execution success rate across 390 cases from 10 apps.","keywords":["Android testing","large language models","GUI refinement","test case execution","error recovery","model distillation","chain-of-thought prompting","natural language test description"],"falsifier":"Run LELANTE on the original, unmodified 390 test descriptions without human rewriting; if the success rate reaches 73% rather than staying near the reported 65%, the ambiguity-resolution premise is not load-bearing, and if it stays at 65%, the headline result depends on the preprocessing step.","tokens_in":9917,"feed_emoji":"🤖","tokens_out":8299,"duration_ms":80872,"temperature":0.7,"pith_summary":"The paper aims to show that a large language model can turn a natural-language Android test case description into a working sequence of UI actions without any pre-written test script. The proposed framework, LELANTE, refines each screen into a compact list of interactive elements, prompts the model to reason about the goal and past actions, executes the chosen action, and backtracks with an explicit Back action when it detects an error. Across 390 test cases from 10 popular Android applications, the paper reports a 73% execution success rate, 78% error recovery, and 11.8 seconds per step. If those results hold, mobile test automation could become substantially less dependent on manually maintained scripts and more adaptable as app interfaces change.","feed_headline":"LLM tool runs Android tests straight from plain English, 73% success","feed_subtitle":"No hand-written scripts: it plans UI actions, backtracks on mistakes, and ran 390 cases across 10 Android apps.","key_machinery":"The load-bearing mechanism is a chain-of-thought prompting loop over a refined screen representation. Each screen's raw Android XML is compressed into a list of interactive elements with unique numeric IDs, plus explicit Back and scroll actions, and an icon classifier converts icons into words. The structured prompt forces the model to output a goal action plan, a summary of past actions, a no-further-action boolean, the selected element ID, and any text input, with the special ID -1 signaling completion. This loop carries the entire argument: it maps the natural-language goal to concrete UI actions, allows the model to declare the goal finished, and supports backtracking when an action repeats or fails.","core_discovery":"The central claim is that a large language model, guided by a distilled textual representation of the Android screen and a structured prompt, can convert a natural-language test case description into a usable action sequence without any pre-written test script. LELANTE performs this loop iteratively: it refines each screen's XML into interactive elements with unique IDs, asks the model to reason step by step about the goal and action history, executes the selected action on the device, and uses an explicitly added Back action to undo mistakes. In the evaluation, 73% of the 390 test cases were executed completely, with a 78% error recovery rate and an average of 11.8 seconds per step. The paper presents this as evidence that LLM-driven execution can replace brittle, manually maintained UI test scripts.","pith_inferences":["Going beyond the paper, the 73% headline is an upper bound achieved after humans rewrite ambiguous descriptions; executing the original descriptions as written yields 65%, so a fair comparison for as-written automation should use the 65% figure.","A natural extension would be to measure success as a function of description clarity, using independent QA authors and no rewriting guidelines; the paper's own table predicts that clearer descriptions close most of the gap.","The same screen-refinement and structured-prompt loop is not Android-specific, so porting it to iOS or Web UI automation is a plausible test of the mechanism's generality.","The framework's inability to self-verify outcomes suggests that coupling it to an automatic assertion or screenshot-comparison step would convert 'executed completely' into true pass-or-fail verdicts; this is speculative but directly addresses the stated limitation."],"forward_implications":["If the measured success transfers beyond the 10 evaluation apps, QA teams could execute existing natural-language test suites directly, removing the need to write or maintain UI automation scripts as apps change.","The explicit Back action and reasoning loop make error recovery a first-class behavior: the reported 78% recovery rate means most wrong touches are detected and corrected automatically.","The distilled 8-billion-parameter model retains most of the success (65% vs 73%) at similar per-step speed, so low-cost local execution is plausible rather than requiring paid foundational-model APIs.","Longer test cases succeed less often, so the practical benefit is strongest for short-to-medium scenarios; the remaining gap is in ambiguity handling and error propagation.","A human still must verify the final app state because LELANTE cannot autonomously check whether the expected result was achieved."],"supporting_citations":[{"why":"Supplies the foundational LLM that generated the teacher action data used in distillation.","marker":"[1]"},{"why":"Provides the icon classifier that turns icon images into text in the refined screen representation.","marker":"[5]"},{"why":"Defines knowledge distillation, the technique used to shrink the teacher model into a cheaper student model.","marker":"[7]"},{"why":"Supplies the parameter-efficient fine-tuning method used to train the distilled student model.","marker":"[11]"},{"why":"Provides the GUI automation engine that executes the model-selected actions on Android.","marker":"[26]"},{"why":"Supplies the open-source base model that is fine-tuned for the low-cost variant.","marker":"[27]"},{"why":"Motivates the chain-of-thought structure of the prompt, the central mechanism for action planning.","marker":"[28]"},{"why":"Gives the prior LLM-based Android automation system against which success, recovery, and speed are compared.","marker":"[29]"}],"fun_headline_variants":["LLM executes Android tests from plain English at 73% success","No scripts needed: LLM automates Android testing, 73% success","LELANTE: AI runs Android tests from natural language descriptions","LLM-powered Android testing: 73% of 390 tests executed via narration","Automated Android testing from text: LELANTE achieves 73%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported 73% success rate applies to test cases whose descriptions were manually rewritten to remove ambiguity; if the intended claim is that LELANTE executes descriptions exactly as originally written, the measured success rate is 65%.","fun_headline_variants_meta":{"raw":{"variants":["LLM executes Android tests from plain English at 73% success","No scripts needed: LLM automates Android testing, 73% success","LELANTE: AI runs Android tests from natural language descriptions","LLM-powered Android testing: 73% of 390 tests executed via narration","Automated Android testing from text: LELANTE achieves 73%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000208,"raw_usage":{"total_tokens":1394,"prompt_tokens":923,"completion_tokens":471,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":539,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":539,"tokens_out":471,"duration_ms":4496,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:16:01.288211+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LELANTE on the original, unmodified 390 test descriptions without human rewriting; if the success rate reaches 73% rather than staying near the reported 65%, the ambiguity-resolution premise is not load-bearing, and if it stays at 65%, the headline result depends on the preprocessing step.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the GUI automation engine that executes the model-selected actions on Android."},{"cited_title":"In 2019 Third International conference on I-SMAC (IoT in Social, Mobile, Analytics and Cloud)(I-SMAC)","cited_arxiv_id":null,"evidence_quote":"Supplies the open-source base model that is fine-tuned for the low-cost variant."}],"review_version":1}