{"id":"ddead233-8565-42d9-ae65-5e19b337550b","arxiv_id":"2411.17933","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"LLMigrate uses a multimodal LLM to abstract a source app test into natural language and then dynamically explores the target app to rebuild the test, reaching a 97.5% success rate over 120 transfers.","lead":"This paper introduces LLMigrate, a system that uses a large language model to automatically copy a UI test from one Android app and adapt it to another app with similar functions. In tests across five app categories it transferred 97.5% of cases successfully and cut the manual effort to write the tests by about 91%, improving on prior methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 97.5% success rate is reported after tuning three parameters on the same 120-transfer evaluation set; without a validation split or sensitivity analysis, the headline numbers may be optimistic.","rationale":"The paper's headline claim rests on an evaluation of 120 transfers from CraftDroid. Section 4.1 explicitly discloses that the three adjustable parameters (maximum wrong tries, majority-vote query count n, and inclusion threshold m) were empirically observed as best-performing on the same evaluation set. This is a selection-on-test-set problem: all reported metrics, including the 97.5% success rate, 91.1% reduction, and the 9.1%/38.2% gains over the best prior technique, are computed after tuning to that benchmark. The reader's weakest assumption about LLM generalization is a legitimate external-validity concern, but RQ5 directly addresses it and the observed drop to 93% on 30 transfers is modest and within sampling error. The parameter-tuning issue, by contrast, is internal and entirely unaddressed; it affects the absolute and comparative headline numbers directly. A concrete check with an untuned configuration or a validation split would settle whether the tuned parameters are load-bearing. Since the reader's verdict is already CONDITIONAL and explicitly mentions moving parameter tuning to a validation split, the stress-test does not change the verdict.","tokens_in":24675,"tokens_out":7670,"duration_ms":67497,"concrete_test":"Re-run the full 120-transfer evaluation with the three parameters fixed before seeing results (e.g., n=1, m=1, max wrong tries=1, or values chosen on a 20% validation split). If aggregate success rate or reduction changes by more than 5 percentage points relative to the tuned results, the parameter tuning is load-bearing and the headline numbers should be reported as conditional on a validation procedure.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 states: 'We empirically observed the best-performing values for all these parameters and set them to 3, 3, and 2, respectively, in our evaluation.' The three adjustable parameters (maximum wrong tries at the same step, majority-vote query count n, and inclusion threshold m) were tuned on the same 120 transfers used to report the headline metrics. No validation split, cross-validation, or sensitivity analysis is reported. Therefore the 97.5% success rate, 91.1% reduction, and the claimed 9.1% and 38.2% improvements over the best prior technique are obtained with a configuration selected to maximize performance on this benchmark, rather than the performance of a fixed, a-priori method. This is an internal-validity threat to the central claim: the comparison with CraftDroid and TREADROID is effectively between a benchmark-tuned configuration and fixed prior techniques. The concern would land if a reasonable untuned or cross-validated configuration yields materially lower metrics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents LLMigrate, a technique that uses a multimodal LLM (GPT-4o) to transfer usage-based UI tests between Android apps in the same domain without requiring source code. LLMigrate first executes the source test on the source app, augments each event with widget attributes, and asks the LLM to produce a natural-language 'Abstract Source Test.' It then dynamically explores the target app, using screenshots and cleaned XML hierarchies as the current state, and repeatedly asks the LLM to select the next event, with majority voting, dead-end tracking, backtracking, and repair prompts. The evaluation on 120 transfers from 19 functional CraftDroid apps reports a 97.5% successful transfer rate, a 91.1% manual-effort reduction, and improvements over the best prior technique of 9.1% in success rate and 38.2% in effort reduction. Additional studies target modern popular apps (RQ4) and apps released after GPT-4o's knowledge cutoff (RQ5).","tokens_in":24893,"tokens_out":8812,"duration_ms":72167,"significance":"If the reported results hold, LLMigrate would be a substantial advance over similarity-based test transfer: it handles differing input fields, transfers oracle events, requires no source code, and achieves high success and effort reduction in complex domains where prior work shows negative reduction. The paper's public artifacts, its manually evaluated 120-transfer study on the standard CraftDroid benchmark, and its explicit RQ5 attempt to evaluate on apps released after the model's knowledge cutoff are genuine strengths. However, the headline numbers are produced by a configuration tuned on the same benchmark used for evaluation, and several supporting numbers in RQ4 and RQ5 are internally inconsistent or not traceable to individual transfers. The central idea is promising and within scope for the venue, but the evidence as reported is not yet as strong as the abstract claims.","major_comments":[{"comment":"The three main parameters (maximum wrong tries at the same step, majority-voting run count n, and majority-voting inclusion threshold m) are tuned directly on the same 120-transfer evaluation that produces the headline 97.5% success rate, 91.1% reduction, and the 9.1%/38.2% improvements over TREADROID. Section 4.1 states that the authors 'empirically observed the best-performing values' and set them to 3, 3, and 2, but no validation split or sensitivity analysis is reported. Since prior techniques are evaluated with fixed configurations, the comparison is effectively between a benchmark-tuned configuration and fixed baselines. Please add a hold-out validation split, a sensitivity analysis over the parameter grid, or the metrics of a reasonable untuned configuration; without this, the central numbers may be optimistic.","section":"Section 4.1"},{"comment":"The text states that LLMigrate achieved 'an average success rate of 98%' and 'an average reduction score of 97%' on the new popular apps. Averaging the ten rows in Table 4 gives 95% success (nine 100% entries and one 50% entry) and approximately 83% reduction, not 98% and 97%. Please clarify the weighting or averaging procedure, or correct the stated values and qualify the 'strong performance' characterization accordingly.","section":"Section 4.5, Table 4"},{"comment":"The RQ5 setup says the study involves 30 total test transfers, but Table 6 reports one row per category/test with a single value per metric. With five new categories and three apps each, it is not clear whether the rows aggregate 15 transfers, 30 transfers, or something else; as written, the 93.3% success rate is not traceable to individual transfers. Please report per-app or per-transfer results and reconcile the '30 total test transfers' statement with the displayed aggregation.","section":"Section 4.6, Tables 5 and 6"}],"minor_comments":[{"comment":"The abstract reports 97.5% success and 91.1% reduction, while the conclusion says '97% of tests' and 'more than 90%' of manual work; please use consistent figures.","section":"Abstract and Section 8"},{"comment":"Phrases such as 'outperform ... by almost 10%' and '44% lower' should be expressed in percentage points (9.1 points and 44.5 points, respectively) to avoid ambiguity.","section":"Section 4.3 and Figure 7"},{"comment":"The parenthetical differences in Table 3 are not explained and do not always correspond to the difference from the same baseline; for example, the total oracle recall shows 5.7%, while LLMigrate (94.71) minus TREADROID (87.44) is 7.27. Define the comparison basis in the caption or remove the parentheticals.","section":"Table 3"},{"comment":"TEMdroid's published averages are from a non-identical dataset and the artifacts are unavailable; the paper acknowledges this, but presenting those averages in the same comparison table risks over-reading. Consider using a clearly separated row or footnote that states the 19/23 overlap and the lack of artifact-based re-evaluation.","section":"Section 4.2 and Table 3"},{"comment":"The claim that apps are 'unseen by LLMs' is operationalized only by release date after the knowledge cutoff; it would be useful to state explicitly that this is a proxy and that the model may still have seen similar app types or patterns.","section":"Section 4.6"}],"recommendation":"major_revision","confidential_remarks":"I see a strong, publishable idea and a substantial evaluation effort, but the in-sample parameter tuning is a real internal-validity concern for the headline numbers, and the RQ4/RQ5 reporting issues need correction. I would ask the authors for a sensitivity analysis or hold-out validation, a corrected RQ4 average, and a per-transfer breakdown for RQ5 before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"LLMigrate is a real step forward for test transfer, and the paper is worth engaging seriously. The new piece is the two-phase design: abstract the source test into natural language, then let a multimodal LLM dynamically rebuild the test on the target app using screenshots and cleaned XML. That is a genuinely different mechanism from the similarity-based and classification-based tools in this line of work, and the evaluation is more thorough than most in the subfield: 120 transfers on CraftDroid, a follow-up on current apps, and an unseen-app study using apps released after GPT-4o's cutoff with manually written tests. The artifacts are public, and the paper is candid about what it could not run (TEMdroid) and about LLM nondeterminism.\n\nThe soft spot is the one the stress-test flags, and it is real. Section 4.1 says the three parameters (max wrong tries, majority-vote n and m) were chosen by 'empirically observed best-performing values' on the same 120 transfers used to report the headline numbers. So the 97.5% success rate and 91.1% reduction are in-sample, and the comparison against CraftDroid/TREADROID is effectively a tuned system versus fixed baselines. That is an internal-validity threat to the abstract's claims. It is not fatal: the transfer loop does not use ground-truth tests during migration, and RQ5 gives a genuinely out-of-sample check with 93% success and 88% reduction. But the paper should either move tuning to a validation split, report sensitivity, or soften the headline numbers. The TEMdroid comparison also leans on published averages over a slightly different app set; the authors acknowledge this, and it is a minor issue. Finally, they report no variance or repeated-run statistics, which is a known weakness with LLM agents; temperature zero and majority voting help but do not quantify robustness.\n\nWho is this for: anyone working on mobile UI testing or LLM agents for software engineering. It deserves a serious referee; the novelty and the public artifacts justify the time. I would ask for a validation split or sensitivity analysis and a discussion of variance before accepting.","headline":"A genuinely new LLM-based test-transfer pipeline with public artifacts and a thoughtful unseen-app study, but the benchmark-tuned parameters make the headline 97.5% in-sample; a referee should require a validation split or sensitivity analysis.","tokens_in":25420,"tokens_out":2609,"would_cite":true,"duration_ms":24050,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A method called LLMigrate transfers UI tests across Android apps at a 97.5% success rate on the CraftDroid benchmark.","keywords":["mobile UI testing","test transfer","test migration","large language models","multimodal LLM","Android apps","GUI test automation","oracle generation"],"falsifier":"Run LLMigrate on a held-out set of same-domain app pairs released after GPT-4o's knowledge cutoff, with ground-truth tests authored independently, and count successful transfers per flow; if the overall success rate falls clearly below the claimed range on a statistically meaningful sample, or if any reproducible flow loops forever without transferring all oracles, the central claim fails. A sharper check is to choose flows in which the target app inserts an extra confirmation step, such as a password-confirmation dialog or a terms checkbox, that the source flow does not contain, and measure how often the loop still reaches the final oracle.","tokens_in":24493,"feed_emoji":"📱","tokens_out":9488,"duration_ms":82705,"temperature":0.7,"pith_summary":"This paper claims that a multimodal large language model can transfer an existing end-to-end UI test from one Android app to another app with the same functionality, without access to the source code. The method first rewrites the source test as a short natural-language description, then lets an LLM drive exploration of the target app by choosing the next event from a screenshot and a simplified layout hierarchy. On 120 transfers drawn from the CraftDroid benchmark, the paper reports a 97.5% success rate and a 91.1% reduction in manual test-writing effort, improving on the best prior technique by 9.1 and 38.2 percentage points. If this holds beyond the benchmark, testers could stop writing routine UI tests from scratch for every app in a domain.","feed_headline":"LLMigrate transfers Android UI tests at 97.5% success","feed_subtitle":"Reuse one app's test to build the same flow in another app, cutting manual test-writing effort by about 91%.","key_machinery":"The load-bearing object is the two-phase LLMigrate pipeline: Source Abstraction produces the Abstract Source Test, a one-paragraph natural-language description of what the source test does and which step completes it, and Test Migration runs a loop in which a Screen Analyzer sub-agent interprets a screenshot plus cleaned XML layout into a widget report and an event-generation sub-agent picks the best next event from that report, the abstract test, and the already-performed events. Reliability mechanisms are temperature-zero sampling with majority voting (three queries, key threshold two), a repair prompt that feeds the last exception back to the model with chain-of-thought instructions, and an Explorer that records dead-end events, backtracks by restarting the app, and falls back when repeated guesses fail. The loop ends when all source oracles have been transferred or after three times the source event count.","core_discovery":"On its own terms, LLMigrate's discovery is that test transfer can be reframed as goal-directed exploration instead of widget-to-widget matching. The source test is executed once and summarized by an LLM into a natural-language Abstract Source Test; the migration phase then repeatedly captures the target app's current screen, translates the screenshot plus cleaned XML hierarchy into a widget report, selects the next GUI, system, or oracle event, and repairs invalid events with exception-feedback prompts and backtracking. The paper reports total average precision of 98.39% for GUI events and 94.71% for oracle events, total average recall of 99.53% and 94.71%, a 97.5% successful transfer rate, and a 91.1% reduction in manual effort across 120 transfers, with a 93.3% success rate on newly released apps outside GPT-4o's training data.","pith_inferences":["Beyond the paper, the natural-language Abstract Source Test could be treated as a reusable asset: once a functionality is described once, that description might serve any number of target apps, turning one-to-one test transfer into specification-based test generation.","Because the loop is guided by a goal rather than by source-app widget mappings, the same machinery could transfer shared sub-flows such as login, search, or form-fill across apps in different categories.","A direct user study measuring developer time to accept or edit an LLMigrate-produced test would test whether the 91.1% Levenshtein-based reduction metric translates into actual time savings; the paper does not present that study.","The shopping-category oracle gap suggests that pairing the multimodal screen analysis with stronger visual verification, such as icon classification or optical character recognition, could close the remaining transfer failures."],"forward_implications":["A developer with one tested app in a domain can obtain usage-based tests for other Android apps in that domain without seeing their source code.","The transfer is fast and cheap enough for routine use: an average of 247 seconds and roughly USD 0.70 per transferred test in GPT-4o token costs.","Complex flows that defeated similarity-based transfer show the largest gains: in the shopping category, GUI-event precision rises from about 44–45% for prior techniques to 100% for LLMigrate.","Newer apps not present in the LLM's training data still transfer at a 93.3% average success rate, indicating the method is not merely replaying memorized apps.","The method handles input-field differences by generating plausible values for fields absent from the source test and skipping fields not used by the target app."],"supporting_citations":[{"why":"Provides the CraftDroid benchmark dataset (subject apps, tests, and 120 transfer cases) and the original semantic-mapping transfer technique that LLMigrate is compared against.","marker":"[59]"},{"why":"Is the best-performing prior technique on the benchmark; LLMigrate reports a 9.1% higher success rate and 38.2% higher effort reduction against it.","marker":"[66]"},{"why":"Defines the reduction and successful-transfer metrics used to measure manual-effort savings and test usefulness.","marker":"[102]"},{"why":"Is the TEMdroid baseline whose category averages are used for comparison because its transfer artifacts were unavailable to the authors.","marker":"[98]"},{"why":"Is AppFlow, the earlier machine-learning test-transfer approach whose category dependence motivates the shift to LLM-based semantic understanding.","marker":"[44]"},{"why":"Supplies chain-of-thought prompting, which the repair event prompt uses to improve the LLM's recovery from invalid events.","marker":"[91]"},{"why":"Identifies GPT-4o, the off-the-shelf multimodal LLM that performs all screen analysis and event selection in the evaluation.","marker":"[79]"},{"why":"Is the Appium framework used to execute source-test events and to extract and clean UI layout hierarchies in both phases.","marker":"[33]"}],"fun_headline_variants":["LLMigrate: 97.5% success reusing Android UI tests across apps","LLM-powered test transfer: 97.5% success, 91% less effort","Reuse one app's tests for another: LLMigrate hits 97.5%","Android UI test migration via LLM: 97.5% success rate","LLMigrate automates Android test transfer at 97.5% success"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a screenshot plus a simplified listing of the on-screen elements gives the language model enough information to choose the correct next action on every screen of an arbitrary target app, even when the model has never seen that app; if there is a class of apps or flows where that inference fails, the reported success rate will not generalize.","fun_headline_variants_meta":{"raw":{"variants":["LLMigrate: 97.5% success reusing Android UI tests across apps","LLM-powered test transfer: 97.5% success, 91% less effort","Reuse one app's tests for another: LLMigrate hits 97.5%","Android UI test migration via LLM: 97.5% success rate","LLMigrate automates Android test transfer at 97.5% success"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000788,"raw_usage":{"total_tokens":3470,"prompt_tokens":936,"completion_tokens":2534,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":552,"completion_tokens_details":{"reasoning_tokens":2425}},"tokens_in":552,"tokens_out":2534,"duration_ms":15758,"temperature":1.0,"reasoning_tokens":2425,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T11:41:36.110192+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run LLMigrate on a held-out set of same-domain app pairs released after GPT-4o's knowledge cutoff, with ground-truth tests authored independently, and count successful transfers per flow; if the overall success rate falls clearly below the claimed range on a statistically meaningful sample, or if any reproducible flow loops forever without transferring all oracles, the central claim fails. A sharper check is to choose flows in which the target app inserts an extra confirmation step, such as a password-confirmation dialog or a terms checkbox, that the source flow does not contain, and measure how often the loop still reaches the final oracle.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the CraftDroid benchmark dataset (subject apps, tests, and 120 transfer cases) and the original semantic-mapping transfer technique that LLMigrate is compared against."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the best-performing prior technique on the benchmark; LLMigrate reports a 9.1% higher success rate and 38.2% higher effort reduction against it."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is AppFlow, the earlier machine-learning test-transfer approach whose category dependence motivates the shift to LLM-based semantic understanding."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Identifies GPT-4o, the off-the-shelf multimodal LLM that performs all screen analysis and event selection in the evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Is the Appium framework used to execute source-test events and to extract and clean UI layout hierarchies in both phases."}],"review_version":1}