An Empirical Study for Android-to-OpenHarmony GUI Test Migration

Li Li; Mingyi Zhou; Shaokun Zhang; Xiang Gao; Xinjia Chen; Yakun Zhang; Yiyun Chen; Yunming Ye; Yuxia Zhang

Android GUI tests migrate poorly to OpenHarmony; OpenHarmony-aware fixes raise success from 26% to 81%.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-15 08:55 UTC pith:7SCFWKVC

load-bearing objection Abstract-only Android-to-OpenHarmony GUI test migration study: useful niche benchmark claim, but the 15%/26%→81% result is unauditable without full text and artifacts. the 4 major comments →

arxiv 2607.11245 v2 pith:7SCFWKVC submitted 2026-07-13 cs.SE cs.AI

An Empirical Study for Android-to-OpenHarmony GUI Test Migration

Yakun Zhang , Xinjia Chen , Yiyun Chen , Yuxia Zhang , Mingyi Zhou , Xiang Gao , Shaokun Zhang , Li Li

show 1 more author

Yunming Ye

This is my paper

classification cs.SE cs.AI

keywords GUI test migrationAndroidOpenHarmonyempirical studyReSPlayITeMATH Benchmarkcross-platform testing

verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Migrating GUI tests from Android to OpenHarmony would save large engineering cost, yet prior work offered no OpenHarmony-specific methods and no systematic evaluation. This paper builds the ATH Benchmark of 36 high-download commercial apps and 108 hand-written tests, adapts two leading migration tools (ReSPlay and ITeM) so they can run on OpenHarmony, and measures them. Both tools succeed on only a minority of cases (15% and 26%). Failure analysis attributes most of the gap to OpenHarmony architecture and ecosystem traits rather than ordinary widget mismatch. Feeding those traits into an extended ITeM variant (ITeM-HM) lifts success to 81% (a 214% relative gain), showing that platform-aware adaptation is necessary and effective.

Core claim

Existing Android-to-OpenHarmony GUI test migration approaches achieve only 15% (ReSPlay) and 26% (ITeM) success on a new benchmark of 36 commercial apps and 108 tests; an OpenHarmony-aware extension of ITeM raises success to 81%.

What carries the argument

The ATH Benchmark (36 commercial apps, 108 manually designed GUI tests) together with OpenHarmony-adapted versions of ReSPlay and ITeM, and the subsequent ITeM-HM variant that injects OpenHarmony-specific architectural and ecosystem features into the migration process.

Load-bearing premise

The 108 hand-crafted tests on the 36 selected commercial apps are representative of real migration workloads, and the authors’ adaptations of ReSPlay and ITeM faithfully preserve the original tools’ migration logic.

What would settle it

Run the same 108 ATH tests (or an independently constructed set of comparable size) with an independently re-implemented OpenHarmony adaptation of ITeM that does not add the authors’ OpenHarmony-specific features; if success remains near 26% rather than rising toward 81%, the claimed impact of those features is refuted.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

Platform-agnostic GUI migration tools cannot be trusted for Android-to-OpenHarmony without OpenHarmony-specific handling.
Failure modes are dominated by technical-architecture and ecosystem differences, so future tools should target those differences first.
ITeM-HM’s 81% success rate on commercial apps demonstrates that modest platform-aware extensions can close most of the measured gap.
The ATH Benchmark supplies a reusable yardstick for subsequent migration research on OpenHarmony.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

The same failure categories (architecture and ecosystem) are likely to reappear when migrating tests to other emerging mobile platforms that diverge from Android’s widget and lifecycle model.
A fully automated, continuously updated OpenHarmony feature catalogue could keep the 81% figure from eroding as the platform evolves.
If the hand-written tests under-represent deep navigation or third-party SDK interactions, real-world success of ITeM-HM may be lower than reported.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Desk Editor's Note

Abstract-only Android-to-OpenHarmony GUI test migration study: useful niche benchmark claim, but the 15%/26%→81% result is unauditable without full text and artifacts.

read the letter

The one thing you need to know: this is an abstract-only claim of the first systematic Android-to-OpenHarmony GUI test migration study. They build ATH (36 commercial apps, 108 manual tests), adapt ReSPlay and ITeM, report 15% and 26% success, then ship ITeM-HM at 81% (214% relative lift) after an OpenHarmony-specific failure analysis. That is a concrete, field-relevant package if the full paper and artifacts hold up.

What looks new and useful: a dual-platform migration path and shared benchmark for a real market (Android + OpenHarmony, mainly China). The failure taxonomy tied to architecture and ecosystem differences is the part that could actually help practitioners. Adapting ReSPlay/ITeM is expected engineering; the value is the platform-specific diagnosis and the ITeM-HM lift, not the baseline ports themselves.

Soft spots, in proportion: we only have the abstract. The load-bearing assumptions are exactly the ones the stress-test flags—faithfulness of the ReSPlay/ITeM adaptations, and representativeness of the 108 hand-written tests. No error bars, no inter-rater protocol, no artifact-release statement, no evidence the baselines were not weakened by the port. Mild circularity risk exists because they both built ATH and evaluated their own extension on it. None of that proves the result is wrong; it means the 81% figure cannot yet be attributed cleanly to OpenHarmony-aware design versus implementation choices. Soundness score of 4 is fair for abstract-only; I would not invent deeper flaws.

Who it is for: mobile SE and GUI-testing people who care about dual-platform China ecosystems. Not a theory paper. It deserves a serious referee if the full text, adaptation details, and preferably code/data appear—important enough and concrete enough not to desk-reject on abstract alone. I would not cite it yet, and I would not bring abstract-only to reading group. Send to review once the manuscript is complete; reject or major-revise if adaptations and benchmark construction stay opaque.

Referee Report

4 major / 3 minor

Summary. The paper claims to present the first systematic empirical study of GUI test migration from Android to OpenHarmony. It constructs the ATH Benchmark (36 commercial apps with high download volume and 108 manually designed test cases), adapts two state-of-the-art migration tools (ReSPlay and ITeM) for OpenHarmony, and evaluates them on success rate, failure root causes, and OpenHarmony-specific factors. Reported baseline success is low (15% for ReSPlay, 26% for ITeM). From the failure analysis the authors propose ITeM-HM, an OpenHarmony-aware extension of ITeM, which raises success to 81% (stated as a 214% relative improvement over ITeM).

Significance. If the results hold under full methodological scrutiny, this would be the first empirical baseline and practical guidance for Android-to-OpenHarmony GUI test migration, a timely problem given OpenHarmony’s ecosystem growth. The ATH Benchmark and the concrete mapping of failures to platform architecture and ecosystem traits would be reusable community assets. The reported jump from 26% to 81% for ITeM-HM, if robust and not an in-sample artifact, would show that modest platform-aware adaptations can substantially close the migration gap. Credit is due for framing an under-studied platform pair and for attempting root-cause analysis rather than only aggregate rates; those strengths remain conditional on the full paper’s evidence.

major comments (4)

Abstract: The central claim that existing approaches achieve only 15%/26% success (and that the gap is attributable to OpenHarmony) rests on the unstated assumption that the authors’ adaptations of ReSPlay and ITeM faithfully preserve the original tools’ migration semantics. The abstract supplies no adaptation design, fidelity checks, or comparison against the tools’ original behavior on Android. Without that evidence, measured failures cannot be cleanly attributed to OpenHarmony rather than to adaptation error; this is load-bearing for the ‘existing approaches are less effective’ claim.
Abstract: The ATH Benchmark’s 108 ‘manually designed’ tests on 36 selected commercial apps are the sole evaluation substrate. The abstract gives no selection protocol beyond download volume, no inter-rater or design protocol for the manual tests, and no evidence that the suite represents real migration workloads. Because the same suite is used both to diagnose OpenHarmony-specific failures and to measure ITeM-HM, design or selection bias would directly inflate the reported improvement.
Abstract: ITeM-HM is proposed ‘utilizing these findings’ from the evaluation on ATH and is then measured on the same benchmark, with no held-out apps, cross-validation, or external suite mentioned. The 81% figure and the 214% relative improvement therefore risk being an in-sample fit rather than a generalizable result. A load-bearing revision is needed: either a held-out evaluation or an explicit limitation that the gain is diagnostic rather than confirmatory.
Abstract: Success rates (15%, 26%, 81%) and the 214% relative-improvement figure are stated without error bars, confidence intervals, statistical tests, or per-app/per-test breakdowns. For an empirical SE claim of this magnitude, the absence of any uncertainty quantification makes it impossible to judge whether the improvement is stable or driven by a few easy cases.

minor comments (3)

Abstract: The phrasing ‘214% success-rate relative improvement’ is slightly ambiguous; stating both absolute rates and the relative figure side-by-side (as the parenthetical already begins to do) would reduce misreading.
Abstract: ‘an average of over 9 billion downloads’ should clarify mean vs. median and whether the figure is cumulative across the 36 apps or a per-app average.
Abstract: No statement of artifact availability (ATH Benchmark, adapted ReSPlay/ITeM, ITeM-HM implementation, or evaluation scripts). For an empirical software-engineering study this is expected and should be explicit.

Circularity Check

0 steps flagged

No significant circularity: empirical evaluation paper with measured success rates, not a derivation that reduces to its inputs by construction.

full rationale

This is an abstract-only empirical software-engineering study, not a first-principles derivation. The load-bearing claims are measured success rates (ReSPlay 15%, ITeM 26%, ITeM-HM 81%) on a newly constructed ATH Benchmark of 36 apps and 108 tests. Those rates are experimental outcomes of running adapted tools, not parameters fitted to data and then re-presented as predictions, nor quantities defined in terms of themselves. No equations, uniqueness theorems, or self-cited mathematical results appear in the abstract that would force the reported improvement by construction. Constructing a benchmark and evaluating one's own extension on it is standard empirical practice and does not match any of the enumerated circularity patterns (self-definitional, fitted-input-as-prediction, load-bearing self-citation of an unverified uniqueness claim, ansatz smuggling, or renaming of a known result). Concerns about benchmark representativeness or faithfulness of the ReSPlay/ITeM adaptations are validity/threats-to-validity issues, not circular reductions that can be exhibited by quoting the paper. With only the abstract available, no specific reduction of the form 'Eq. X = Eq. Y by construction' or 'fitted parameter renamed as prediction' can be shown. Score 0 is therefore the correct, proportionate finding.

Axiom & Free-Parameter Ledger

0 free parameters · 3 axioms · 2 invented entities

Empirical SE paper; almost no free parameters in the mathematical sense. Load-bearing content is domain assumptions about what constitutes a successful GUI-test migration and the representativeness of a hand-built benchmark. Two invented artifacts (ATH Benchmark, ITeM-HM) carry the central claims; neither has independent external evidence beyond the paper’s own evaluation.

axioms (3)

domain assumption A migrated GUI test is counted successful if it executes and produces the expected observable outcome on OpenHarmony under the authors’ oracle.
Success-rate figures (15/26/81%) rest on this unstated but standard oracle definition; abstract does not detail flakiness handling or partial-match criteria.
domain assumption The 36 commercial apps and 108 manually written tests are representative of real Android-to-OpenHarmony migration workloads.
All quantitative claims are conditioned on ATH Benchmark; no sampling argument is given in the abstract.
ad hoc to paper Adaptations of ReSPlay and ITeM preserve the original tools’ migration semantics so measured failures are attributable to OpenHarmony, not to adaptation bugs.
Baseline success rates are only meaningful if the ports are faithful; abstract asserts adaptation but supplies no validation.

invented entities (2)

ATH Benchmark no independent evidence
purpose: Provide the evaluation corpus (36 apps, 108 tests) on which all success rates are measured.
Self-constructed dataset; independent_evidence false because no external release or third-party validation is stated in the abstract.
ITeM-HM no independent evidence
purpose: OpenHarmony-aware extension of ITeM that encodes platform-specific UI and architecture features to raise migration success.
New tool variant introduced by the authors; gains are measured only on the authors’ own benchmark, so independent_evidence is false from the abstract alone.

pith-pipeline@v1.1.0-grok45 · 6232 in / 2840 out tokens · 39312 ms · 2026-07-15T08:55:35.941928+00:00 · methodology

0 comments

read the original abstract

To reduce the substantial engineering effort required to test the corresponding applications from Android to OpenHarmony, migrating existing GUI test cases has become a critical problem. However, current research neither proposes solutions tailored for OpenHarmony nor provides a systematic evaluation of migration approaches on this system, leaving developers with limited empirical guidance in practice. In this paper, we present the first systematic empirical study of test migration from Android to OpenHarmony. Specifically, we first construct a dataset referred to as the ATH Benchmark, comprising 36 commercial applications with an average of over 9 billion downloads, along with 108 manually designed test cases. Second, we select two state-of-the-art test migration approaches (i.e., ReSPlay and ITeM) and adapt these two approaches to enable their execution on OpenHarmony. Third, we use the preceding infrastructure to evaluate these two approaches from three perspectives, including testing performance, root causes of failures, and the impact of OpenHarmony characteristics. Our results reveal that existing test migration approaches are less effective (15% success-rate on ReSPlay and 26% success-rate on ITeM) in Android-to-OpenHarmony scenarios. Through an in-depth analysis of failed cases, we identify that test performance is primarily hindered by OpenHarmony-specific characteristics, including technical architecture differences and unique ecosystem traits. Utilizing these findings, we propose an enhanced approach based on ITeM, referred as ITeM-HM, which incorporates specific OpenHarmony system features. As a result, ITeM-HM successfully achieves a 214% success-rate relative improvement over the original ITeM (from 26% to 81%).

Figures

Figures reproduced from arXiv: 2607.11245 by Li Li, Mingyi Zhou, Shaokun Zhang, Xiang Gao, Xinjia Chen, Yakun Zhang, Yiyun Chen, Yunming Ye, Yuxia Zhang.