Pith. sign in

REVIEW 4 major objections 5 minor 32 references

A control-flow graph directs Monkey++ to full UI coverage in under 20 interactions, versus 500 for random Monkey.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-04 00:55 UTC pith:UEPRMGWA

load-bearing objection Graph-guided DFS over a hand-built CFG reaches full 'coverage' on three tiny apps in tens of interactions, but the coverage is measured on the same graph the search traverses, and the comparison to Monkey is asymmetric. the 4 major comments →

arxiv 2608.00236 v1 pith:UEPRMGWA submitted 2026-07-31 cs.SE

Improving Mobile User Interface Testing with Model Driven Monkey Search

Jordan Doyle , Takfarinas Saber , Paolo Arcaini , Anthony Ventresque This is my paper
classification cs.SE
keywords Android testingcontrol flow graphExerciser MonkeyUI coveragedepth-first searchmodel-based testingautomated test generationstatic analysis
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper aims to show that a static control-flow graph of an Android app—whose vertices are statements, methods, and user-interface controls—can replace the blind randomness of the standard Monkey testing tool. It introduces Monkey++, which runs a depth-first search over this graph to choose which UI control to click next, and reports that Monkey++ reaches 100% interface coverage in at most 19 interactions across three small apps, while Monkey needs 500 interactions and averages 85–88% coverage. The significance is that targeted, model-driven event generation can make mobile UI testing substantially faster and more complete, without manual script writing.

Core claim

The paper's central claim is that representing an Android application as a directed graph with three vertex types—statements, methods, and user-interface controls—and using that graph to guide the Monkey tool turns a slow random search into a fast deterministic one. In the experiments, Monkey++ achieves full interface coverage with 13–19 interactions per app, whereas Monkey's random touch events reach at most 100% coverage only after 180–500 interactions and average only an 8% hit rate on interactive controls. The graph supplies two things Monkey lacks: a list of all reachable UI controls and a memory of which ones have already been exercised, so redundant events are skipped. The paper argue

What carries the argument

The control flow structure: a directed graph G=(V,E) where V splits into statement vertices, method vertices (lifecycle, callback, and standard Java methods), and UI-control vertices. Edges represent static control flow, including method calls and in-method statement order. Because Android lifecycle transitions are runtime-dependent, the graph is supplemented by a primitive runtime model that tracks the activity stack and each activity's lifecycle state; during a depth-first search, this model decides which lifecycle edge to follow when an activity is launched, finished, or resumed. The DFS uses the graph to pick UI vertices to click and the runtime model to navigate between activities.

Load-bearing premise

The hand-written runtime model of Android's activity stack and lifecycle must correctly mirror the framework's actual behavior; if it does not, the DFS may miss UI controls and the claimed 100% coverage would be an artifact of the model rather than a property of the app.

What would settle it

Run Monkey++ on an app that uses fragments or starts activities from asynchronous callbacks—situations the paper's runtime model does not explicitly handle—and check whether the DFS still reaches every UI control; a reproducible miss would show the coverage claim depends on the model's completeness.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Monkey++ reaches 100% interface coverage in at most 19 interactions, one order of magnitude fewer than the 500 interactions Monkey needs.
  • Because every Monkey++ event is aimed at a known UI control, no events are wasted; Monkey's average hit rate is 8%.
  • The same control-flow graph can host other search strategies (e.g., breadth-first or model-based) for larger apps.
  • The graph's coverage metric gives a concrete, comparable definition of 'how much of the interface has been tested'.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If this approach scales beyond small apps, one-time static analysis plus a targeted search could replace random Monkey runs in continuous-integration pipelines, reducing test time from minutes to seconds.
  • The paper only measures interface coverage, not bug detection; a natural extension is to check whether the higher coverage translates to more real faults found, which is not guaranteed.
  • The reliance on a hand-written lifecycle model suggests a dynamic or learned model could make the method robust to apps whose runtime behavior depends on system events, fragments, or configuration changes.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a static control-flow structure for Android apps that includes statement, method, and UI vertices, and uses it to guide an enhanced Exerciser Monkey tool (Monkey++) with a depth-first search. The authors compare coverage of Monkey++ with standard random Monkey on three small open-source Android apps. They report that Monkey++ reaches 100% interface coverage in 13–19 interactions, whereas Monkey requires up to 500 interactions and achieves only 85–100% coverage, with only about 8% of events hitting interactive controls. The claimed contribution is a graph representation that removes redundant interactions and improves both effectiveness and efficiency relative to random Monkey.

Significance. The idea of using a static, UI-aware graph to drive a random monkey tool is sensible and potentially useful for automated Android testing. If the results hold under more rigorous validation, the reported order-of-magnitude improvement in interaction count would be practically meaningful, especially for large apps where random testing is expensive. The paper is honest about some limitations, such as the immature graph generation and the small chosen AUTs. However, the central quantitative claim rests on an asymmetric measurement protocol and an unvalidated runtime lifecycle model, so the significance cannot be fully assessed without additional experiments and model validation.

major comments (4)
  1. [§IV-A, §V (Listing 4), §IV-B] The coverage metric is applied asymmetrically. For Monkey++, interface coverage is computed inside the graph: v.visit() is called when the DFS traverses a UI vertex, without verifying that the corresponding control was actually present and clickable on the device at that moment. For Monkey, coverage is reconstructed from device events and UI Automator screen dumps. Thus a wrong lifecycle transition can mark a UI vertex as visited even if the screen never showed that control, inflating the reported 100% coverage. Please validate Monkey++ coverage against actual UI hierarchy dumps taken after each interaction, or explain why the graph-internal visit is a faithful proxy.
  2. [§III-A, §V-A] The paper explicitly omits runtime-dependent lifecycle edges from the static graph and replaces them with a hand-written activity-stack/lifecycle-status model. Section V describes this model only informally, and there is no validation that the predicted transitions match Android framework behaviour. Since the DFS's ability to reach every UI vertex depends entirely on this model, an error in the model would make the 100% coverage figure an artifact of the model rather than a property of the app. Please provide a precise specification of the runtime model and an empirical check (e.g., comparing predicted vs actual activity transitions on the AUTs).
  3. [§VI, Table II] The comparison is not apples-to-apples. Monkey is always run for 500 interactions, even when it reaches 100% coverage earlier (e.g., Mo Clock Max 100% at 300, Volume Control Max 100% at 180), whereas Monkey++ stops at full coverage. This protocol inflates the efficiency gap and makes the 'one order of magnitude faster' claim less clean. Report interactions-to-100% for Monkey under a stopping rule, with medians and spreads over the 10 runs, in addition to the fixed-budget coverage results.
  4. [§III-B.3] The mapping from UI controls to callback methods is described as not fully solved: 'Currently we instrument the AUT so that the control ID for each callback method can be found in the Jimple statements.' This mapping is load-bearing because a missed control or a wrong callback would directly affect V_ui and the coverage denominator. Please specify the instrumentation more completely and validate the control-to-callback mapping on the three AUTs, for example by comparing with the UI Automator view hierarchy.
minor comments (5)
  1. [Abstract/§VI] The phrase 'one order of magnitude faster' is not consistently defined. Table II shows Monkey++ at 13–19 interactions versus 180–400 for the best Monkey run, but the average Monkey coverage is at 500 interactions. Clarify which comparison supports the 'order of magnitude' statement.
  2. [§VI.B] The 8% 'hit rate' is not formally defined. State explicitly whether it is the fraction of touches landing on any clickable view, or on a UI control in V_ui, and report how it was computed.
  3. [§IV-A, Definition 2] Definition 2 requires the first vertex of a test to be an interface vertex, but the initial launch of an activity may not itself be a UI vertex. Clarify how the first vertex is chosen in both Monkey and Monkey++.
  4. [§VI] The claim that Monkey++ is deterministic and therefore needs only one run is reasonable for the graph algorithm, but device timing and UI rendering can still introduce variability. Please report whether the same UI-vertex sequence was observed on repeated executions.
  5. [Table I] Typo in header: 'Vol ume Control' should be 'Volume Control'. Also, the x-axis label in Figures 4a–4c would benefit from repeating 'No. Interactions' in each subplot for readability.

Circularity Check

1 steps flagged

Monkey++ full-coverage result is entailed by its own graph-based coverage definition, not independently verified.

specific steps
  1. self definitional [Definition 4 (Section IV-A); Listing 4 (Section V-A); Table II (Section VI-B)]
    "The interface coverage achieved by the tool is defined as: IC(T,A, G) = | S_{t∈TS} {v∈t|v∈V ui }| / |V ui| ... protected void search(Vertex v) { v.visit(); v.localVisit(); for(DefaultEdge e : graph.outgoingEdgesOf(v)) { Vertex target = graph.getEdgeTarget(e); if(!target.hasLocalVisit()) { search(target); } } ... }"

    Monkey++ coverage is measured on the very graph the DFS is written to exhaust. Definition 4 counts V_ui vertices that occur in the generated test suite; Listing 4 visits every vertex reachable via graph edges (with lifecycle edges supplied by a hand-written activity-stack model) and marks it covered. Reaching 100% interface coverage therefore follows from the search completing on its own model, independent of whether the corresponding UI controls were ever actually present and clickable on a device. Table II's 13-19 interactions equal the number of graph controls plus back-stack moves, fixing the headline by construction. Monkey's coverage, by contrast, is reconstructed from UI Automator dumps (Section IV-B), so the comparison is not symmetric.

full rationale

The central headline claim—Monkey++ achieves full interface coverage in 13–19 interactions—is partially circular: interface coverage (Definition 4) is defined as the fraction of graph vertices V_ui that appear in the test suite, and Monkey++'s DFS (Listing 4) is designed to visit every reachable vertex of that same graph. Thus 100% coverage is a logical consequence of the search terminating on its own model, not an empirically verified property of the actual device UI. The paper itself flags the supporting model as unvalidated and immature: Section III-A says lifecycle edges are runtime-dependent and are replaced by a primitive runtime model; Section III-B.3 admits 'Currently we instrument the AUT so that the control ID for each callback method can be found in the Jimple statements. Finding better methods of retrieving the interface data is part of future work'; Section VI-A says the AUT selection criteria were 'due to the immature nature of our graph generation.' There are no fitted parameters and no load-bearing self-citations, so the paper is not circular in the fitted-input or self-citation senses. The comparison to random Monkey is an independent check only if both coverage numbers are measured on actual device UI states; for Monkey++ the paper reports graph-visit coverage without equivalent UI Automator dumps or post-condition checks. This is a partial 'prediction by construction' circularity, not evidence of intent.

Axiom & Free-Parameter Ledger

2 free parameters · 4 axioms · 0 invented entities

No free parameters are fitted to data; the only numerical choices are experimental (500-tap budget, 0.5 s throttle). The central claim rests on domain assumptions about Soot/FlowDroid completeness, the hand-built runtime lifecycle model, and the ability to instrument apps to map controls to callbacks. No new physical or conceptual entities are introduced; the control-flow structure and runtime model are software artifacts, not postulated entities requiring independent evidence.

free parameters (2)
  • Monkey interaction budget = 500
    Monkey is always run for 500 interactions even after reaching 100% coverage, while Monkey++ stops once all UI vertices are visited; this choice inflates the efficiency gap.
  • Monkey event throttle = 0.5 s
    Chosen 'to allow plenty of time between events'; affects wall-clock time comparisons between the tools.
axioms (4)
  • domain assumption Soot and FlowDroid static analysis can produce a sufficiently complete call graph and statement-level CFG for Android apps, including lifecycle method bodies.
    Used in Section III-B to populate the graph; if incorrect, the graph omits app behavior and DFS cannot reach all UI elements.
  • ad hoc to paper A hand-built activity-stack/lifecycle runtime model (Section V-A) correctly predicts Android framework transitions that are absent from the static graph.
    The graph deliberately excludes runtime-dependent lifecycle edges; the DFS relies entirely on this model to decide where to continue, and the paper does not validate it against real framework behavior.
  • domain assumption Instrumenting the AUT is sufficient to link UI controls to their callback methods, overcoming FlowDroid's failure to provide this link.
    Section III-B.3 states FlowDroid fails to link controls to callbacks and that instrumentation is used; this step may not generalize and is not fully specified.
  • standard math DFS on a finite graph terminates and visits all reachable vertices.
    Assumed by the local-visit/no-repeat scheme in Listing 4; standard but load-bearing for the 'full coverage' claim.

pith-pipeline@v1.3.0-alltime-deepseek · 10869 in / 11716 out tokens · 108112 ms · 2026-08-04T00:55:17.703632+00:00 · methodology

0 comments
read the original abstract

Testing mobile applications often relies on tools, such as Exerciser Monkey for Android systems, that simulate user input. Exerciser Monkey, for example, generates random events (e.g., touches, gestures, navigational keys) that give developers a sense of what their application will do when deployed on real mobile phones with real users interacting with it. These tools, however, have no knowledge of the underlying applications' structures and only interact with them randomly or in a predefined manner (e.g., if developers designed scenarios, a labour-intensive task) -- making them slow and poor at finding bugs. In this paper, we propose a novel control flow structure able to represent the code of Android applications, including all the interactive elements. We show that our structure can increase the effectiveness (higher coverage) and efficiency (removing duplicate/redundant tests) of the Exerciser Monkey by giving it knowledge of the test environment. We compare the interface coverage achieved by the Exerciser Monkey with our new Monkey++ using a depth first search of our control flow structure and show that while the random nature of Exerciser Monkey creates slow test suites of poor coverage, the test suite created by a depth first search is one order of magnitude faster and achieves full coverage of the user interaction elements. We believe this research will lead to a more effective and efficient Exerciser Monkey, as well as better targeted search based techniques for automated Android testing.

Figures

Figures reproduced from arXiv: 2608.00236 by Anthony Ventresque, Jordan Doyle, Paolo Arcaini, Takfarinas Saber.

Figure 1
Figure 1. Figure 1: Control flow structure representing a small Android [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Control flow structure for Activity Lifecycle applica [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Interaction Scenario: When “SayHi” is touched we [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Interface Coverage (over 10 runs of Monkey and 1 run [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

32 extracted references

  1. [1]

    Statcounter global stats: Mobile operating system market share world- wide,

    “Statcounter global stats: Mobile operating system market share world- wide,” https://gs.statcounter.com/os-market-share/mobile/worldwide/, accessed: 19-01-2021

    2021

  2. [2]

    Statista: Mobile app monetization - statistics and facts,

    J. Clement, “Statista: Mobile app monetization - statistics and facts,” https://www.statista.com/topics/983/mobile-app-monetization/, 2020, accessed: 19-01-2021

    2020

  3. [3]

    Tortoise or hare? quantifying the effects of performance on mobile app retention,

    A. Zuniga, H. Flores, E. Lagerspetz, P. Nurmi, S. Tarkoma, P. Hui, and J. Manner, “Tortoise or hare? quantifying the effects of performance on mobile app retention,” inThe World Wide Web Conference, 2019, pp. 2517–2528

    2019

  4. [4]

    Auto- mated testing of android apps: A systematic literature review,

    P. Kong, L. Li, J. Gao, K. Liu, T. F. Bissyand ´e, and J. Klein, “Auto- mated testing of android apps: A systematic literature review,”IEEE Transactions on Reliability, vol. 68, no. 1, pp. 45–66, 2018

    2018

  5. [5]

    Real challenges in mobile app development,

    M. E. Joorabchi, A. Mesbah, and P. Kruchten, “Real challenges in mobile app development,” inACM/IEEE International Symposium on Empirical Software Engineering and Measurement, 2013, pp. 15–24

    2013

  6. [6]

    Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing,

    M. Linares-V ´asquez, K. Moran, and D. Poshyvanyk, “Continuous, evolutionary and large-scale: A new perspective for automated mobile app testing,” inICSME, 2017, pp. 399–410

    2017

  7. [7]

    Static control- flow analysis of user-driven callbacks in Android applications,

    S. Yang, D. Yan, H. Wu, Y . Wang, and A. Rountev, “Static control- flow analysis of user-driven callbacks in Android applications,” inICSE, vol. 1. IEEE, 2015, pp. 89–99

    2015

  8. [8]

    Monkeyrunner,

    Android Developers, “Monkeyrunner,” https://developer.android.com/ studio/test/monkeyrunner/index.html, 2015

    2015

  9. [9]

    Androidviewclient,

    D. T. Milano, “Androidviewclient,” https://github.com/dtmilano/ AndroidViewClient, 2016

    2016

  10. [10]

    UI Automator,

    Android Developers, “UI Automator,” https://developer.android.com/ training/testing/ui-automator, 2014

    2014

  11. [11]

    Espresso,

    Android Developers, “Espresso,” https://developer.android.com/training/ testing/espresso, 2016

    2016

  12. [12]

    Verma,Mobile Test Automation With Appium

    N. Verma,Mobile Test Automation With Appium. Packt Publishing Ltd, 2017

    2017

  13. [13]

    Zadgaonkar,Robotium automated testing for Android

    H. Zadgaonkar,Robotium automated testing for Android. Packt Publishing Birmingham, 2013

    2013

  14. [14]

    Barista: A technique for recording, encoding, and running platform independent Android tests,

    M. Fazzini, E. N. D. A. Freitas, S. R. Choudhary, and A. Orso, “Barista: A technique for recording, encoding, and running platform independent Android tests,” inICST, 2017, pp. 149–160

    2017

  15. [15]

    Reran: Timing-and touch-sensitive record and replay for Android,

    L. Gomez, I. Neamtiu, T. Azim, and T. Millstein, “Reran: Timing-and touch-sensitive record and replay for Android,” inICSE, 2013, pp. 72– 81

    2013

  16. [16]

    Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosys- tem,

    M. Halpern, Y . Zhu, R. Peri, and V . J. Reddi, “Mosaic: cross-platform user-interaction record and replay for the fragmented android ecosys- tem,” inISPASS, 2015, pp. 215–224

    2015

  17. [17]

    Espresso test recorder,

    Android Developers, “Espresso test recorder,” https://developer.android. com/studio/test/espresso-test-recorder.html, 2016

    2016

  18. [18]

    UI/Application Exerciser Monkey,

    Android Developers, “UI/Application Exerciser Monkey,” https:// developer.android.com/studio/test/monkey.html, 2012

    2012

  19. [19]

    Dynodroid: An input generation system for Android apps,

    A. Machiry, R. Tahiliani, and M. Naik, “Dynodroid: An input generation system for Android apps,” inFSE, 2013, pp. 224–234

    2013

  20. [20]

    Targeted and depth-first exploration for systematic testing of Android apps,

    T. Azim and I. Neamtiu, “Targeted and depth-first exploration for systematic testing of Android apps,” inProceedings of the 2013 ACM SIGPLAN international conference on Object oriented programming systems languages & applications, 2013, pp. 641–660

    2013

  21. [21]

    Crashscope: A practical tool for automated testing of android applications,

    K. Moran, M. Linares-V ´asquez, C. Bernal-C ´ardenas, C. Vendome, and D. Poshyvanyk, “Crashscope: A practical tool for automated testing of android applications,” inICSE, 2017, pp. 15–18

    2017

  22. [22]

    Using GUI ripping for automated testing of Android applications,

    D. Amalfitano, A. R. Fasolino, P. Tramontana, S. De Carmine, and A. M. Memon, “Using GUI ripping for automated testing of Android applications,” inASE, 2012, pp. 258–261

    2012

  23. [23]

    Guided GUI testing of Android apps with minimal restart and approximate learning,

    W. Choi, G. Necula, and K. Sen, “Guided GUI testing of Android apps with minimal restart and approximate learning,”SIGPLAN Not., vol. 48, no. 10, pp. 623–640, 2013

    2013

  24. [24]

    Guided, stochastic model-based GUI testing of Android apps,

    T. Su, G. Meng, Y . Chen, K. Wu, W. Yang, Y . Yao, G. Pu, Y . Liu, and Z. Su, “Guided, stochastic model-based GUI testing of Android apps,” inFSE, 2017, pp. 245–256

    2017

  25. [25]

    MobiGUITAR: Automated model-based testing of mobile apps,

    D. Amalfitano, A. R. Fasolino, P. Tramontana, B. D. Ta, and A. M. Memon, “MobiGUITAR: Automated model-based testing of mobile apps,”IEEE software, vol. 32, no. 5, pp. 53–59, 2014

    2014

  26. [26]

    A general framework for comparing automatic testing techniques of android mobile apps,

    D. Amalfitano, N. Amatucci, A. M. Memon, P. Tramontana, and A. R. Fasolino, “A general framework for comparing automatic testing techniques of android mobile apps,”Journal of Systems and Software, vol. 125, pp. 322–343, 2017

    2017

  27. [27]

    An empirical study of Android test generation tools in industrial cases,

    W. Wang, D. Li, W. Yang, Y . Cao, Z. Zhang, Y . Deng, and T. Xie, “An empirical study of Android test generation tools in industrial cases,” in ASE, 2018, pp. 738–748

    2018

  28. [28]

    The Soot framework for Java program analysis: a retrospective,

    P. Lam, E. Bodden, O. Lhot ´ak, and L. Hendren, “The Soot framework for Java program analysis: a retrospective,” inCetus Users and Compiler Infastructure Workshop (CETUS 2011), vol. 15, 2011, p. 35

    2011

  29. [29]

    Jimple: Simplifying Java bytecode for analyses and transformations,

    R. Vallee-Rai and L. J. Hendren, “Jimple: Simplifying Java bytecode for analyses and transformations,” 1998

    1998

  30. [30]

    Static analysis of Android apps: A systematic literature review,

    L. Li, T. F. Bissyand ´e, M. Papadakis, S. Rasthofer, A. Bartel, D. Octeau, J. Klein, and L. Traon, “Static analysis of Android apps: A systematic literature review,”IST, vol. 88, pp. 67–95, 2017

    2017

  31. [31]

    FlowDroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps,

    S. Arzt, S. Rasthofer, C. Fritz, E. Bodden, A. Bartel, J. Klein, Y . Le Traon, D. Octeau, and P. McDaniel, “FlowDroid: Precise context, flow, field, object-sensitive and lifecycle-aware taint analysis for Android apps,”Acm Sigplan Notices, vol. 49, no. 6, pp. 259–269, 2014

    2014

  32. [32]

    Fossdroid: Free and open source Android apps,

    “Fossdroid: Free and open source Android apps,” https://fossdroid.com, accessed: 19-01-2021

    2021