Pith. sign in

REVIEW 4 major objections 9 minor 22 references

An AI-driven mobile tester combined with fault injection scales chaos testing to 180,000 production runs.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

An integrated LLM-based mobile testing system and service-level fault injector ran 180,000+ chaos tests at Uber, finding 23 resilience defects that manual and backend-only testing missed.

T0 review reviewed 2026-08-03 challenge →

load-bearing objection Genuine industrial integration paper with real operational scale; worth serious review, with caveats about in-sample RCA evaluation and triage ground truth. the 4 major comments →

arxiv 2602.06223 v2 pith:VHBMRQBN submitted 2026-02-05 cs.SE

Scaling Mobile Chaos Testing with AI-Driven Test Execution

classification cs.SE
keywords chaos engineeringmobile testingfault injectionLLM-based testingAI-driven test executionresilience engineeringroot cause analysisservice-level fault injection
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that the combinatorial explosion in mobile chaos testing can be sidestepped by letting an LLM-driven agent navigate the app under real backend failures, instead of scripting test cases for every flow, city, and failure combination. It reports a production system pairing such an agent (DragonCrawl) with service-level fault injection (uHavoc), and claims that over 180,000 automated runs across 47 core flows uncovered 23 resilience risks, two of which caused crashes invisible to backend-only testing. The authors contend this makes continuous mobile resilience validation practical at scale, replacing ad-hoc manual work equivalent to roughly 39,000 hours of testing effort.

Core claim

The central claim is that adaptive AI-driven mobile test execution—which reads screens semantically and chooses actions by goal rather than fixed selectors—remains reliable enough under injected backend faults to serve as the primary observation point for chaos engineering. Combined with safe, header-based fault injection at the RPC layer, this yields a system that can validate mobile resilience continuously in production, achieving 99%+ pass rates under faults and 88% precision@5 in automated root-cause attribution to backend services.

What carries the argument

The load-bearing mechanism is the pairing of DragonCrawl, an LLM-based crawler that converts screens into text and frames test goals and assertions as natural-language questions so it can adapt to degraded UI states, with uHavoc, which injects abort, timeout, and latency faults into backend RPCs via test-tenancy headers. A baseline control run without faults isolates injected failures from environmental noise, and an automated root-cause pipeline ranks candidate RPCs using a weighted causal score tying error screens to network requests.

Load-bearing premise

The labeling of the ~1,300 non-passing runs as genuine app resilience problems rather than LLM navigation failures under degraded screens is not validated against an independent ground-truth set.

What would settle it

Have human experts replay a random sample of failed runs and classify each as an app resilience defect or an agent navigation failure; if more than a few percent of non-passing runs are navigation failures mislabeled as resilience issues, the 23-issue count and 99.27% pass rate are overstated.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the claims hold, mobile chaos testing can run nightly at scale without per-test scripting across flows, cities, and fault scenarios.
  • The 23 discovered issues—12 blocking trips or food orders and 2 causing app crashes—would have become production incidents without proactive detection.
  • The two crash-level issues demonstrate that some resilience defects are observable only through the mobile UI, not through backend testing alone.
  • Automated root-cause analysis reduces debugging from hours to minutes, expanding the pool of engineers who can triage resilience failures.
  • The 99.27% pass rate under continuous nightly execution suggests the system is safe enough for routine production-adjacent operation.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same integration pattern—goal-based semantic UI agents as resilience sensors plus service-level fault injection—could transfer to other large-scale mobile ecosystems with mature tracing and service tier tagging.
  • The roughly 1,300 non-passing runs (0.73% of 180,000) are not independently validated against a ground-truth set, so the true count of resilience defects may shift if some AI navigation failures are mislabeled as app issues.
  • The faster resolution of teams assigned multiple tickets hints that organizational ownership dynamics, not just the detection technology, drive realized resilience gains.
  • Extending fault injection beyond RPC-level errors to infrastructure and third-party failures (payments, maps) could uncover additional UX-level risks, but would require tighter safety controls.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 9 minor

Summary. The paper reports on an industrial system that couples DragonCrawl, an LLM-driven mobile UI test executor, with uHavoc, a service-level fault injection platform, to perform chaos testing on Uber's Rider, Driver, and Eats apps. The claims include: more than 180,000 automated chaos test executions across 47 critical flows since Q1 2024; discovery of 23 resilience risks, of which 12 blocked core flows and 2 caused app crashes; 99%+ test pass rate under fault injection; and automated root-cause analysis achieving 88% precision@5. The paper also describes system architecture, implementation challenges, operational results, and lessons learned. The central operational narrative is plausible and internally consistent, but the quantitative evaluation has important gaps in statistical reporting, validation methodology, and the separation of AI navigation failures from genuine resilience defects.

Significance. If substantiated, this is a valuable industrial experience report: it demonstrates a concrete way to overcome the combinatorial explosion of mobile chaos testing by combining an adaptive LLM-based test driver with backend fault injection, and it surfaces UX-level resilience defects that backend-only testing misses. The strengths of the paper are the real production scale (180k+ runs), the explicit safety model (test tenancy isolation), and the attempt to automate root-cause attribution. However, the headline numbers—23 issues, 88% RCA precision@5, 99%+ pass rate—are not yet backed by the evidence presented. The paper would make a stronger contribution if the evaluation included statistical tests, a held-out validation protocol for the RCA weights, and an independent audit of the failure-to-resilience attribution.

major comments (4)
  1. [§5.3, Table 3] The pass-rate comparison between baseline (0.993) and T2 fault injection (0.99) is reported as a degradation, but no confidence intervals, failure counts, or significance tests are given. With 300 executions per scenario, the difference corresponds to roughly one additional failure in 300 runs. The paper should report exact counts, binomial confidence intervals, or a test of equivalence; otherwise the claim of a 'drop' for T2 is not statistically supported. The same applies to the precision@k differences in Table 1 and latency differences in Table 2, where the per-scenario sample sizes are not clearly connected to the tables.
  2. [§3.2 and §5.5, Table 5] The RCA precision@k results are computed without any described held-out evaluation. The Score(causal) weights—F(status_code), F(tier), and F(category)—are stated to be 'established through empirical observation' (§3.2), and no separate test set, temporal split, or cross-validation is described. As written, precision@5 = 0.88 could reflect in-sample fit rather than predictive performance. The paper also does not define the ground-truth labeling process for the RCA evaluation (who annotated the true root cause and with what inter-rater reliability) nor report the number of cases evaluated. Without this, the '88% precision@5' claim is not interpretable.
  3. [§4 and §6.1] The central discovery claim—23 genuine resilience risks—depends on the assertion that failures under fault injection are attributable to backend resilience issues rather than to DragonCrawl's own navigation failures. Section 4 states that AI-related failures account for 0.4% of executions and are 'easily distinguishable from genuine resilience risks,' but no operational criteria, manual audit, or inter-rater reliability is provided for that distinction. With roughly 1,300 non-passing runs out of 180,000, even a small misclassification rate could materially affect the 23-issue count and the '2 crashes' claim. The 'golden screen' argument in §5.4 is not a sufficient control: both a genuine backend failure and an AI navigation failure can prevent the goal screen from appearing. The authors should provide an independent validation set—e.g., a blind manual triage of a random sample of failure
  4. [§5.4] The sentence 'The 23 issues we found—12 blocking trips/orders, 2 causing crashes—show this approach captures genuine failures rather than masking them' uses the very finding that needs validation as evidence for the validation. This is circular. The issue count and severity distribution are the quantities to be established, not proof of test fidelity. Please separate the design argument for golden-screen assertions from the empirical evidence that mislabeling does not occur.
minor comments (9)
  1. [Abstract and §1] The paper alternates between '61 critical user flows' (Introduction) and '47 critical flows' (Abstract, §6). The relationship between the two numbers is explained only in §6.2; please clarify earlier.
  2. [§5, Table 1] In Table 1, Precision@1 is higher than Precision@2 and Precision@3 for all scenarios. This is possible in a single-relevant-item retrieval setting, but the metric definition should be stated explicitly to avoid confusion.
  3. [§3.2] The term 'normalFailureRate' is used in the Score(causal) formula but never precisely defined. Is it the fraction of failing requests in baseline runs, or the deviation from a baseline error rate? Please provide the exact definition.
  4. [§5.3, Table 3] The pass-rate table reports only four significant digits; with 300 runs, the standard error is about 0.005, so the reported values are at the edge of measurement precision. Include raw counts.
  5. [§4] 'The LLM entered problematic loops 2% of executions' and 'These failures account for 0.4%' are two different percentages; clarify the relationship (e.g., loops that remain unresolved after cycle detection).
  6. [§6.1] The claim that the 23 issues 'were not detected through other means' needs a concrete basis: which other systems were run against the same flows and on what schedule? Without this, the reader cannot assess the claim of complementarity with backend testing.
  7. [Figure 5] Typo: 'As the the dotted trendline shows' should be 'As the dotted trendline shows.'
  8. [§5.4] Typo: 'LlaVa-NeXT' should be 'LLaVA-NeXT'.
  9. [§6.2] 'This represents 39,000 hours of automated manual work that has been automated away' is redundant; consider rephrasing to '39,000 hours of manual testing effort that has been replaced by automation.'

Circularity Check

0 steps flagged

No demonstrated circularity: the paper's central claims rest on in-paper experiments and operational counts; the RCA precision@k and 23-issue triage are validation concerns, not reductions to fitted inputs.

full rationale

The paper is an operational experience report whose main claims—DragonCrawl can navigate degraded screens, the integrated system sustains ~99% pass rates under fault injection, RCA attributes failures to backend RPCs with precision@k, and 23 resilience risks were found in production—are supported by in-paper measurements (Tables 1–5) and operational counts, not by a derivation that reduces to its own inputs. The most plausible circularity candidate is the RCA heuristic: Section 3.2 says the Score(causal) parameter values "were established through empirical observation," and Section 5.5 reports precision@k without explicitly describing a held-out set. If the same incidents used to set the weights were also used to compute Table 5, the precision numbers would be in-sample. However, the paper does not state that the evaluation was constructed that way, and the claim that the parameters "have proven consistent across daily execution of all 47 core flows" suggests ongoing operational confirmation. The absence of a described train/test split is an evaluation-reporting gap, not a demonstrated circularity under the evidence rule. The self-citation to Uber's DragonCrawl blog [17] is used for background and metric framing, while Tables 1–3 independently measure DragonCrawl's precision, latency, and pass rate, so the citation is not load-bearing. The attribution of failures to "genuine resilience risks" rather than LLM navigation failures is a validity threat—0.4% of executions are AI-caused and not triaged, and no independent audit is described—but this is a labeling/validation concern, not a circular derivation: the 23-issue count is an operational triage outcome rather than a prediction forced by construction. Overall, no circular step reaches the evidentiary bar required here.

Axiom & Free-Parameter Ledger

3 free parameters · 5 axioms · 0 invented entities

No new theoretical entities are postulated. The paper's load-bearing assumptions are operational: correct service-tier tagging, golden-screen fidelity as a resilience proxy, test-traffic isolation, sufficient tracing coverage, and LLM classification accuracy. The three RCA weight sets are free parameters fitted to observed data, and the reported precision@k is therefore partly in-sample.

free parameters (3)
  • F(status_code) weights = 1.0 (5xx), 0.5 (4xx), 0.2 (2xx)
    Hand-set weights in the RCA causal score (§3.2), chosen from 'empirical observation'; no sensitivity analysis or held-out tuning reported.
  • F(tier) weights = 1.0 (tier-0), 0.9 (tier-1), 0.7 (tier-2), 0.4 (tier-3), 0.1 (tier-4/5)
    Empirically established weights in the causal ranking heuristic (§3.2); the resulting precision@k (§5.5) is reported without evidence of a held-out set.
  • F(category) semantic weights = 3.0 (direct), 2.0 (indirect), 1.2 (supporting), 0.3 (unrelated)
    Weights for LLM relevance categories in the RCA score (§3.2); tuned on observed failures, so the reported 88% precision@5 is likely in-sample.
axioms (5)
  • domain assumption Service tier tagging correctly reflects production criticality of every service.
    Explicitly acknowledged in §8 Limitations: 'A portion of our success comes from the rigorous effort teams have put in to correctly identify the priority of their services.' If incorrect, the 70% dependency-violation finding is partially an artifact of mislabeling.
  • domain assumption Reaching the test's end-state 'golden screen' implies the critical user flow remained functional under fault injection.
    Load-bearing for defining pass rate as a resilience signal; stated in §5.4 ('If these screens are not reached, the test fails... the golden screen simply never appears'). No independent ground-truth validation is provided.
  • domain assumption uHavoc's header-based fault injection affects only test tenancy requests and does not alter production traffic.
    Safety property asserted in §2.4 and §8; the paper provides no independent audit or verification of the isolation mechanism.
  • domain assumption Jaeger distributed tracing coverage is complete enough for attribution to specific backend services.
    Acknowledged as a dependency in §8: services lacking trace instrumentation create attribution gaps and unresolved issues. The RCA precision numbers assume sufficient coverage.
  • domain assumption GPT-4o's screen understanding and RCA classifications are accurate enough to serve as ground truth.
    Used for action selection, VQA assertions, and RCA categorization (§5); the only validation is the paper's internal confusion matrix and precision tables, not an independent benchmark.

reviewed 2026-08-03 · how reviews work

0 comments
Cite this review

Pith. "Pith review of Scaling Mobile Chaos Testing with AI-Driven Test Execution." pith.science (2026). https://pith.science/paper/VHBMRQBN

@misc{pith2026260206223,
  author       = {Pith},
  title        = {Pith review of: Scaling Mobile Chaos Testing with AI-Driven Test Execution},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/VHBMRQBN}},
  note         = {Machine review of arXiv:2602.06223}
}
Share X Bluesky LinkedIn Reddit HN
read the original abstract

Mobile applications in large-scale distributed systems are susceptible to backend service failures, yet traditional chaos engineering approaches cannot scale mobile testing due to the combinatorial explosion of flows, locations, and failure scenarios that need validation. We present an automated mobile chaos testing system that integrates DragonCrawl, an LLM-based mobile testing platform, with uHavoc, a service-level fault injection system. The key insight is that adaptive AI-driven test execution can navigate mobile applications under degraded backend conditions, eliminating the need to manually write test cases for each combination of user flow, city, and failure type. Since Q1 2024, our system has executed over 180,000 automated chaos tests across 47 critical flows in Uber's Rider, Driver, and Eats applications, representing approximately 39,000 hours of manual testing effort that would be impractical at this scale. We identified 23 resilience risks, with 70% being architectural dependency violations where non-critical service failures degraded core user flows. Twelve issues were severe enough to prevent trip requests or food orders. Two caused application crashes detectable only through mobile chaos testing, not backend testing alone. Automated root cause analysis reduced debugging time from hours to minutes, achieving 88% precision@5 in attributing mobile failures to specific backend services. This paper presents the system design, evaluates its performance under fault injection (maintaining 99% test reliability), and reports operational experience demonstrating that continuous mobile resilience validation is achievable at production scale.

Figures

Figures reproduced from arXiv: 2602.06223 by Ali Zamani, Arturo Bravo Rovirosa, Ashish Samant, Juan Marcano, Kaelan Mikowicz, Kai Song, Lingchao Chen, Mayank Bansal, Mengdie Zhang, Sowjanya Puligadda, Srikanth Prodduturi, Tim Smyth.

Figure 1
Figure 1. Figure 1: uHavoc Platform Architecture showing fault injec [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: High-level overview of DragonCrawl. By analyzing what it sees on each screen and understanding the testing goal, DragonCrawl dynamically decides what actions to take rather than following pre-scripted sequences on a per test basis. This eliminates the need for engineers to hardcode separate test cases for every combination of language, location, fault-injection scenario, and UI variation - directly solving… view at source ↗
Figure 3
Figure 3. Figure 3: DragonCrawl and uHavoc integration architecture. [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Automated Root Cause Analysis Pipeline. Root Cause Analysis Heuristic: Our approach combines tradi￾tional statistical analysis with modern large language model (LLM) capabilities to perform automated root cause analysis in mobile application testing. The system operates in three primary phases: error detection, causal ranking, and trace analysis. The RCA service first attempts to detect the presence of err… view at source ↗
Figure 5
Figure 5. Figure 5: Monthly resilience issue discovery and resolution [PITH_FULL_IMAGE:figures/full_fig_p007_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Distribution of issue resolution times. Regarding resolution times, we observed that it is much longer than typical backend or mobile bugs. Since these discoveries do not represent issues affecting customers at the time of discovery, teams may postpone fixing these issues to deliver on their quarterly commitments. Furthermore, as we mentioned earlier, to determine the appropriate and durable fix, the engin… view at source ↗
Figure 7
Figure 7. Figure 7: Service tier interaction heatmap showing resilience [PITH_FULL_IMAGE:figures/full_fig_p008_7.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

22 extracted references · 7 linked inside Pith

  1. [1]

    Kolton Andrus, Naresh Gopalani, and Ben Schmaus. 2014. FIT: Failure In- jection Testing. Netflix Tech Blog. Retrieved September 26, 2025 from https://netflixtechblog.com/fit-failure-injection-testing-35d8e2a9a3d4

  2. [2]

    Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos Engineering.IEEE Software 33, 3 (May 2016), 35–41. doi:10.1109/MS.2016.60

  3. [3]

    Ali Basiri, Lorin Hochstein, Nora Jones, and Haley Tucker. 2019. Automating Chaos Experiments in Production. InProceedings of the 41st International Con- ference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’19). IEEE, Montreal, QC, Canada, 31–40. doi:10.1109/ICSE-SEIP.2019.00012

  4. [4]

    Ishaan Chintapatla, Karthik Choji, Nikhil Agarwal, Andrew Lin, Haoxuan You, Cuong Duong, Kevin Zhu, Shawn O’Brien, and Varun Sharma. 2025. COREVQA: Scaling Mobile Chaos Testing with AI-Driven Test Execution , , A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark. arXiv:2507.13405

  5. [5]

    Jonathan David Gilbey and Carola-Bibiane Schönlieb. 2021. An End-to-End Optical Character Recognition Approach for Ultra-Low-Resolution Printed Text Images. arXiv:2105.04515

  6. [6]

    Gunawi, Mingzhe Hao, Tanakorn Do, Tiratat Joshi, Riza O

    Haryadi S. Gunawi, Mingzhe Hao, Tanakorn Do, Tiratat Joshi, Riza O. Suminto, Cesar Laksono, Andi Wijaya Satria, Jeffry Adityatama, and Kurnia J. Eliazar. 2014. What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems. In Proceedings of the ACM Symposium on Cloud Computing (SoCC ’14). ACM, Seattle, WA, USA, 1–14. doi:10.1145/2670979.2670986

  7. [7]

    Neil Gunther, Ali Basiri, and Casey Rosenthal. 2011. The Netflix Chaos Monkey. Netflix Tech Blog. Retrieved September 26, 2025 from https://netflixtechblog. com/the-netflix-chaos-monkey-891ffdd6f08d

  8. [8]

    Sofia Karlsson, Adnan Čaušević, Daniel Sundmark, Sasikumar Punnekkat, and Sigrid Eldh. 2020. Model-based Automated Testing of Mobile Applications: An Industrial Case Study. arXiv:2008.08859

  9. [9]

    Gang Li and Yang Li. 2022. Spotlight: Mobile UI Understanding using Vision- Language Models with a Focus. arXiv:2209.14927

  10. [10]

    Christopher Meiklejohn. 2021. Filibuster: Systematic Fault Injection for Resilient Microservices at DoorDash. DoorDash Engineering Blog. Retrieved September 26, 2025 from https://doordash.engineering/2021/07/28/filibuster-fault-injection

  11. [11]

    Meiklejohn, Andrea Estrada, Yiwen Song, Heather Miller, and Rohan Padhye

    Christopher S. Meiklejohn, Andrea Estrada, Yiwen Song, Heather Miller, and Rohan Padhye. 2021. Service-Level Fault Injection Testing. InProceedings of the ACM Symposium on Cloud Computing (SoCC ’21). Association for Computing Machinery, Seattle, WA, USA, 388–402. doi:10.1145/3472883.3487005

  12. [12]

    2017.Chaos Engineering: Building Confidence in System Behavior through Experiments

    Casey Rosenthal, Lorin Hochstein, and Ali Basiri. 2017.Chaos Engineering: Building Confidence in System Behavior through Experiments. O’Reilly Media, Sebastopol, CA

  13. [13]

    Matthew Rosenthal. 2019. How to Run Chaos Engineering Experiments with Gremlin. Gremlin Blog. Retrieved September 26, 2025 from https://www.gremlin. com/blog

  14. [14]

    Ahmed Samir, Hossam Maghawry, and Nagwa Badr. 2022. A Survey on Auto- mated User Interface Testing for Mobile Applications.International Journal of Intelligent Computing and Information Sciences22, 2 (2022), 1–11

  15. [15]

    Italo Santos, Júlio César Campos Filho, and Simone Rodrigues da Silva Souza

  16. [16]

    Hellerstein

    Rebecca Taft, Sanjay Ghemawat, and Joseph M. Hellerstein. 2017. Testing Span- ner’s Resilience with Chaos. Google Cloud Blog. Retrieved September 26, 2025 from https://cloud.google.com/blog/topics/developers-practitioners/testing- spanners-resilience-chaos

  17. [17]

    Uber Engineering. 2024. Generative AI for High-Quality Mobile Testing. Uber Engineering Blog. Retrieved September 26, 2025 from https://www.uber.com/ blog/generative-ai-for-high-quality-mobile-testing/

  18. [18]

    Sriram Vaidyanathan. 2021. Chaos Engineering at Lyft with Envoy and Clutch. Lyft Engineering Blog. Retrieved September 26, 2025 from https://eng.lyft.com/ chaos-engineering-at-lyft-with-envoy-and-clutch-8d59a0b43a4d

  19. [19]

    Wenyu Wang and Wei Yang. 2021. VET: Identifying and Avoiding UI Exploration Tarpits in Mobile App Testing. arXiv:2102.06377

  20. [20]

    Christoph Wick, Christian Reul, and Frank Puppe. 2018. Calamari – A High- Performance Tensorflow-based Deep Learning Package for Optical Character Recognition. arXiv:1807.02004

  21. [21]

    Hao Yu, Haonan Fang, Zhijie Du, Yiyang Ding, and Zhendong Su. 2024. Practical, Automated Scenario-based Mobile App Testing (ScenTest). arXiv:2406.08340

  22. [2020]

    InProceedings of the 2020 XLVI Latin American Computing Conference (CLEI)

    A Survey on the Practices of Mobile Application Testing. InProceedings of the 2020 XLVI Latin American Computing Conference (CLEI). IEEE, Loja, Ecuador, 232–241. doi:10.1109/CLEI52000.2020.00034

This paper was first reviewed by deepseek-v4-flash on August 3, 2026.