REVIEW 4 major objections 9 minor 22 references
An AI-driven mobile tester combined with fault injection scales chaos testing to 180,000 production runs.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 03:58 UTC pith:VHBMRQBN
load-bearing objection Genuine industrial integration paper with real operational scale; worth serious review, with caveats about in-sample RCA evaluation and triage ground truth. the 4 major comments →
Scaling Mobile Chaos Testing with AI-Driven Test Execution
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The central claim is that adaptive AI-driven mobile test execution—which reads screens semantically and chooses actions by goal rather than fixed selectors—remains reliable enough under injected backend faults to serve as the primary observation point for chaos engineering. Combined with safe, header-based fault injection at the RPC layer, this yields a system that can validate mobile resilience continuously in production, achieving 99%+ pass rates under faults and 88% precision@5 in automated root-cause attribution to backend services.
What carries the argument
The load-bearing mechanism is the pairing of DragonCrawl, an LLM-based crawler that converts screens into text and frames test goals and assertions as natural-language questions so it can adapt to degraded UI states, with uHavoc, which injects abort, timeout, and latency faults into backend RPCs via test-tenancy headers. A baseline control run without faults isolates injected failures from environmental noise, and an automated root-cause pipeline ranks candidate RPCs using a weighted causal score tying error screens to network requests.
Load-bearing premise
The labeling of the ~1,300 non-passing runs as genuine app resilience problems rather than LLM navigation failures under degraded screens is not validated against an independent ground-truth set.
What would settle it
Have human experts replay a random sample of failed runs and classify each as an app resilience defect or an agent navigation failure; if more than a few percent of non-passing runs are navigation failures mislabeled as resilience issues, the 23-issue count and 99.27% pass rate are overstated.
If this is right
- If the claims hold, mobile chaos testing can run nightly at scale without per-test scripting across flows, cities, and fault scenarios.
- The 23 discovered issues—12 blocking trips or food orders and 2 causing app crashes—would have become production incidents without proactive detection.
- The two crash-level issues demonstrate that some resilience defects are observable only through the mobile UI, not through backend testing alone.
- Automated root-cause analysis reduces debugging from hours to minutes, expanding the pool of engineers who can triage resilience failures.
- The 99.27% pass rate under continuous nightly execution suggests the system is safe enough for routine production-adjacent operation.
Where Pith is reading between the lines
- The same integration pattern—goal-based semantic UI agents as resilience sensors plus service-level fault injection—could transfer to other large-scale mobile ecosystems with mature tracing and service tier tagging.
- The roughly 1,300 non-passing runs (0.73% of 180,000) are not independently validated against a ground-truth set, so the true count of resilience defects may shift if some AI navigation failures are mislabeled as app issues.
- The faster resolution of teams assigned multiple tickets hints that organizational ownership dynamics, not just the detection technology, drive realized resilience gains.
- Extending fault injection beyond RPC-level errors to infrastructure and third-party failures (payments, maps) could uncover additional UX-level risks, but would require tighter safety controls.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports on an industrial system that couples DragonCrawl, an LLM-driven mobile UI test executor, with uHavoc, a service-level fault injection platform, to perform chaos testing on Uber's Rider, Driver, and Eats apps. The claims include: more than 180,000 automated chaos test executions across 47 critical flows since Q1 2024; discovery of 23 resilience risks, of which 12 blocked core flows and 2 caused app crashes; 99%+ test pass rate under fault injection; and automated root-cause analysis achieving 88% precision@5. The paper also describes system architecture, implementation challenges, operational results, and lessons learned. The central operational narrative is plausible and internally consistent, but the quantitative evaluation has important gaps in statistical reporting, validation methodology, and the separation of AI navigation failures from genuine resilience defects.
Significance. If substantiated, this is a valuable industrial experience report: it demonstrates a concrete way to overcome the combinatorial explosion of mobile chaos testing by combining an adaptive LLM-based test driver with backend fault injection, and it surfaces UX-level resilience defects that backend-only testing misses. The strengths of the paper are the real production scale (180k+ runs), the explicit safety model (test tenancy isolation), and the attempt to automate root-cause attribution. However, the headline numbers—23 issues, 88% RCA precision@5, 99%+ pass rate—are not yet backed by the evidence presented. The paper would make a stronger contribution if the evaluation included statistical tests, a held-out validation protocol for the RCA weights, and an independent audit of the failure-to-resilience attribution.
major comments (4)
- [§5.3, Table 3] The pass-rate comparison between baseline (0.993) and T2 fault injection (0.99) is reported as a degradation, but no confidence intervals, failure counts, or significance tests are given. With 300 executions per scenario, the difference corresponds to roughly one additional failure in 300 runs. The paper should report exact counts, binomial confidence intervals, or a test of equivalence; otherwise the claim of a 'drop' for T2 is not statistically supported. The same applies to the precision@k differences in Table 1 and latency differences in Table 2, where the per-scenario sample sizes are not clearly connected to the tables.
- [§3.2 and §5.5, Table 5] The RCA precision@k results are computed without any described held-out evaluation. The Score(causal) weights—F(status_code), F(tier), and F(category)—are stated to be 'established through empirical observation' (§3.2), and no separate test set, temporal split, or cross-validation is described. As written, precision@5 = 0.88 could reflect in-sample fit rather than predictive performance. The paper also does not define the ground-truth labeling process for the RCA evaluation (who annotated the true root cause and with what inter-rater reliability) nor report the number of cases evaluated. Without this, the '88% precision@5' claim is not interpretable.
- [§4 and §6.1] The central discovery claim—23 genuine resilience risks—depends on the assertion that failures under fault injection are attributable to backend resilience issues rather than to DragonCrawl's own navigation failures. Section 4 states that AI-related failures account for 0.4% of executions and are 'easily distinguishable from genuine resilience risks,' but no operational criteria, manual audit, or inter-rater reliability is provided for that distinction. With roughly 1,300 non-passing runs out of 180,000, even a small misclassification rate could materially affect the 23-issue count and the '2 crashes' claim. The 'golden screen' argument in §5.4 is not a sufficient control: both a genuine backend failure and an AI navigation failure can prevent the goal screen from appearing. The authors should provide an independent validation set—e.g., a blind manual triage of a random sample of failure
- [§5.4] The sentence 'The 23 issues we found—12 blocking trips/orders, 2 causing crashes—show this approach captures genuine failures rather than masking them' uses the very finding that needs validation as evidence for the validation. This is circular. The issue count and severity distribution are the quantities to be established, not proof of test fidelity. Please separate the design argument for golden-screen assertions from the empirical evidence that mislabeling does not occur.
minor comments (9)
- [Abstract and §1] The paper alternates between '61 critical user flows' (Introduction) and '47 critical flows' (Abstract, §6). The relationship between the two numbers is explained only in §6.2; please clarify earlier.
- [§5, Table 1] In Table 1, Precision@1 is higher than Precision@2 and Precision@3 for all scenarios. This is possible in a single-relevant-item retrieval setting, but the metric definition should be stated explicitly to avoid confusion.
- [§3.2] The term 'normalFailureRate' is used in the Score(causal) formula but never precisely defined. Is it the fraction of failing requests in baseline runs, or the deviation from a baseline error rate? Please provide the exact definition.
- [§5.3, Table 3] The pass-rate table reports only four significant digits; with 300 runs, the standard error is about 0.005, so the reported values are at the edge of measurement precision. Include raw counts.
- [§4] 'The LLM entered problematic loops 2% of executions' and 'These failures account for 0.4%' are two different percentages; clarify the relationship (e.g., loops that remain unresolved after cycle detection).
- [§6.1] The claim that the 23 issues 'were not detected through other means' needs a concrete basis: which other systems were run against the same flows and on what schedule? Without this, the reader cannot assess the claim of complementarity with backend testing.
- [Figure 5] Typo: 'As the the dotted trendline shows' should be 'As the dotted trendline shows.'
- [§5.4] Typo: 'LlaVa-NeXT' should be 'LLaVA-NeXT'.
- [§6.2] 'This represents 39,000 hours of automated manual work that has been automated away' is redundant; consider rephrasing to '39,000 hours of manual testing effort that has been replaced by automation.'
Circularity Check
No demonstrated circularity: the paper's central claims rest on in-paper experiments and operational counts; the RCA precision@k and 23-issue triage are validation concerns, not reductions to fitted inputs.
full rationale
The paper is an operational experience report whose main claims—DragonCrawl can navigate degraded screens, the integrated system sustains ~99% pass rates under fault injection, RCA attributes failures to backend RPCs with precision@k, and 23 resilience risks were found in production—are supported by in-paper measurements (Tables 1–5) and operational counts, not by a derivation that reduces to its own inputs. The most plausible circularity candidate is the RCA heuristic: Section 3.2 says the Score(causal) parameter values "were established through empirical observation," and Section 5.5 reports precision@k without explicitly describing a held-out set. If the same incidents used to set the weights were also used to compute Table 5, the precision numbers would be in-sample. However, the paper does not state that the evaluation was constructed that way, and the claim that the parameters "have proven consistent across daily execution of all 47 core flows" suggests ongoing operational confirmation. The absence of a described train/test split is an evaluation-reporting gap, not a demonstrated circularity under the evidence rule. The self-citation to Uber's DragonCrawl blog [17] is used for background and metric framing, while Tables 1–3 independently measure DragonCrawl's precision, latency, and pass rate, so the citation is not load-bearing. The attribution of failures to "genuine resilience risks" rather than LLM navigation failures is a validity threat—0.4% of executions are AI-caused and not triaged, and no independent audit is described—but this is a labeling/validation concern, not a circular derivation: the 23-issue count is an operational triage outcome rather than a prediction forced by construction. Overall, no circular step reaches the evidentiary bar required here.
Axiom & Free-Parameter Ledger
free parameters (3)
- F(status_code) weights =
1.0 (5xx), 0.5 (4xx), 0.2 (2xx)
- F(tier) weights =
1.0 (tier-0), 0.9 (tier-1), 0.7 (tier-2), 0.4 (tier-3), 0.1 (tier-4/5)
- F(category) semantic weights =
3.0 (direct), 2.0 (indirect), 1.2 (supporting), 0.3 (unrelated)
axioms (5)
- domain assumption Service tier tagging correctly reflects production criticality of every service.
- domain assumption Reaching the test's end-state 'golden screen' implies the critical user flow remained functional under fault injection.
- domain assumption uHavoc's header-based fault injection affects only test tenancy requests and does not alter production traffic.
- domain assumption Jaeger distributed tracing coverage is complete enough for attribution to specific backend services.
- domain assumption GPT-4o's screen understanding and RCA classifications are accurate enough to serve as ground truth.
Cite this review
Pith. "Pith review of Scaling Mobile Chaos Testing with AI-Driven Test Execution." pith.science (2026). https://pith.science/paper/VHBMRQBN
@misc{pith2026260206223,
author = {Pith},
title = {Pith review of: Scaling Mobile Chaos Testing with AI-Driven Test Execution},
year = {2026},
howpublished = {\url{https://pith.science/paper/VHBMRQBN}},
note = {Machine review of arXiv:2602.06223}
}
read the original abstract
Mobile applications in large-scale distributed systems are susceptible to backend service failures, yet traditional chaos engineering approaches cannot scale mobile testing due to the combinatorial explosion of flows, locations, and failure scenarios that need validation. We present an automated mobile chaos testing system that integrates DragonCrawl, an LLM-based mobile testing platform, with uHavoc, a service-level fault injection system. The key insight is that adaptive AI-driven test execution can navigate mobile applications under degraded backend conditions, eliminating the need to manually write test cases for each combination of user flow, city, and failure type. Since Q1 2024, our system has executed over 180,000 automated chaos tests across 47 critical flows in Uber's Rider, Driver, and Eats applications, representing approximately 39,000 hours of manual testing effort that would be impractical at this scale. We identified 23 resilience risks, with 70% being architectural dependency violations where non-critical service failures degraded core user flows. Twelve issues were severe enough to prevent trip requests or food orders. Two caused application crashes detectable only through mobile chaos testing, not backend testing alone. Automated root cause analysis reduced debugging time from hours to minutes, achieving 88% precision@5 in attributing mobile failures to specific backend services. This paper presents the system design, evaluates its performance under fault injection (maintaining 99% test reliability), and reports operational experience demonstrating that continuous mobile resilience validation is achievable at production scale.
Figures
Reference graph
Works this paper leans on
-
[1]
Kolton Andrus, Naresh Gopalani, and Ben Schmaus. 2014. FIT: Failure In- jection Testing. Netflix Tech Blog. Retrieved September 26, 2025 from https://netflixtechblog.com/fit-failure-injection-testing-35d8e2a9a3d4
2014
-
[2]
Ali Basiri, Niosha Behnam, Ruud de Rooij, Lorin Hochstein, Luke Kosewski, Justin Reynolds, and Casey Rosenthal. 2016. Chaos Engineering.IEEE Software 33, 3 (May 2016), 35–41. doi:10.1109/MS.2016.60
-
[3]
Ali Basiri, Lorin Hochstein, Nora Jones, and Haley Tucker. 2019. Automating Chaos Experiments in Production. InProceedings of the 41st International Con- ference on Software Engineering: Software Engineering in Practice (ICSE-SEIP ’19). IEEE, Montreal, QC, Canada, 31–40. doi:10.1109/ICSE-SEIP.2019.00012
arXiv 2019
-
[4]
Ishaan Chintapatla, Karthik Choji, Nikhil Agarwal, Andrew Lin, Haoxuan You, Cuong Duong, Kevin Zhu, Shawn O’Brien, and Varun Sharma. 2025. COREVQA: Scaling Mobile Chaos Testing with AI-Driven Test Execution , , A Crowd Observation and Reasoning Entailment Visual Question Answering Benchmark. arXiv:2507.13405
Pith/arXiv arXiv 2025
-
[5]
Jonathan David Gilbey and Carola-Bibiane Schönlieb. 2021. An End-to-End Optical Character Recognition Approach for Ultra-Low-Resolution Printed Text Images. arXiv:2105.04515
Pith/arXiv arXiv 2021
-
[6]
Gunawi, Mingzhe Hao, Tanakorn Do, Tiratat Joshi, Riza O
Haryadi S. Gunawi, Mingzhe Hao, Tanakorn Do, Tiratat Joshi, Riza O. Suminto, Cesar Laksono, Andi Wijaya Satria, Jeffry Adityatama, and Kurnia J. Eliazar. 2014. What Bugs Live in the Cloud? A Study of 3000+ Issues in Cloud Systems. In Proceedings of the ACM Symposium on Cloud Computing (SoCC ’14). ACM, Seattle, WA, USA, 1–14. doi:10.1145/2670979.2670986
arXiv 2014
-
[7]
Neil Gunther, Ali Basiri, and Casey Rosenthal. 2011. The Netflix Chaos Monkey. Netflix Tech Blog. Retrieved September 26, 2025 from https://netflixtechblog. com/the-netflix-chaos-monkey-891ffdd6f08d
2011
-
[8]
Sofia Karlsson, Adnan Čaušević, Daniel Sundmark, Sasikumar Punnekkat, and Sigrid Eldh. 2020. Model-based Automated Testing of Mobile Applications: An Industrial Case Study. arXiv:2008.08859
Pith/arXiv arXiv 2020
-
[9]
Gang Li and Yang Li. 2022. Spotlight: Mobile UI Understanding using Vision- Language Models with a Focus. arXiv:2209.14927
Pith/arXiv arXiv 2022
-
[10]
Christopher Meiklejohn. 2021. Filibuster: Systematic Fault Injection for Resilient Microservices at DoorDash. DoorDash Engineering Blog. Retrieved September 26, 2025 from https://doordash.engineering/2021/07/28/filibuster-fault-injection
2021
-
[11]
Meiklejohn, Andrea Estrada, Yiwen Song, Heather Miller, and Rohan Padhye
Christopher S. Meiklejohn, Andrea Estrada, Yiwen Song, Heather Miller, and Rohan Padhye. 2021. Service-Level Fault Injection Testing. InProceedings of the ACM Symposium on Cloud Computing (SoCC ’21). Association for Computing Machinery, Seattle, WA, USA, 388–402. doi:10.1145/3472883.3487005
arXiv 2021
-
[12]
2017.Chaos Engineering: Building Confidence in System Behavior through Experiments
Casey Rosenthal, Lorin Hochstein, and Ali Basiri. 2017.Chaos Engineering: Building Confidence in System Behavior through Experiments. O’Reilly Media, Sebastopol, CA
2017
-
[13]
Matthew Rosenthal. 2019. How to Run Chaos Engineering Experiments with Gremlin. Gremlin Blog. Retrieved September 26, 2025 from https://www.gremlin. com/blog
2019
-
[14]
Ahmed Samir, Hossam Maghawry, and Nagwa Badr. 2022. A Survey on Auto- mated User Interface Testing for Mobile Applications.International Journal of Intelligent Computing and Information Sciences22, 2 (2022), 1–11
2022
-
[15]
Italo Santos, Júlio César Campos Filho, and Simone Rodrigues da Silva Souza
-
[16]
Hellerstein
Rebecca Taft, Sanjay Ghemawat, and Joseph M. Hellerstein. 2017. Testing Span- ner’s Resilience with Chaos. Google Cloud Blog. Retrieved September 26, 2025 from https://cloud.google.com/blog/topics/developers-practitioners/testing- spanners-resilience-chaos
2017
-
[17]
Uber Engineering. 2024. Generative AI for High-Quality Mobile Testing. Uber Engineering Blog. Retrieved September 26, 2025 from https://www.uber.com/ blog/generative-ai-for-high-quality-mobile-testing/
2024
-
[18]
Sriram Vaidyanathan. 2021. Chaos Engineering at Lyft with Envoy and Clutch. Lyft Engineering Blog. Retrieved September 26, 2025 from https://eng.lyft.com/ chaos-engineering-at-lyft-with-envoy-and-clutch-8d59a0b43a4d
2021
-
[19]
Wenyu Wang and Wei Yang. 2021. VET: Identifying and Avoiding UI Exploration Tarpits in Mobile App Testing. arXiv:2102.06377
Pith/arXiv arXiv 2021
-
[20]
Christoph Wick, Christian Reul, and Frank Puppe. 2018. Calamari – A High- Performance Tensorflow-based Deep Learning Package for Optical Character Recognition. arXiv:1807.02004
Pith/arXiv arXiv 2018
-
[21]
Hao Yu, Haonan Fang, Zhijie Du, Yiyang Ding, and Zhendong Su. 2024. Practical, Automated Scenario-based Mobile App Testing (ScenTest). arXiv:2406.08340
Pith/arXiv arXiv 2024
-
[2020]
InProceedings of the 2020 XLVI Latin American Computing Conference (CLEI)
A Survey on the Practices of Mobile Application Testing. InProceedings of the 2020 XLVI Latin American Computing Conference (CLEI). IEEE, Loja, Ecuador, 232–241. doi:10.1109/CLEI52000.2020.00034
arXiv 2020
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.