{"id":"f0a5e773-d0e5-40d0-a557-97c608ca36b2","arxiv_id":"2608.07775","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"A perturbation benchmark shows mobile agents are brittle to state, transition, and action changes, and a test-time recovery wrapper partially restores their performance.","lead":"AndroidReality adds 24 realistic interface perturbations to the AndroidWorld benchmark, and eight mobile agents lose 13 to 36 points of success rate when tested on them. The paper proposes a training-free recovery module that recovers some of these failures, arguing robustness should be a standard evaluation axis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The quantitative headline claims rest on one run per task with no error bars; a paired re-test could show the 13–36 point drops and TTIR gains are within noise.","rationale":"Read in good faith, the paper makes a useful organizational contribution: the MDP-grounded taxonomy is a clean way to separate state, transition, and action variability, the manual error analysis is plausible, and TTIR is a direct, training-free response to the four identified failure categories. The benchmark and implementation are promised for release, which would support reproducibility. The reader's weakest assumption—that the perturbations faithfully emulate real-world variability—is genuine but is ultimately a calibration claim that the paper does not validate against external telemetry. The more decisive and immediately checkable problem is statistical: all quantitative conclusions are point estimates from one run per task under a fixed seed, with no error bars or paired significance tests. The main direction of the finding is probably right, since all six GUI-Owl models drop under perturbation, but the specific ranges and the TTIR improvement percentages are not established by the reported evidence. The reader already conditioned the verdict on these issues; elevating the single-run concern from secondary to primary does not change the conditional verdict.","tokens_in":18972,"tokens_out":5481,"duration_ms":56469,"concrete_test":"Use the already logged per-task binary outcomes to compute exact paired statistics: for each model, a McNemar exact test on clean vs perturbed success, and for GUI-Owl-7B, McNemar tests on TTIR vs no-TTIR in both the perturbed and clean settings, each with a paired bootstrap 95% confidence interval for the per-task difference. If the interval for the 60→50 or 44→35 TTIR reduction includes zero, the recovery claim is not supported; if intervals for several per-category drops include zero, the 'uniform degradation' claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline number in Section 4.2 and Section 5.2 is computed from exactly one run per task under one fixed perturbation seed: Appendix B states 'Each task–perturbation pair is evaluated once under the fixed seed.' No confidence intervals, repeated trials, or significance tests are reported. Task success is binary, per-category subsets are as small as roughly 10–30 tasks, and VLM sampling temperature is 0.1, so sampling noise alone can produce success-rate shifts of the reported magnitude. This is load-bearing because the paper's central empirical evidence—agents degrade by 13–36 points and TTIR reduces failed tasks from 60 to 50 and 44 to 35—is entirely quantitative. The taxonomy and the qualitative error analysis are not threatened by this issue, but the headline robustness and recovery claims are not statistically anchored.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AndroidReality, a perturbation-based evaluation framework built on top of AndroidWorld. It organizes real-world interface variability into a taxonomy of state-, transition-, and action-level perturbations grounded in an MDP formulation. The authors evaluate eight open-source mobile agents under these perturbations, report substantial success-rate drops, and identify four recurring failure categories. They then propose Test-Time Introspective Recovery (TTIR), a training-free, model-agnostic mechanism that diagnoses these failure modes and applies targeted recovery strategies, and report that it reduces failed tasks in both perturbed and clean settings. The paper argues that robustness should be a standard dimension in mobile-agent evaluation and that benchmark perturbation can surface latent weaknesses.","tokens_in":19181,"tokens_out":5992,"duration_ms":48826,"significance":"If the quantitative claims are reliable, AndroidReality addresses a genuine gap: existing mobile-agent benchmarks are clean and stationary, whereas deployment requires coping with interface variability. The MDP-grounded taxonomy is principled, the perturbation configurations are parameterized and reproducible (fixed seeds, shared task-perturbation pairs, released code), and the four error categories are well illustrated with concrete trajectory examples. TTIR is training-free and model-agnostic, and the observation that it also helps in the clean setting is intriguing. However, the headline robustness and recovery numbers are not statistically anchored: results come from single runs without error bars, UI-TARS clean baselines are taken from external papers rather than rerun, and TTIR is evaluated in-sample on the same benchmark used to develop its categories. The 'real world' framing additionally rests on an unvalidated assumption that the synthetic perturbations faithfully emulate real-world variability. With stronger experimental grounding, the benchmark and taxonomy would be a useful contribution to the community.","major_comments":[{"comment":"The claim that 'every evaluated agent suffers substantial degradation under perturbation, with absolute drops of 13–36 points on overall success rate' is contradicted by the table itself: UI-TARS-7B-SFT drops from 33.00 to 25.86, a 7.14-point drop, while UI-TARS-1.5-7B drops 36.61 points. The actual range across the table is 7.14 to 36.61, not 13 to 36, and the statement that degradation is 'uniformly across model families' is not supported by this exception. Because this range appears in the central observation of the paper, the text should be corrected or qualified to reflect the data.","section":"Section 4.2, Table 2"},{"comment":"Appendix B states that 'Each task–perturbation pair is evaluated once under the fixed seed.' All headline numbers in Sections 4.2 and 5.2 are therefore single-run success rates with no confidence intervals or significance tests. Task success is binary, per-category subsets can be as small as a handful of tasks (see Table 5 and Figures 6–7, where several panels show denominators of roughly 5–15 tasks), and VLM sampling temperature is 0.1, so sampling noise alone can produce success-rate shifts of the reported magnitude. Please report repeated trials or binomial confidence intervals, and explicitly state the number of tasks behind each perturbation profile.","section":"Section 4.1 and Appendix B (execution protocol)"},{"comment":"The clean success rates for UI-TARS-7B-SFT and UI-TARS-1.5-7B are marked with a dagger indicating they are taken from the original paper, while the perturbed numbers are produced by the authors' harness. Since the clean baselines were not rerun under the same serving conditions, perturbation wrapper, and evaluation protocol, the computed drops (e.g., -36.61 for UI-TARS-1.5-7B) conflate environment differences with perturbation effects. Please rerun these models on clean AndroidWorld in the same harness, or explicitly label the comparison as cross-paper and avoid making absolute drop claims for these rows.","section":"Table 2 (UI-TARS rows)"},{"comment":"TTIR is evaluated on a single model (GUI-Owl-7B), with no repeated trials, no significance test, and no comparison against a simpler baseline (e.g., a generic 'try a different action' heuristic or a random re-grounding strategy). Moreover, the four error categories and TTIR's detection/recovery strategies were derived from the same AndroidReality trajectories on which TTIR is evaluated (Section 4.3 feeds into Section 5.1), so the reported improvements from 60 to 50 and 44 to 35 failed tasks are in-sample. Please evaluate on held-out perturbation types or a second benchmark, and compare against a non-introspective recovery baseline, to support the claim that TTIR recovers failures rather than overfitting the diagnosed categories.","section":"Section 5.2, Figure 4"},{"comment":"The 'real world' conclusion rests on the assumption that the injected perturbations faithfully emulate real-world interface variability while preserving task semantics and reward. The paper provides no validation of this assumption, such as comparison with real device logs, deployment telemetry, crowdsourced real-world interaction data, or user studies. The results support the conclusion that agents are brittle to these specific controlled perturbations, but not directly the claim that they are 'far from the real world.' Please add validation tying the perturbation distributions to measured real-world conditions, or soften the real-world claims in the title and abstract.","section":"Section 3.1 (Eq. 1–2) and title"}],"minor_comments":[{"comment":"The sentence 'we constructed AndroidReality, it is an online mobile benchmark' should be revised to 'we constructed AndroidReality, an online mobile benchmark' (remove 'it is').","section":"Section 6 (first sentence)"},{"comment":"The default stuck-loop detection threshold is inconsistent: Section 5.1 states 'the last N steps (default N=3)', while Appendix D.1 says 'the last N steps (default N=5)' for the deterministic hard-stuck rule. Please harmonize these values or explain the discrepancy.","section":"Section 5.1 vs Appendix D.1"},{"comment":"The captions use 'Clean-before' and 'Perturbed-after' but it would be clearer to state explicitly that the clean condition is the same task subset without the wrapper, and the perturbed condition applies one perturbation profile per task.","section":"Figures 6 and 7 captions"},{"comment":"For the transition-level perturbations, the 'Tunable Parameters' column lists generic entries such as 'Surface style', 'Blocking', and 'Auto-dismiss duration' without default values; providing defaults or a pointer to Table 4 would improve reproducibility.","section":"Table 3"},{"comment":"The sentence 'The top-three challenging types are in Table 5' appears before Table 5 is referenced; consider adding a forward pointer or reordering the text so the reader can locate the table.","section":"Section 4.2"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and taxonomy are valuable and likely of interest to the mobile-agent community, but the quantitative support for the headline claims needs substantial strengthening. The most serious scientific risk is the in-sample evaluation of TTIR: the error categories and recovery strategies are developed and evaluated on the same benchmark, and the single-run evaluation without error bars leaves the magnitude of the improvement uncertain. The internal inconsistency in the reported drop range (13–36 points vs. the table's 7.14–36.61) also suggests that table-text consistency checks are needed. I would be willing to review a revised version that addresses these points."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Know this paper as a useful, reusable perturbation benchmark with a clean MDP-based taxonomy (state/transition/action) and 24 parameterized perturbations built on AndroidWorld. The qualitative finding—agents of every size and family drop sharply when the interface deviates from the canonical layout—is credible and consistent with prior robustness work. The four error categories (stale-coordinate stuck, silent action failures, off-task engagement, goal-memory drift) are well illustrated and feel right. That part is a genuine contribution.\n\nThe soft spots are exactly where you'd expect. Every headline number in Sections 4.2 and 5.2 comes from one run per task under a fixed seed, with no error bars or repeated trials. Some perturbation categories have only 5-15 tasks, so a single task flip moves success by several points. Sampling temperature is 0.1, so it's not fully deterministic; the reported 13-36 point drops could shift by ±10 even on rerun. The UI-TARS clean numbers are taken from the original paper rather than rerun, which is inconsistent with the other models. That matters because the drops for UI-TARS-1.5-7B (-36.61) are the largest headline.\n\nTTIR is a reasonable idea, but the evidence for it is thin: evaluated on one model (GUI-Owl-7B), with no significance testing, and the four detection heuristics were designed from the same benchmark's error analysis, so the improvement is in-sample. The clean-setting gain (44 to 35 failures) is suggestive but could partly be the diagnoser correcting ordinary mistakes, which is fine, but it doesn't validate generalization to other models or real deployments.\n\nThe 'real world' framing is the weakest wording, not the core of the work. Synthetic perturbations are a sensible first proxy, but 'how far from the real world' suggests a validated distance that the paper doesn't measure. A title and conclusion more honest to the method would say 'under interface perturbation.'\n\nBottom line: the benchmark is worth having, the taxonomy is useful, and the qualitative brittleness story will likely hold up. But the quantitative claims should be re-anchored with repeated runs or confidence intervals, UI-TARS rerun under the same protocol, and TTIR tested on at least a second model. I'd send it to peer review, not desk-reject, with a strong request for revision. If the authors add the runs, this becomes a solid reference benchmark for the mobile-agent robustness subfield. I'd probably cite the benchmark once it's published; I'm not citing the TTIR numbers as they stand.","headline":"Useful perturbation taxonomy and benchmark with a credible qualitative brittleness story, but the headline numbers rest on single runs and the recovery mechanism is only demonstrated in-sample on one model.","tokens_in":19661,"tokens_out":2346,"would_cite":true,"duration_ms":20752,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Mobile agents that score well on clean Android benchmarks lose 13 to 36 percentage points in success rate when realistic interface variations are injected, a gap this paper attributes to four recurring failure modes and partially closes…","keywords":["mobile agents","robustness evaluation","perturbation benchmark","Markov decision process","test-time introspection","GUI agents","Android emulator","error analysis"],"falsifier":"Collect a corpus of naturally occurring real-world Android sessions that include the same variations (dark mode, pop-ups, loading delays, dropped taps) and run the same eight agents on those sessions; if the real-world success-rate drop is much smaller or larger than the 13-to-36-point drop measured under AndroidReality, the benchmark's estimate of the deployment gap is wrong. A cheaper check: rerun the benchmark with multiple seeds and repeated trials to see whether the per-category drops are stable or within noise.","tokens_in":1665,"feed_emoji":"📱","tokens_out":4449,"duration_ms":88109,"temperature":0.7,"pith_summary":"Mobile agents score well on clean online benchmarks, but that success may be misleading. This paper establishes robustness as a separate evaluation dimension by wrapping a standard Android task benchmark with a perturbation layer that injects realistic interface variability along three axes: the observed state (theme, font size, orientation, density), the transition (pop-ups, notifications, consent sheets), and the action (delays, drops, freezes). Under these perturbations, all eight tested agents lose 13 to 36 percentage points in overall success rate, with state-level visual changes causing the most uniform damage. A manual failure analysis attributes most failures to four recurring patterns, and a training-free Test-Time Introspective Recovery module that checks for these patterns at each step reduces failure counts by 17 percent on perturbed tasks and 20 percent on clean tasks. The upshot is that benchmark perturbation can expose latent weaknesses that clean evaluation hides, and that some of them are addressable without retraining.","feed_headline":"Mobile agents lose 13-36 points on realistic interface changes","feed_subtitle":"Clean-benchmark scores overestimate real-world readiness; a training-free fix recovers some failures.","key_machinery":"The load-bearing mechanism is the perturbation wrapper defined in equations (1)-(2): it takes the canonical MDP $M_0$ and applies operators $\\phi^S_\\theta$, $\\phi^A_\\theta$, and $\\tilde{T}_\\theta$ to produce a perturbed MDP that shares reward and task semantics, yielding a taxonomy (state, transition, action) that allows failure attribution by MDP component. The second mechanism is the TTIR diagnose-then-recover loop: a diagnoser that inspects the screenshot plus action history for four signals (stuck loop, unsatisfied action effect, interference, task misalignment) and a recovery module that dispatches a targeted corrective action before the agent continues.","core_discovery":"On the paper's own terms, the discovery is that current mobile agents solve clean Android tasks well but are brittle to interface variations that are ordinary in real use: a perturbation wrapper that keeps the task goal and success verifier fixed while altering the observation, the transition, or the action execution converts the canonical environment into a family of perturbed Markov decision processes, and every evaluated agent's success rate falls substantially, by 13 to 36 points. The degradation is not random: state-level perturbations that change visual layout produce the largest and most uniform drops, and the failures cluster into four categories — stale-coordinate stuck loops, silent action failures, off-task engagement with distractions, and goal-memory drift after a distraction is dismissed. The same four categories also occur in clean settings, which the paper reads as evidence that these are latent weaknesses rather than artifacts of perturbation. Finally, the paper shows that a training-free, model-agnostic introspection loop that detects the four failure signs and launches targeted recovery actions reduces failed tasks on both perturbed and clean settings, without modifying the agent's underlying policy.","pith_inferences":["If the realism assumption holds, the same perturbation methodology transfers to other GUI settings (web, desktop) and other task distributions, where the taxonomy may surface different recurring failure categories.","TTIR's gains on clean tasks hint that the four failure categories are a general diagnostic vocabulary for agent behaviour, not a perturbation-specific fix; a natural extension would be to train agents to internalize the verification and re-grounding steps so the recovery costs fewer extra steps.","The step overhead and budget effects imply that robustness should be measured not only in success rate but in step efficiency; the paper's delta-step metric invites a cost-benefit analysis of recovery strategies."],"forward_implications":["Success-rate comparisons on clean benchmarks should be supplemented by a robustness score under controlled perturbations; the paper's taxonomy provides a way to attribute failures to a specific MDP component.","Simply scaling models or adding post-training does not close the robustness gap: losses are comparable across model families and sizes from 2B to 32B.","State-level visual perturbations (theme, density, orientation) are the most uniformly damaging, so visual re-grounding is a priority target.","Even agents that correctly dismiss a pop-up can lose the original task goal, implying that memory of the goal matters as much as distraction detection.","A training-free introspection loop can recover a non-trivial fraction of failures on both perturbed and clean tasks, suggesting that explicit verification is a low-cost intervention."],"supporting_citations":[{"why":"Supplies the canonical Android task environment, task definitions, and success verifier that AndroidReality perturbs.","marker":"[9]"},{"why":"Provides the GUI-OWL agents whose clean and perturbed success rates anchor the measured robustness gap.","marker":"[5]"},{"why":"Provides the GUI-OWL-1.5 agents used in the evaluation, covering larger-scale and post-trained models.","marker":"[6]"},{"why":"Provides the UI-TARS and UI-TARS-1.5 agents, the other evaluated model family.","marker":"[4]"},{"why":"Earlier robustness-oriented benchmark for GUI agents under naturalistic pop-ups, giving a comparison point for the paper's coverage.","marker":"[12]"},{"why":"Earlier dynamic robustness framework for GUI agents under real-world anomalies, providing a baseline for transition-level perturbation results.","marker":"[13]"}],"fun_headline_variants":["Mobile agents drop 13-36 points on realistic interface changes","Perturbation tests expose mobile agents' hidden fragility","Training-free introspection rescues mobile agents on perturbed tasks","Benchmarking mobile agents: real-world variability costs 13-36 points","Android agents fail under realistic interface perturbations, study finds"],"cache_read_input_tokens":21888,"weakest_assumption_plain":"The central assumption is that the injected perturbations faithfully reproduce the range of ordinary real-world interface variability while preserving the task's meaning; if the injected glitches are unrepresentative or accidentally change what counts as success, the measured drops do not tell us how far agents are from the real world.","fun_headline_variants_meta":{"raw":{"variants":["Mobile agents drop 13-36 points on realistic interface changes","Perturbation tests expose mobile agents' hidden fragility","Training-free introspection rescues mobile agents on perturbed tasks","Benchmarking mobile agents: real-world variability costs 13-36 points","Android agents fail under realistic interface perturbations, study finds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000172,"raw_usage":{"total_tokens":1266,"prompt_tokens":928,"completion_tokens":338,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":253}},"tokens_in":544,"tokens_out":338,"duration_ms":3805,"temperature":1.0,"reasoning_tokens":253,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:12:45.319850+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect a corpus of naturally occurring real-world Android sessions that include the same variations (dark mode, pop-ups, loading delays, dropped taps) and run the same eight agents on those sessions; if the real-world success-rate drop is much smaller or larger than the 13-to-36-point drop measured under AndroidReality, the benchmark's estimate of the deployment gap is wrong. A cheaper check: rerun the benchmark with multiple seeds and repeated trials to see whether the per-category drops are stable or within noise.","supporting_citations":[{"cited_title":"Androidworld: A dynamic benchmarking environment for autonomous agents","cited_arxiv_id":null,"evidence_quote":"Supplies the canonical Android task environment, task definitions, and success verifier that AndroidReality perturbs."},{"cited_title":"Gui-robust: A comprehensive dataset for testing gui agent robustness in real-world anomalies","cited_arxiv_id":null,"evidence_quote":"Earlier robustness-oriented benchmark for GUI agents under naturalistic pop-ups, giving a comparison point for the paper's coverage."},{"cited_title":"D-gara: A dynamic benchmarking framework for gui agent robustness in real-world anomalies","cited_arxiv_id":null,"evidence_quote":"Earlier dynamic robustness framework for GUI agents under real-world anomalies, providing a baseline for transition-level perturbation results."}],"review_version":1}