REVIEW 3 major objections 6 minor 24 references
GUI agents often fail not from wrong reasoning but from decoding too late; pre-compiling a small policy tree off the critical path recovers contested windows.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-31 08:38 UTC pith:FGTFXX7K
load-bearing objection Solid causal isolation of decode-on-critical-path for contested GUI windows; novelty is the controlled manipulation and pre-registered evidence, not the tree machinery itself. the 3 major comments →
Why Are GUI Agents Correct but Late? Decode on the Decision-Time Critical Path, Tested with Pre-Compiled Policy Trees
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Holding model, task, and information fixed, removing full autoregressive generation from the event-time critical path recovers otherwise-missed transient GUI actions. On a contested window AAPT raises paired success from 0.50 to 0.79 (exact McNemar p=1.8e-3) with no incorrect actions; baselines that still decode at decision time remain at zero. The effect follows a latency-coverage sizing rule, replicates on an untuned same-class generalist, and appears only where fast observer decoding, valid tree planning, and accurate branch routing all hold.
What carries the argument
Adaptive Anticipatory Policy Trees (AAPT): during idle periods the frozen model compiles a flat conditional tree of observable guards, pre-authorized allowlisted actions, and branch deadlines sized so coverage time meets planner p95 latency plus a safety margin; at event time a change gate plus low-token observer routes one frame to a branch and executes without generation.
Load-bearing premise
During a quiet prep interval long enough to cover the planner’s own latency, the true outcomes and actions can be enumerated in advance as schema-valid guards with pre-authorized actions; if the right response is revealed only at fire time, the tree cannot bind it.
What would settle it
A paired contested-window trial in which candidate actions are enumerable and prep time exceeds measured planner latency plus margin, yet AAPT still fails to beat a live reactive baseline once observer routing is accurate and trees are valid—or conversely succeeds when outcomes cannot be pre-enumerated.
If this is right
- Missed transient GUI events can be treated as a scheduling failure rather than a pure reasoning failure.
- Anticipation becomes a pipeline property: valid tree, true outcome covered, correct route, route before deadline.
- A hybrid controller should first decide whether the next transition is pre-enumerable, then choose AAPT or reactive execution.
- Branch budget must match the enumerable outcome-set size; mis-setting it converts misses into wrong actions.
- Reproducing thresholded agent results requires freezing both functional identity (kernels, schemas) and regime identity (latency quantiles, contested windows).
Where Pith is reading between the lines
- Distilled or template-based guard matchers could close much of the remaining oracle-to-live routing gap without raising planning cost.
- The same critical-path logic likely applies to any embodied or tool-using agent facing irreversible sub-second windows, not only GUIs.
- Long-lived skill memories could amortize AAPT preparation by promoting repeatedly validated sub-trees, complementary to one-shot contingency trees.
- Serving non-stationarity of only ~100 ms can flip a window from contested to ceiling, so latency monitoring belongs in the evaluation contract itself.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper argues that many GUI-agent failures on transient events are scheduling failures: correct actions arrive after the window closes because full autoregressive decoding sits on the decision-time critical path. Holding model, task, and information fixed, it introduces Adaptive Anticipatory Policy Trees (AAPT): during idle periods the same frozen multimodal model compiles a bounded flat policy tree (observable guards, pre-authorized actions, branch deadlines) sized by T_cover ≥ L_p95 + M; at event time a low-token observer routes a change-gated frame and fires a prepared action with no free-form generation. On a custom contested-window benchmark, paired exact McNemar tests raise success from 0.50 to 0.79 at 650 ms (p=1.8e-3, zero incorrect actions at matched budget), while open-loop and predict-and-replan baselines that still decode at decision time score 0/42. The effect tracks a prep-budget crossover, a three-gate capability matrix (fast observer, valid flat trees, routing ≥0.85), and replicates on an untuned same-class generalist (126 pairs, p=4.9e-13). On DynaCU-Bench the aggregate is a tie with complementary category strengths when actions are pre-enumerable.
Significance. If the result holds in the stated regime, it cleanly separates anticipation-as-pipeline from anticipation-as-model-trait and gives a falsifiable systems criterion (latency coverage, schema-valid trees, live routing) rather than another agent architecture. Strengths that raise the bar for this literature include: pre-registered endpoints and a prediction ledger that retains failed predictions at full weight (notably the UI-Venus oracle probe); paired per-seed exact McNemar tests; mechanism baselines R1/P1 at zero that rule out “act early” alone; quantitative agreement of the prep sweep with Eq. (1); and an honest transfer boundary on DynaCU. The work is significant for computer-use agents, speculative/anticipatory planning, and latency-critical multimodal systems, even though scope is deliberately narrow (enumerable outcomes, quiet prep ≥ L_p95+M, primarily Qwen3.5-MoE class).
major comments (3)
- [§5.5, Table 13, Abstract] §5.5 and Table 13: the abstract/conclusion claim that AAPT and reactive execution have complementary strengths, and that AAPT is preferred when candidates are pre-enumerable, rests on an aggregate tie (7/39 vs 6/39, p=1.0) and only 4 vs 5 discordant tasks, with all AAPT-only wins concentrated in one category (dashboards). That is enough to license a qualitative boundary sketch, but not a stable deployment rule. Either enlarge the external task set (or pre-register a larger majority-of-k design) or soften the abstract/conclusion language to match the underpowered split and state explicitly that the complementary-strengths claim is hypothesis-generating.
- [Abstract, §5.1, Appendix 12] Abstract and §5.1 vs Appendix 12: the headline “producing no incorrect actions” is true only at matched branch budget and under the suppress-on-late/unmatched design. Appendix 12 shows that mis-set K_max (under- or over-budget) converts misses into wrong keypresses (e.g., 13/42, 12/42, 15/42 wrong actions). The body already notes this is a design property at the operating point, not a general safety guarantee; the abstract and any claim ceiling should carry that caveat in one clause so the zero-error reading cannot be detached from budget matching.
- [§4.1, §5.1, Appendix 17.3] §4.1 and §5.1 primary-window handling: two pre-registered primaries left the contested regime (700 ms near R0 ceiling, n.s.; ablation 650 ms recalibrated to 600 ms by reactive-only scan). The paper reports deviations and confirms at a declared 650 ms primary, which is good practice, but the main text should state more sharply how many window choices were pre-registered secondaries versus post-scan selections, and that absolute rates are not portable across seed draws. Without that, readers may over-read the 0.50→0.79 headline as a stable operating point rather than a paired within-regime contrast.
minor comments (6)
- [Figure 1] Figure 1 latencies are clear; add the observer token cap (64) and change-gate bound (≤33 ms) to the caption so the critical-path cartoon is self-contained.
- [§3.1 Eq. (1)] Eq. (1): define M once as a fixed experimental constant (500 ms) and note it is not fit to the prep sweep, to avoid the appearance of a tuned threshold.
- [Table 2, §5.4] Table 2 gate (c) threshold 0.85 is used as a hard pass/fail; a one-sentence justification (or sensitivity) would help, since UI-Venus at 0.39 is far below and the oracle probe does the causal work.
- [§2 Table 1] Related work Table 1 is useful; a short note on how AAPT differs from classical behavior trees / POMDP contingency plans beyond GUI latency would orient non-agent readers.
- [Abstract, §1] Typos/spacing: “windowhasalreadyclosed”, “Weidentifythemaincause”, and similar missing spaces appear in the abstract/intro PDF text; clean for camera-ready.
- [§5, Appendix 18.1] Appendix 18.1 prediction ledger is a model of transparency; consider promoting a condensed version into the main text or supplement pointer earlier than §5.
Circularity Check
No significant circularity: causal isolation rests on external server-side scoring, independent latency measurements, and pre-registered falsifiable predictions (including failed ones).
full rationale
This is an experimental systems paper that isolates decode-on-critical-path as a cause of missed GUI windows by holding model, task, and information fixed and moving generation off the event-time path. Success is scored by a deterministic server-side keypress-before-deadline criterion, not a model judge. Eq. (1) (T_cover ≥ L_p95 + M) is not a fit renamed as prediction: L_p95 is independently measured planner latency and the prep-budget sweep is a genuine out-of-sample check that the advantage appears only above that threshold. R1/P1 baselines share anticipation but keep decode on-path and score zero, so the central contrast is not definitional. Capability gates, the k×n knee at outcome-set size, the oracle-on-Venus probe, and the Qwen generalist confirmation were pre-registered; the oracle probe explicitly rejected the authors' committed hypothesis and is reported at full weight. No load-bearing uniqueness theorem, self-citation chain, or fitted parameter is smuggled in as a first-principles result. The enumerability/prep-coverage contract is an explicit scope boundary (falsified by click_target and the DynaCU split), not a circular premise. Score 0.
Axiom & Free-Parameter Ledger
free parameters (6)
- Safety margin M in T_cover ≥ L_p95 + M =
500 ms
- Planner-confidence floor τ_m =
0.45
- Observer token cap =
64 tokens
- Change-gate thresholds =
0.005 / 20 / 500 ms / 30 FPS
- Branch budget K_max and depth n_max =
K_max=3 (3-outcome), n=1 recommended
- cover_ms / T_cover validation budget =
3500 ms (validation reference)
axioms (5)
- domain assumption Success ⇔ correct action ∧ action before environment deadline
- ad hoc to paper Tree must bridge planner unavailability: T_cover ≥ L_p95 + M
- ad hoc to paper Flat independent guards suffice; routing is single-shot classification over siblings, not deep traversal
- domain assumption Only allowlisted pre-authorized actions may execute; unmatched/expired/low-confidence routes abstain and replan
- standard math Paired per-seed McNemar comparisons are the valid unit when absolute rates are seed- and latency-sensitive
invented entities (2)
-
Adaptive Anticipatory Policy Trees (AAPT)
independent evidence
-
key_prompt contested-window benchmark
no independent evidence
read the original abstract
Computer-use agents often fail on transient GUI events because they produce the correct action only after the relevant window has already closed. We identify the main cause as expensive autoregressive decoding on the decision-time critical path. We propose Adaptive Anticipatory Policy Trees (AAPT), which eliminates this delay without modifying the underlying model. During idle screen periods, the same frozen multimodal model constructs a bounded conditional policy tree with observable guards, pre-authorized actions, and branch-specific deadlines. The tree is sized to cover the model's own decoding latency. When an event occurs, a lightweight observer matches change-gated frames to a prepared branch and immediately executes the corresponding action without generating new text. In paired trials with pre-registered endpoints and exact McNemar tests, AAPT improves the success rate from 0.50 to 0.79 within a contested decision window ($p=1.8\times10^{-3}$), while producing no incorrect actions. Both open-loop and predict-and-replan baselines achieve zero success because they still decode during execution. A preparation-time sweep shows that the gain emerges where the latency-based tree-sizing rule predicts, and ablations reveal three key requirements: fast observer decoding, valid tree planning, and accurate branch routing. A pre-registered oracle probe rejects our initial hypothesis and instead points to branch routing as the causal bottleneck. We further reproduce the effect on an independent general-purpose multimodal model over 126 paired trials ($p=4.9\times10^{-13}$). On an external benchmark, AAPT matches the overall performance of a reactive baseline, although the two methods exhibit complementary strengths. Together, these results suggest that AAPT performs best when candidate actions can be enumerated in advance, whereas reactive execution remains stronger when they cannot.
Figures
Reference graph
Works this paper leans on
-
[2]
NeurIPS2023DatasetsandBenchmarks Track, arXiv:2306.06070
Mind2Web: Towards a Generalist AgentfortheWeb. NeurIPS2023DatasetsandBenchmarks Track, arXiv:2306.06070. Ding,H.;Liu,P.;Wang,J.;Ji,Z.;Cao,M.;Zhang,R.;Ai,L.; Yang, E.; Shi, T.; and Yu, L
-
[5]
SKILL.nb: Selective Formalization and Gated Execution for Durable Agent Workflows. arXiv:2606.08049. Gu, Y.; Zheng, B.; Gou, B.; Zhang, K.; Chang, C.; Srivas- tava, S.; Xie, Y.; Qi, P.; Sun, H.; and Su, Y
-
[6]
Is Your LLM Secretly a World Model of the Internet? Model-Based Planning for Web Agents. arXiv:2411.06559. Guan, Y.; Hua, W.; Lan, Q.; Fei, S.; Ding, D.; Acharya, D.; Wang, C.; and Wang, W. Y
-
[8]
Computer-Using World Model. arXiv:2602.17365. HCompany.2026.Holo-3.1-35B-A3Bmodelcard.Hugging Face. He, R.; Brunskill, E.; and Roy, N
arXiv 2026
-
[9]
Interactive Speculative Planning: Enhance Agent Efficiency through Co-design of System and User In- terface. arXiv:2410.00079. inclusionAI
-
[11]
Agent-Computer Observation In- terfacesEnableDynamicComputerUse. arXiv:2606.29472. Liang, Y.; Zhou, S.; Gu, Y.; Tan, H.; Wu, G.; Dernoncourt, F.; Kil, J.; Rossi, R. A.; and Zhang, R
-
[12]
Anticipatory Planning for Multimodal AI Agents. arXiv:2603.16777. Liu, E.; Pan, L.; Gao, Z.; Yang, Y.; Shi, C.; Liu, Y.; Wu, J.; and Li, Q
-
[13]
Benchmarking and Improving GUI Agents in High-Dynamic Environments. arXiv:2604.25380. Luo, D.; Tang, B.; Li, K.; Papoudakis, G.; Song, J.; Gong, S.; Hao, J.; Wang, J.; and Shao, K
-
[14]
ViMo: A Generative Visual GUI World Model for App Agents. arXiv:2504.13936. Meituan
-
[16]
AndroidWorld: A Dynamic Benchmarking Environment for Autonomous Agents. arXiv:2405.14573. Tongyi-MiA
-
[17]
UI-Ins-32B model card. Hugging Face. Wang,Z.Z.;Mao,J.;Fried,D.;andNeubig,G.2024. Agent Workflow Memory. arXiv:2409.07429. Xiao, Z.; Tu, J.; Zou, C.; Zuo, Y.; Li, Z.; Wang, P.; Yu, B.; Huang, F.; Lin, J.; and Liu, Z
Pith/arXiv arXiv 2024
-
[18]
Web- World:ALarge-ScaleWorldModelforWebAgentTraining. arXiv:2602.14721. Xie, T.; Zhang, D.; Chen, J.; Li, X.; Zhao, S.; Cao, R.; Hua, T. J.; Cheng, Z.; Shin, D.; Lei, F.; Liu, Y.; Xu, Y.; Zhou, S.; Savarese, S.; Xiong, C.; Zhong, V.; and Yu, T
-
[19]
NeurIPS 2024, arXiv:2404.07972
OS- World: Benchmarking Multimodal Agents for Open-Ended Tasks in Real Computer Environments. NeurIPS 2024, arXiv:2404.07972. Yang,H.;Tang,F.;Zhao,L.;Zhuang,X.;Lu,Y.;An,X.;Hu, M.; Zhang, X.; Swikir, A.; He, J.; Ge, Z.; Khan, M. H.; and Razzak,I.2025. StreamAgent:TowardsAnticipatoryAgents for Streaming Video Understanding. arXiv:2508.01875. Yao, J.; Huang,...
Pith/arXiv arXiv 2024
-
[20]
Benchmarking Living-Screen-Native GUI Agents on Short-Video Platforms. arXiv:2606.04701. Ye, N.; Ahuja, A.; Liargkovas, G.; Lu, Y.; Kaffes, K.; and Peng, T
-
[21]
Speculative Actions: A Lossless Framework for Faster Agentic Systems. arXiv:2510.04371. Ye,N.;Somani,A.;Hsu,D.;andLee,W.S.2017. DESPOT: Online POMDP Planning with Regularization.Journal of Artificial Intelligence Research, 58: 231–266. Zheng,Y.;Zhong,L.;Wang,Y.;Dai,R.;Liu,K.;Chu,X.;Lv, L.;Torr,P.;andLin,K.Q.2026. Code2World:AGUIWorld ModelviaRenderableCod...
Pith/arXiv arXiv 2017
-
[22]
ActionEngine: From Reactive to Programmatic GUI Agents via State Ma- chine Memory. arXiv:2602.20502. Zhou, S.; Xu, F. F.; Zhu, H.; Zhou, X.; Lo, R.; Sridhar, A.; Cheng, X.; Ou, T.; Bisk, Y.; Fried, D.; Alon, U.; and Neubig, G
-
[23]
WebArena: A Realistic Web Environment for Building Autonomous Agents. arXiv:2307.13854. 8 Protocol details Stage ladder.The study ran as a pre-registered lad- der of stages, each with a written plan (goals, windows, seeds, endpoints, decision rules) committed before execu- tion: pilot and planner diagnostics; guided-decoding adop- tion and the below-floor...
-
[24]
an action or an action array
is a meaningful fraction of the shortest ofthese.Atorabove10s(regulatedtakeover,WCAG,TOTP, push approval, SSH) the reactive loop suffices and AAPT’s premium is not justified. The survey bounds the claim hon- estly in both directions: real sub-second windows exist in deployed systems, and most windows above a few seconds do not need anticipation. 16 Method...
2026
-
[2003]
Optimal Limited Contin- gencyPlanning.InProceedingsoftheNineteenthConference on Uncertainty in Artificial Intelligence. Qin,Y.;Ye,Y.;Fang,J.;Wang,H.;Liang,S.;Tian,S.;Zhang, J.;Li,J.;Li,Y.;Huang,S.;Zhong,W.;Li,K.;Yang,J.;Miao, Y.; Lin, W.; Liu, L.; Jiang, X.; Ma, Q.; Li, J.; Xiao, X.; Cai, K.; Li, C.; Zheng, Y.; Jin, C.; Li, C.; Zhou, X.; Wang, M.; Chen, H...
Pith/arXiv arXiv 2025
-
[2020]
A Survey of Behavior Trees in Robotics and AI. arXiv:2005.05842. Kim,H.;Wu,Y.;andTambe,T.2026. AgenticCache:Cache- DrivenAsynchronousPlanningforEmbodiedAIAgents. In Proceedingsofthe9thConferenceonMachineLearningand Systems. Li,B.2026. PreAct:Computer-UsingAgentsthatGetFaster on Repeated Tasks. arXiv:2606.17929. Li, B.; and Shi, N
Pith/arXiv arXiv 2005
-
[2023]
InProceedings of AREA 2023 (EPTCS 391), 89–101
Rollout Heuristics for Online Stochastic Contingent Planning. InProceedings of AREA 2023 (EPTCS 391), 89–101. Cao, Y.; Zhong, Y.; Zeng, Z.; Zheng, L.; Huang, J.; Qiu, H.; Shi,P.;Mao,W.;andWan,G.2026. MobileDreamer:Gener- ativeSketchWorldModelforGUIAgent.arXiv:2601.04035. Chen,Q.;Bellucci,A.;Sun,Z.;andJacucci,G.2026. Skill- Droid: Compile Once, Reuse Forev...
arXiv 2023
-
[2024]
XGrammar: Flexible and Efficient Structured Generation Engine for Large Language Models. arXiv:2411.15100. El Hattami, A.; Chapados, N.; and Pal, C
-
[2025]
Dynamic Speculative Agent Planning. arXiv:2509.01920. Guan, Y.; Yu, R.; Zhang, J.; Wang, L.; Li, L.; Qiao, B.; Qin, S.; Huang, H.; Yang, F.; Zhao, P.; Wutschitz, L.; Kessler, S.; Inan, H. A.; Sim, R.; Rajmohan, S.; Lin, Q.; and Zhang, D
-
[2026]
DynaWeb: Model-Based ReinforcementLearningofWebAgents. arXiv:2601.22149. Dong, Y.; Ruan, C. F.; Cai, Y.; Lai, R.; Xu, Z.; Zhao, Y.; and Chen, T
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.