REVIEW 4 major objections 3 minor 41 references
CogniWeb, an adaptive web agent, reports 43.96% success on WebArena with 75% lower token use by toggling between fast intuitive and slow deliberative processing.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
A web agent called CogniWeb reportedly reaches 43.96% success on WebArena with 75% fewer tokens by switching between fast and slow reasoning, but the supplied body is a different paper and the claims could not be checked.
T0 review reviewed 2026-08-05 challenge →
load-bearing objection The abstract promises a useful dual-process web agent, but the submitted full text is a different paper, so the claims are neither checkable nor refereeable as-is. the 4 major comments →
Cognitive Duality for Adaptive Web Agents
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
Core claim
The central claim is that a principled dual-process decomposition yields an efficient web agent. CogniWeb toggles between a cheap learned-intuitive module and an expensive deliberative planner, choosing per task which one should act. On WebArena the adaptive system keeps competitive performance (43.96% success) while reducing token consumption by 75%, which the authors present as evidence that most web-navigation steps do not require full deliberative reasoning. The paper frames this as a unifying perspective: offline learning teaches the fast system, online exploration trains the slow system, and the router combines them.
What carries the argument
The load-bearing mechanism is the adaptive toggle: a task-complexity signal that routes each decision to either a System-1 module (fast, reactive, offline-learned) or a System-2 module (slow, deliberative, online-planning). CogniWeb is the modular architecture that instantiates this routing. The dual-process decomposition itself is the identity that connects the two learning paradigms—offline imitation supplies the intuitions, online search supplies the deliberation—making the router the point where they meet.
Load-bearing premise
The 43.96%-at-75%-saving result holds only if the task-complexity signal reliably tells when the cheap System-1 module suffices, and only if that signal and the module choices were not tuned on WebArena itself—the abstract names no baseline for the 75% reduction.
What would settle it
Freeze CogniWeb's complexity threshold and run it on a different web-navigation benchmark such as Mind2Web or WebShop. If success drops far below the always-deliberative agent while token savings remain large, the routing is overfit to WebArena; if the router's decisions are no better than random at predicting which steps the System-1 module solves correctly, the savings are a benchmark artifact, not a cognitive division of labor.
If this is right
- If the 43.96% WebArena result holds up, adaptive routing gives near-parity accuracy at roughly a quarter of the token budget of always running the planner.
- Offline and online learning for web agents can be combined rather than chosen between, with each paradigm training one of the two systems.
- The complexity signal itself becomes a first-class component of agent design, as important as the policy modules it routes between.
- Benchmark success rates alone may mislead: average scores mix easy steps handled by System 1 and hard steps that need System 2, so efficiency should be reported alongside accuracy.
Where Pith is reading between the lines
- A testable extension is to treat the complexity router as a learned classifier and evaluate it on unseen websites; the paper's efficiency claim stands or falls on whether this signal generalizes beyond WebArena.
- The 75% figure likely compares CogniWeb to its own fully deliberative mode; a fairer comparison would include an oracle router that always picks correctly, giving an upper bound on achievable savings.
- If the routing works broadly, dual-process design could transfer to other sequential decision domains—tool use, code repair, robotics—where a cheap reactive policy handles typical steps and a planner is reserved for rare or novel states.
- One consequence the authors leave implicit: the observed success rate may be partly an artifact of WebArena containing many simple tasks; a benchmark with a higher density of hard tasks would shrink the measured savings.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The abstract of arXiv:2508.05081 proposes CogniWeb, a dual-process (System 1 / System 2) web-agent architecture that adaptively toggles between fast and deliberative processing based on task complexity. It claims a WebArena success rate of 43.96% with a 75% reduction in token usage relative to an unspecified baseline. The submitted full text, however, is not the CogniWeb paper: it is MedMKEB, a medical multimodal knowledge-editing benchmark paper identified as arXiv:2508.05083. The body contains no description of CogniWeb, no WebArena evaluation, no baselines, no router mechanism, and no experimental setting that could support the abstract's two headline numbers. The manuscript is therefore internally disconnected: the abstract's claims have no evidentiary core in the submitted material.
Significance. If the abstract's claims were backed by a full paper, the result would be practically significant: a 43.96% success rate on WebArena with a 75% token reduction would demonstrate that adaptive dual-process routing can materially improve efficiency without catastrophic accuracy loss, and the dual-process framing could offer a useful organizing principle for web agents. The manuscript also has the potential to provide a principled decomposition linking offline imitation learning and online deliberation. However, as submitted, none of these contributions are supported: the architecture, the complexity signal, the evaluation protocol, and the cost baseline are absent. The significance of the claimed results therefore cannot be assessed from the material under review.
major comments (4)
- [Full Text (all sections)] The submitted full text is the MedMKEB paper (arXiv:2508.05083), not the CogniWeb paper promised by the abstract. The body and appendices describe a medical knowledge-editing benchmark with equations for reliability, locality, generality, portability, and robustness, and experiments on BLIP2-OPT, MiniGPT-4, LLaVA, and medical MLLMs. None of this material concerns WebArena, web navigation, or adaptive dual-process routing. As a result, the abstract's central claims—43.96% success and 75% token reduction—have no supporting evidence in the manuscript. This is not a local omission; the entire evidential core of the claimed contribution is missing.
- [Abstract] The abstract states that CogniWeb 'achieves competitive performance (43.96% success rate)' without naming any baseline. WebArena success rates are meaningless without a comparison point: competitive with what—GPT-4? a human? a prior agent? The same sentence reports '75% reduction in token usage' without defining the denominator. Is the baseline a non-adaptive agent using the same System 2 module for all tasks? Is it the average token count per task, or total across trajectories? Without this definition, the headline efficiency number is uninterpretable and cannot be verified.
- [Abstract] The load-bearing mechanism is the toggle 'based on task complexity,' but the manuscript provides no description of the complexity signal, its threshold, or how the fast and slow modules are instantiated. If the threshold or module choices were selected using WebArena itself, then the 43.96%/75% trade-off is a fitted result, not a general property of the architecture. The paper would need to report the threshold selection procedure, ablations over the threshold, and performance on a validation set or held-out environments to support the claimed adaptivity.
- [Full Text (Abstract mismatch)] The presence of an unrelated manuscript body creates a verification failure beyond ordinary editorial issues. The paper's own content asserts the MedMKEB benchmark statistics and editing results, which are unrelated to the abstract's subject. As submitted, the manuscript cannot be checked, reproduced, or even located as a coherent document. This is a load-bearing problem: it makes the abstract's claims unverifiable. The authors would need to replace the entire body, not patch a section, to make the claims assessable.
minor comments (3)
- [Title and metadata] The title and abstract describe a web-agent paper, but the PDF body is a medical benchmark paper with different authors and a different arXiv identifier. This appears to be a submission or file-assembly error that should be corrected at the source.
- [Abstract] Even after the body is corrected, the abstract should report error bars or statistical significance for the 43.96% success rate and the 75% token-reduction figure, and should state the WebArena task subset and number of runs.
- [Abstract] The phrase 'principled decomposition' is not supported by any derivation or formal statement in the provided text. If the full paper contains such a derivation, it should be referenced explicitly in the abstract or introduction.
Circularity Check
No circularity found: the abstract and the supplied full text are different papers, so there is no derivation chain that could reduce to its own inputs; the evidential gap is a verification issue, not circularity.
full rationale
The manuscript supplied for arXiv:2508.05081 contains an abstract describing CogniWeb, a dual-process web agent with a 43.96% WebArena success rate and 75% token reduction, but the full text is entirely a different paper, MedMKEB (arXiv:2508.05083), about a medical multimodal knowledge editing benchmark. There is no derivation chain in the submitted text connecting the abstract's claims to any definitions, equations, fitted parameters, or cited prior work. In particular, there is no equation where the reported success rate or token reduction is defined as the output of a fitted routing threshold, no renamed known result, and no self-citation chain invoked to force a conclusion. The 'adaptive toggle based on task complexity' mentioned in the abstract could in principle be a fitted mechanism, but the paper provides no details to exhibit such a reduction, and speculation is explicitly disallowed. The mismatch between the abstract and the full text means the central claims are unsupported and unverifiable from the provided evidence, but that is an evidentiary or integrity problem, not a circularity problem. Therefore the appropriate circularity score is 0, with no circular steps identified.
Axiom & Free-Parameter Ledger
free parameters (2)
- task-complexity toggle threshold
- fast/slow module selection (System 1 and System 2 instantiations)
axioms (3)
- domain assumption The dual-process (System 1/System 2) decomposition is a valid and useful model of web-agent decision-making
- domain assumption WebArena success rate is the appropriate measure of competitive agent performance
- domain assumption The 75% token reduction is measured against a meaningful and matched baseline
invented entities (2)
-
CogniWeb task-complexity router
no independent evidence
-
System 1 and System 2 agent modules
no independent evidence
Cite this review
Pith. "Pith review of Cognitive Duality for Adaptive Web Agents." pith.science (2026). https://pith.science/paper/7ZB32E4U
@misc{pith2026250805081,
author = {Pith},
title = {Pith review of: Cognitive Duality for Adaptive Web Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/7ZB32E4U}},
note = {Machine review of arXiv:2508.05081}
}
read the original abstract
Web navigation represents a critical and challenging domain for evaluating artificial general intelligence (AGI), demanding complex decision-making within high-entropy, dynamic environments with combinatorially explosive action spaces. Current approaches to building autonomous web agents either focus on offline imitation learning or online exploration, but rarely integrate both paradigms effectively. Inspired by the dual-process theory of human cognition, we derive a principled decomposition into fast System 1 and slow System 2 cognitive processes. This decomposition provides a unifying perspective on existing web agent methodologies, bridging the gap between offline learning of intuitive reactive behaviors and online acquisition of deliberative planning capabilities. We implement this framework in CogniWeb, a modular agent architecture that adaptively toggles between fast intuitive processing and deliberate reasoning based on task complexity. Our evaluation on WebArena demonstrates that CogniWeb achieves competitive performance (43.96% success rate) while maintaining significantly higher efficiency (75% reduction in token usage).
Reference graph
Works this paper leans on
-
[1]
Alan D. Baddeley and Graham Hitch. Working memory. volume 8 of Psychology of Learning and Motivation, pages 47--89. Academic Press, 1974. doi:https://doi.org/10.1016/S0079-7421(08)60452-1. URL https://www.sciencedirect.com/science/article/pii/S0079742108604521
-
[2]
The arcade learning environment: An evaluation platform for general agents
Marc G Bellemare, Yavar Naddaf, Joel Veness, and Michael Bowling. The arcade learning environment: An evaluation platform for general agents. Journal of artificial intelligence research, 47: 0 253--279, 2013
2013
-
[3]
Andrei Broder, Ravi Kumar, Farzin Maghoul, Prabhakar Raghavan, Sridhar Rajagopalan, Raymie Stata, Andrew Tomkins, and Janet Wiener. Graph structure in the web. Computer networks, 33 0 (1-6): 0 309--320, 2000
work page 2000
-
[4]
Mind2web: Towards a generalist agent for the web
Xiang Deng, Yu Gu, Boyuan Zheng, Shijie Chen, Sam Stevens, Boshi Wang, Huan Sun, and Yu Su. Mind2web: Towards a generalist agent for the web. Advances in Neural Information Processing Systems, 36: 0 28091--28114, 2023
2023
-
[5]
Bert: Pre-training of deep bidirectional transformers for language understanding
Jacob Devlin, Ming-Wei Chang, Kenton Lee, and Kristina Toutanova. Bert: Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the 2019 conference of the North American chapter of the association for computational linguistics: human language technologies, volume 1 (long and short papers), pages 4171--4186, 2019
2019
-
[6]
Alexandre Drouin, Maxime Gasse, Massimo Caccia, Issam H Laradji, Manuel Del Verme, Tom Marty, L \'e o Boisvert, Megh Thakkar, Quentin Cappart, David Vazquez, et al. Workarena: How capable are web agents at solving common knowledge work tasks? arXiv preprint arXiv:2403.07718, 2024
Pith/arXiv arXiv 2024
-
[7]
A real-world webagent with planning, long context understanding, and program synthesis
Izzeddin Gur, Hiroki Furuta, Austin Huang, Mustafa Safdari, Yutaka Matsuo, Douglas Eck, and Aleksandra Faust. A real-world webagent with planning, long context understanding, and program synthesis. arXiv preprint arXiv:2307.12856, 2023
Pith/arXiv arXiv 2023
-
[8]
Reasoning with language model is planning with world model
Shibo Hao, Yi Gu, Haodi Ma, Joshua Jiahua Hong, Zhen Wang, Daisy Zhe Wang, and Zhiting Hu. Reasoning with language model is planning with world model. arXiv preprint arXiv:2305.14992, 2023
Pith/arXiv arXiv 2023
-
[9]
Cogagent: A visual language model for gui agents
Wenyi Hong, Weihan Wang, Qingsong Lv, Jiazheng Xu, Wenmeng Yu, Junhui Ji, Yan Wang, Zihan Wang, Yuxiao Dong, Ming Ding, et al. Cogagent: A visual language model for gui agents. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14281--14290, 2024
work page 2024
-
[10]
Thinking, fast and slow
Daniel Kahneman. Thinking, fast and slow. macmillan, 2011
2011
-
[11]
Autowebglm: A large language model-based web navigating agent
Hanyu Lai, Xiao Liu, Iat Long Iong, Shuntian Yao, Yuxuan Chen, Pengbo Shen, Hao Yu, Hanchen Zhang, Xiaohan Zhang, Yuxiao Dong, et al. Autowebglm: A large language model-based web navigating agent. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining, pages 5295--5306, 2024
work page 2024
-
[12]
The Soar cognitive architecture
John E Laird. The Soar cognitive architecture. MIT press, 2019
2019
-
[13]
Pix2struct: Screenshot parsing as pretraining for visual language understanding
Kenton Lee, Mandar Joshi, Iulia Raluca Turc, Hexiang Hu, Fangyu Liu, Julian Martin Eisenschlos, Urvashi Khandelwal, Peter Shaw, Ming-Wei Chang, and Kristina Toutanova. Pix2struct: Screenshot parsing as pretraining for visual language understanding. In International Conference on Machine Learning, pages 18893--18912. PMLR, 2023
work page 2023
-
[14]
Shane Legg. Solomonoff induction. Centre for Discrete Mathematics and Theoretical Computer Science, 1997
work page 1997
-
[15]
Universal intelligence: A definition of machine intelligence
Shane Legg and Marcus Hutter. Universal intelligence: A definition of machine intelligence. Minds and machines, 17: 0 391--444, 2007
2007
-
[16]
An introduction to Kolmogorov complexity and its applications, volume 3
Ming Li, Paul Vit \'a nyi, et al. An introduction to Kolmogorov complexity and its applications, volume 3. Springer, 2008
work page 2008
-
[17]
Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks
Bill Yuchen Lin, Yicheng Fu, Karina Yang, Faeze Brahman, Shiyu Huang, Chandra Bhagavatula, Prithviraj Ammanabrolu, Yejin Choi, and Xiang Ren. Swiftsage: A generative agent with fast and slow thinking for complex interactive tasks. Advances in Neural Information Processing Systems, 36: 0 23813--23825, 2023
work page 2023
-
[18]
Wepo: Web element preference optimization for llm-based web navigation, 2024
Jiarun Liu, Jia Hao, Chunhong Zhang, and Zheng Hu. Wepo: Web element preference optimization for llm-based web navigation, 2024. URL https://arxiv.org/abs/2412.10742
Pith/arXiv arXiv 2024
-
[19]
Weblinx: Real-world website navigation with multi-turn dialogue
Xing Han L \`u , Zden e k Kasner, and Siva Reddy. Weblinx: Real-world website navigation with multi-turn dialogue. arXiv preprint arXiv:2402.05930, 2024
arXiv 2024
-
[20]
Human-level control through deep reinforcement learning
Volodymyr Mnih, Koray Kavukcuoglu, David Silver, Andrei A Rusu, Joel Veness, Marc G Bellemare, Alex Graves, Martin Riedmiller, Andreas K Fidjeland, Georg Ostrovski, et al. Human-level control through deep reinforcement learning. nature, 518 0 (7540): 0 529--533, 2015
2015
-
[21]
Extending cognitive architecture with episodic memory
Andrew M Nuxoll and John E Laird. Extending cognitive architecture with episodic memory. In AAAI, pages 1560--1564, 2007
work page 2007
-
[22]
Computer-using agent: Introducing a universal interface for ai to interact with the digital world
OpenAI. Computer-using agent: Introducing a universal interface for ai to interact with the digital world. 2025. URL https://openai.com/index/computer-using-agent
work page 2025
-
[23]
Dynathink: Fast or slow? a dynamic decision-making framework for large language models, 2024
Jiabao Pan, Yan Zhang, Chen Zhang, Zuozhu Liu, Hongwei Wang, and Haizhou Li. Dynathink: Fast or slow? a dynamic decision-making framework for large language models, 2024. URL https://arxiv.org/abs/2407.01009
Pith/arXiv arXiv 2024
-
[24]
Adapt: As-needed decomposition and planning with language models, 2024
Archiki Prasad, Alexander Koller, Mareike Hartmann, Peter Clark, Ashish Sabharwal, Mohit Bansal, and Tushar Khot. Adapt: As-needed decomposition and planning with language models, 2024. URL https://arxiv.org/abs/2311.05772
Pith/arXiv arXiv 2024
-
[25]
Markov decision processes: discrete stochastic dynamic programming
Martin L Puterman. Markov decision processes: discrete stochastic dynamic programming. John Wiley & Sons, 2014
2014
-
[26]
Interactive continual learning: Fast and slow thinking
Biqing Qi, Xinquan Chen, Junqi Gao, Dong Li, Jianxing Liu, Ligang Wu, and Bowen Zhou. Interactive continual learning: Fast and slow thinking. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 12882--12892, June 2024
work page 2024
-
[27]
Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025
Zehan Qi, Xiao Liu, Iat Long Iong, Hanyu Lai, Xueqiao Sun, Wenyi Zhao, Yu Yang, Xinyue Yang, Jiadai Sun, Shuntian Yao, Tianjie Zhang, Wei Xu, Jie Tang, and Yuxiao Dong. Webrl: Training llm web agents via self-evolving online curriculum reinforcement learning, 2025. URL https://arxiv.org/abs/2411.02337
Pith/arXiv arXiv 2025
-
[28]
O1 replication journey: A strategic progress report--part 1
Yiwei Qin, Xuefeng Li, Haoyang Zou, Yixiu Liu, Shijie Xia, Zhen Huang, Yixin Ye, Weizhe Yuan, Hector Liu, Yuanzhi Li, et al. O1 replication journey: A strategic progress report--part 1. arXiv preprint arXiv:2410.18982, 2024
Pith/arXiv arXiv 2024
-
[29]
Direct preference optimization: Your language model is secretly a reward model
Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. Direct preference optimization: Your language model is secretly a reward model. Advances in Neural Information Processing Systems, 36: 0 53728--53741, 2023
2023
-
[30]
Deepseekmath: Pushing the limits of mathematical reasoning in open language models
Zhihong Shao, Peiyi Wang, Qihao Zhu, Runxin Xu, Junxiao Song, Xiao Bi, Haowei Zhang, Mingchuan Zhang, YK Li, Y Wu, et al. Deepseekmath: Pushing the limits of mathematical reasoning in open language models. arXiv preprint arXiv:2402.03300, 2024
Pith/arXiv arXiv 2024
-
[31]
Reflexion: Language agents with verbal reinforcement learning, 2023
Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning, 2023. URL https://arxiv.org/abs/2303.11366
Pith/arXiv arXiv 2023
-
[32]
Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024
Charlie Snell, Jaehoon Lee, Kelvin Xu, and Aviral Kumar. Scaling llm test-time compute optimally can be more effective than scaling model parameters, 2024. URL https://arxiv.org/abs/2408.03314
Pith/arXiv arXiv 2024
-
[33]
Step: Stacked llm policies for web actions
Paloma Sodhi, SRK Branavan, Yoav Artzi, and Ryan McDonald. Step: Stacked llm policies for web actions. arXiv preprint arXiv:2310.03720, 2023
Pith/arXiv arXiv 2023
-
[34]
Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments
Hongjin Su, Ruoxi Sun, Jinsung Yoon, Pengcheng Yin, Tao Yu, and Sercan \"O Ar k. Learn-by-interact: A data-centric framework for self-adaptive agents in realistic environments. arXiv preprint arXiv:2501.10893, 2025
Pith/arXiv arXiv 2025
-
[35]
Cognitive architectures for language agents
Theodore Sumers, Shunyu Yao, Karthik Narasimhan, and Thomas Griffiths. Cognitive architectures for language agents. Transactions on Machine Learning Research, 2023
work page 2023
-
[36]
Fast-slow-thinking: Complex task solving with large language models, 2025
Yiliu Sun, Yanfang Zhang, Zicheng Zhao, Sheng Wan, Dacheng Tao, and Chen Gong. Fast-slow-thinking: Complex task solving with large language models, 2025. URL https://arxiv.org/abs/2504.08690
Pith/arXiv arXiv 2025
-
[37]
Webshop: Towards scalable real-world web interaction with grounded language agents
Shunyu Yao, Howard Chen, John Yang, and Karthik Narasimhan. Webshop: Towards scalable real-world web interaction with grounded language agents. Advances in Neural Information Processing Systems, 35: 0 20744--20757, 2022
2022
-
[38]
Tree of thoughts: Deliberate problem solving with large language models
Shunyu Yao, Dian Yu, Jeffrey Zhao, Izhak Shafran, Tom Griffiths, Yuan Cao, and Karthik Narasimhan. Tree of thoughts: Deliberate problem solving with large language models. Advances in neural information processing systems, 36: 0 11809--11822, 2023
2023
-
[39]
Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms
Ruichen Zhang, Mufan Qiu, Zhen Tan, Mohan Zhang, Vincent Lu, Jie Peng, Kaidi Xu, Leandro Z Agudelo, Peter Qian, and Tianlong Chen. Symbiotic cooperation for web agents: Harnessing complementary strengths of large and small llms. arXiv preprint arXiv:2502.07942, 2025 a
Pith/arXiv arXiv 2025
-
[40]
Yao Zhang, Zijian Ma, Yunpu Ma, Zhen Han, Yu Wu, and Volker Tresp. Webpilot: A versatile and autonomous multi-agent system for web task execution with strategic exploration. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 39, pages 23378--23386, 2025 b
work page 2025
-
[41]
Webarena: A realistic web environment for building autonomous agents
Shuyan Zhou, Frank F Xu, Hao Zhu, Xuhui Zhou, Robert Lo, Abishek Sridhar, Xianyi Cheng, Tianyue Ou, Yonatan Bisk, Daniel Fried, et al. Webarena: A realistic web environment for building autonomous agents. arXiv preprint arXiv:2307.13854, 2023
Pith/arXiv arXiv 2023
This paper was first reviewed by deepseek-v4-flash on August 5, 2026.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.