REVIEW 4 major objections 6 minor 1 cited by
Every Software as an Agent: Blueprint and Case Study
T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash
Pith's one-line read The paper argues that software agents should stop clicking and start reading source code, then injecting generated code into the running program.
desk verdict Genuinely new whitebox code-injection idea with a thin but real feasibility demo; claims outrun evidence, but the direction deserves referee time. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is the JiT-Codegen loop connecting an LLM-powered Code Agent to an Execution Sandbox embedded in the software runtime. The Code Agent writes action code from the user instruction plus source-code context; the Sandbox runs that code in-process or via IPC against live data structures and returns runtime feedback such as error messages or execution results; the agent iterates. The paper names this just-in-time code generation and in-software execution and draws an analogy to JIT compilation and to eBPF, where injected code runs in a restricted kernel context. The sandbox is the component that must balance flexibility, safety, and developer effort, and its design depends on the software's language and framework.
What would settle it
Run JiT-Codegen on a fixed suite of tasks over a real desktop app with an implemented sandbox, static analysis, audit log, and rollback: if task-completion rate under those safeguards falls to the level of GUI-based baselines, or if a single generated action irreversibly corrupts data that rollback cannot restore, the central viability claim fails.
Extended reading notes
Core claim
The central claim is that the non-intrusive assumption shared by API-based and GUI-based agents—that the LLM cannot see or touch the software's internals—is the fundamental barrier to high-accuracy, efficient, general software agents. JiT-Codegen instead gives the LLM the source code, documentation, and runtime context, including in-memory data structures, open files, databases, and the UI tree, plus permission to inject generated code into the runtime through an execution sandbox. In the paper's state-machine framing, an API call reaches only pre-defined states and a GUI agent reaches intermediate states step by step, but a generated action can transition directly to the target state. The case study on two Electron apps, a markdown editor and a music player, reports an 80% task-completion rate with GPT-4o, including multi-round refinement where runtime error messages guide the next code generation. The conclusion is that JiT-Codegen can complement, and in some cases replace, API and GUI approaches, turning any software whose code is available into an agent.
Load-bearing premise
The load-bearing premise is that generated code with full memory and disk access can be contained by sandboxing and safety rules without destroying the flexibility that makes JiT-Codegen accurate and efficient; the paper discusses but does not implement or measure any of the four safeguards.
Editorial extensions
If this is right
- If JiT-Codegen works at scale, a task that needs five GUI interactions can be finished with one generated action, eliminating per-step rendering and LLM invocation overhead.
- Existing non-intrusive agents on desktop and mobile benchmarks typically complete under 20% of tasks, so the paper argues that the low ceiling comes less from LLM ability than from the interaction interface.
- Because generated code is created on demand, tasks with no GUI representation become expressible as code, and states unreachable through pre-defined APIs become reachable.
- Successful action code can be wrapped into a reusable API, so JiT-Codegen naturally bootstraps an API layer over time rather than conflicting with API-based design.
- The approach's adoption hinges on four un-implemented safeguards: sandboxing, static analysis, audit logs and rollback, and rule-enforced LLM generation, which the paper states as its next step.
Reading between the lines
- The paper leaves implicit that the same runtime access enabling one-step task completion also converts any mis-prompted or adversarially injected instruction into privileged code execution, so the practical test is whether a safety rule set blocks harmful actions without lowering the 8-of-10-level completion rate.
- Because released software codebases are mostly static, a near-term extension is to pre-index each app's functions and data structures as a graph and retrieve only task-relevant slices before generation, a concrete test of the paper's accuracy proposal.
- The multi-agent vision introduces a natural-language permission-delegation surface: one app agent can ask another to perform an action it lacks permission for, so auditing cross-agent requests is a direct stress test of the claimed digital-world scenario.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a new software-agent paradigm, JiT-Codegen, in which an LLM-powered agent is given access to a program's source code and runtime state and is allowed to inject generated code directly into the running software. The authors argue that this whitebox approach is more accurate, more ubiquitous, and more efficient than API-based and GUI-based agents, and they support the proposal with a conceptual comparison (state-machine view and Table 1), a minimal Electron-based prototype, and a case study on two open-source desktop apps (marktext and YesPlayMusic) using five LLMs across ten tasks (Table 2). The paper concludes with challenges and future directions, with particular emphasis on safety, accuracy, deployment, and multi-software-agent systems.
Significance. If the approach can be made safe and its empirical claims substantiated, JiT-Codegen would be a meaningful reframing of software agents: it replaces many-step GUI manipulation with a single code-level action that can reach arbitrary runtime states, and it is complementary to existing API and GUI methods rather than mutually exclusive. The paper is candid about its status as a blueprint and gives concrete, reproducible details about the prototype size, the tested applications, and per-model task outcomes. Its strengths are transparency in reporting the 10-task results, a worked multi-round error-feedback example (Listing 1), and explicit recognition of safety as the primary obstacle. However, the evidence as presented supports feasibility, not the stronger claim of superiority over existing approaches: there are no same-task baselines, no repeated trials, no statistical analysis, and no implemented safety mechanisms.
major comments (4)
- [§4, Table 2] The central claim that JiT-Codegen achieves higher accuracy and efficiency than API-based and GUI-based agents is not tested comparatively. Table 2 reports success counts for five LLMs on ten self-selected tasks, but no API-based or GUI-based agent was run on the same tasks. The 80% success rate for GPT-4o therefore has no baseline against which superiority can be established. The manuscript should either run an existing GUI/API agent on the identical task set or clearly re-scope the claims to feasibility rather than superiority.
- [§4, Table 2 and Listing 2] Each task-model combination is run once, with no repeated trials, no error bars, and no statistical test. Given the stochasticity of LLM code generation and the manual judgment of success, the reported 8/10 figure is fragile. In addition, the manuscript states that failures are often 'salient' and that the agent stops unless the user gives extra feedback, yet Listing 2 shows that with manual prompting a workable solution for 'show my listening history' eventually appears for o1 after the table marks it as failed for all models. The evaluation protocol (how success is judged, whether extra prompting is allowed, and how the table and Listing 2 are reconciled) must be specified.
- [§4, Listing 1 vs. §3] The evidence does not isolate the proposed novelty. Listing 1 shows that success in 'increase the volume slightly' arises from the agent reacting to runtime error messages (TypeError, component-not-found) over multiple rounds. That learning signal is available to any code-execution agent, including a blackbox GUI-less agent, and does not by itself demonstrate that whitebox source-code access or in-runtime injection is the enabling factor. The authors should add an ablation that withholds source code and/or runtime-context access while keeping code execution and error feedback, or otherwise discuss why the current results cannot distinguish these mechanisms.
- [§5.1] Safety is acknowledged as the primary challenge, but the four proposed safeguarding techniques (sandboxing, static analysis, audit logs, and rule-enforced generation) are only discussed, not implemented or evaluated. The paper itself states that action code 'could do harmful and irreversible damage' and that flexibility comes at a safety cost. Because the practical viability and deployability of JiT-Codegen depend on containing such code, at least a minimal feasibility demonstration of one safeguarding mechanism, or an explicit statement that safety is outside the scope of this blue-print, is needed before the broader 'fundamentally overturn' claim can be taken as supported.
minor comments (6)
- [Abstract and title page] There is a typo in the author affiliation: 'Beiing University' should be 'Beijing University'.
- [Table 1] The checkmark scales (✓, ✓✓, ✗, ✗✗) are not defined; the reader cannot tell whether ✓✓ is stronger than ✓ or whether ✗✗ is weaker than ✗. Please add a legend or use a consistent ordinal scale.
- [Figure 4] The marks on the code snippets are visually ambiguous and appear inconsistent with the model labels (for example, the symbols next to GPT-4o and o3-mini are not clearly aligned with the reported success/failure outcomes). Please redraw the figure with explicit pass/fail labels.
- [§4] The prompt templates, source-code retrieval method, and sandbox implementation are not described in enough detail for reproduction. The paper reports the prototype line counts but does not provide a code repository or appendix; adding these would substantially strengthen the reproducibility of the case study.
- [§5.2] The statement that 'most unmanaged programming language like C/Rust do not allow catching runtime error' is imprecise: C and Rust do not have runtime exceptions in the same way as managed languages, but they do provide signal handling, panic hooks, and other error mechanisms. Please rephrase.
- [§4] The sentence attributing the lower performance of o1 and o3-mini to 'ineffective prompt engineering' is speculative; no evidence is given that prompt engineering, rather than model behavior or task selection, caused the difference.
Circularity Check
No circular derivation; the case study is an unbaselined empirical demonstration, and the self-citations are contextual rather than load-bearing.
full rationale
The paper makes no quantitative derivation or fitted-parameter prediction that could reduce to its inputs. Its central proposal, JiT-Codegen, is a design thesis; the case study reports direct empirical counts in Table 2 (e.g., GPT-4o solving 8/10 tasks), with no model fitting, no learned parameters, and no closed-form relation that would make a measured outcome true by construction. The self-citations (e.g., [27] Llamatouch, [19] DroidCall, and the author's surveys) appear in background, motivation, and future-work discussion; the claim that existing agents have low completion rates is supported by cited benchmarks including external ones ([15], [18]), and the cited self-authored artifacts are benchmarks or surveys rather than results that presuppose JiT-Codegen. The absence of API/GUI baselines or an ablation is a real evidentiary limitation for the comparative claim, but absence of evidence is not circularity: the 80% figure is a measured outcome, not a consequence of defining JiT-Codegen to be accurate. No step in the paper's argument is equivalent to its inputs by definition, and no load-bearing premise is imported solely from the author's prior work. Therefore the appropriate circularity score is 0.
Assumptions & free parameters
assumptions (3)
- domain assumption The agent has full access to the software's source code and runtime context.
- domain assumption Generated action code can be safely constrained so it does not cause harmful or irreversible consequences.
- domain assumption LLM-generated action code, given source context and execution feedback, is accurate enough for task completion.
Cite this review
Pith. "Pith review of Every Software as an Agent: Blueprint and Case Study." pith.science (2026). https://pith.science/paper/P44ND3ZL
@misc{pith2026250204747,
author = {Pith},
title = {Pith review of: Every Software as an Agent: Blueprint and Case Study},
year = {2026},
howpublished = {\url{https://pith.science/paper/P44ND3ZL}},
note = {Machine review of arXiv:2502.04747}
}
read the original abstract
The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.
Figures
Figures from the paper (2 more)
Forward citations
Cited by 1 Pith paper
-
Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI
A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.
Reference graph
Works this paper leans on
-
[1]
Anthropic. Anthropic computer use. https://docs.anthropic.com/en/docs/ build-with-claude/computer-use#how-computer-use-works , 2024
work page 2024
-
[2]
Prompting is programming: A query language for large language models
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7(PLDI):1946–1969, 2023
1946
-
[3]
Guiding llms the right way: Fast, non-invasive constrained generation
Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding llms the right way: Fast, non-invasive constrained generation. arXiv preprint arXiv:2403.06988, 2024
arXiv 2024
-
[4]
Tinyagent: Function calling at the edge
Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Tinyagent: Function calling at the edge. arXiv preprint arXiv:2409.00608, 2024
arXiv 2024
-
[5]
Permission re-delegation: Attacks and defenses
Adrienne Porter Felt, Helen J Wang, Alexander Moshchuk, Steve Hanna, and Erika Chin. Permission re-delegation: Attacks and defenses. In USENIX security symposium, volume 30, page 88, 2011
work page 2011
-
[6]
Mobileviews: A large-scale mobile gui dataset
Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.14337, 2024
arXiv 2024
-
[7]
Large language model based multi-agents: A survey of progress and challenges
Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024
arXiv 2024
-
[8]
(1961) shoebox - ibm archives (78-013)
IBM. (1961) shoebox - ibm archives (78-013). https://mediacenter.ibm.com/media/ (1961)+Shoebox+-+IBM+Archives+(78-013)/0_4m2ynnkk , 2024
work page 1961
Show all 27 references
-
[9]
Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts
Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-impro...
2024
-
[10]
Mobilegpt: Augmenting llm with human-like app memory for mobile task automation
Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augmenting llm with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Netwo...
2024
-
[11]
Personal llm agents: Insights and survey about the capability, efficiency and security
Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024
2024 arXiv
-
[12]
Turn every application into an agent: Towards efficient human-agent-computer interaction with api-first llm-based agents
Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Turn every application into an agent: Towards efficient human-agent-computer interaction with api-first llm-based agents. arXiv preprint arXiv:2409...
2024 arXiv
-
[13]
Small language models: Survey, measurements, and insights
Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790, 2024
2024 arXiv
-
[14]
Computer-using agent
OpenAI. Computer-using agent. https://openai.com/index/computer-using-agent/ , 2024. 11
2024
-
[15]
Androidworld: A dynamic benchmarking environment for autonomous agents
Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024
2024 arXiv
-
[16]
Autodroid: Llm-powered task automation in android
Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pag...
2024
-
[17]
Autodroid-v2: Boosting slm-based gui agents via code generation
Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, et al. Autodroid-v2: Boosting slm-based gui agents via code generation. arXiv preprint arXiv:2412.18116, 2024
2024 arXiv
-
[18]
Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments
Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024
2024 arXiv
-
[19]
Droidcall: A dataset for llm-powered android intent invocation
Weikai Xie, Li Zhang, Shihe Wang, Rongjie Yi, and Mengwei Xu. Droidcall: A dataset for llm-powered android intent invocation. arXiv preprint arXiv:2412.00402, 2024
2024 arXiv
-
[20]
A survey of resource-efficient llm and multimodal foundation models
Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092, 2024
2024 arXiv
-
[21]
Edgemoe: Fast on-device inference of moe-based large language models.arXiv preprint arXiv:2308.14352, 2023
Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Fast on-device inference of moe-based large language models.arXiv preprint arXiv:2308.14352, 2023
2023 arXiv
-
[22]
Phonelm: an efficient and capable small language model family through principled pre-training
Rongjie Yi, Xiang Li, Weikai Xie, Zhenyan Lu, Chenghua Wang, Ao Zhou, Shangguang Wang, Xiwen Zhang, and Mengwei Xu. Phonelm: an efficient and capable small language model family through principled pre-training. arXiv preprint arXiv:2411.05046, 2024
2024 arXiv
-
[23]
Llm as a system service on mobile devices
Wangsong Yin, Mengwei Xu, Yuanchun Li, and Xuanzhe Liu. Llm as a system service on mobile devices. arXiv preprint arXiv:2403.11805, 2024
2024 arXiv
-
[24]
Mobile foundation model as firmware
Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, et al. Mobile foundation model as firmware. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , pages 279–295, 2024
2024
-
[25]
Characterizing insecure javascript practices on the web
Chuan Yue and Haining Wang. Characterizing insecure javascript practices on the web. In Proceedings of the 18th international conference on World wide web, pages 961–970, 2009
2009
-
[26]
Large language model-brained gui agents: A survey
Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279, 2024
2024 arXiv
-
[27]
Llamatouch: A faithful and scalable testbed for mobile ui task automation
Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task automation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–1...
2024
Reviewed August 8, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.