Pith. sign in

REVIEW 4 major objections 6 minor 1 cited by

Every Software as an Agent: Blueprint and Case Study

T0 review · 4 major / 6 minor · reviewed 2026-08-08 · deepseek-v4-flash

Pith's one-line read The paper argues that software agents should stop clicking and start reading source code, then injecting generated code into the running program.

desk verdict Genuinely new whitebox code-injection idea with a thin but real feasibility demo; claims outrun evidence, but the direction deserves referee time. read the letter →

arxiv 2502.04747 v1 pith:P44ND3ZL submitted 2025-02-07 cs.SE cs.AI

classification cs.SEcs.AI
keywords softwareagentsjust-in-timecodegenerationlargelanguagemodelsactionruntimecontextGUIautomationagentsafetydesktopapplications
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes a third route for building software agents, alongside API-based and GUI-based agents: just-in-time code generation and in-software execution, which it calls JiT-Codegen. The idea is to let a large language model read the target software's source code and live runtime state, then generate short action code that is executed inside the running program, so a user's natural-language request becomes a code-editing task rather than a sequence of UI clicks. The paper argues that this whitebox design can reach any reachable runtime state in one generated action, avoids error accumulation across GUI steps, and handles tasks that have no GUI representation. A minimal prototype on two Electron desktop apps reports that GPT-4o completes 8 of 10 tasks, while other tested models complete 6 or 7, and that execution feedback can turn initial failures into working code. The authors position the approach as complementary to API and GUI agents and identify safety as the main unresolved challenge.

What carries the argument

The load-bearing mechanism is the JiT-Codegen loop connecting an LLM-powered Code Agent to an Execution Sandbox embedded in the software runtime. The Code Agent writes action code from the user instruction plus source-code context; the Sandbox runs that code in-process or via IPC against live data structures and returns runtime feedback such as error messages or execution results; the agent iterates. The paper names this just-in-time code generation and in-software execution and draws an analogy to JIT compilation and to eBPF, where injected code runs in a restricted kernel context. The sandbox is the component that must balance flexibility, safety, and developer effort, and its design depends on the software's language and framework.

What would settle it

Run JiT-Codegen on a fixed suite of tasks over a real desktop app with an implemented sandbox, static analysis, audit log, and rollback: if task-completion rate under those safeguards falls to the level of GUI-based baselines, or if a single generated action irreversibly corrupts data that rollback cannot restore, the central viability claim fails.

Watch

Extended reading notes

Core claim

The central claim is that the non-intrusive assumption shared by API-based and GUI-based agents—that the LLM cannot see or touch the software's internals—is the fundamental barrier to high-accuracy, efficient, general software agents. JiT-Codegen instead gives the LLM the source code, documentation, and runtime context, including in-memory data structures, open files, databases, and the UI tree, plus permission to inject generated code into the runtime through an execution sandbox. In the paper's state-machine framing, an API call reaches only pre-defined states and a GUI agent reaches intermediate states step by step, but a generated action can transition directly to the target state. The case study on two Electron apps, a markdown editor and a music player, reports an 80% task-completion rate with GPT-4o, including multi-round refinement where runtime error messages guide the next code generation. The conclusion is that JiT-Codegen can complement, and in some cases replace, API and GUI approaches, turning any software whose code is available into an agent.

Load-bearing premise

The load-bearing premise is that generated code with full memory and disk access can be contained by sandboxing and safety rules without destroying the flexibility that makes JiT-Codegen accurate and efficient; the paper discusses but does not implement or measure any of the four safeguards.

Editorial extensions

If this is right

  • If JiT-Codegen works at scale, a task that needs five GUI interactions can be finished with one generated action, eliminating per-step rendering and LLM invocation overhead.
  • Existing non-intrusive agents on desktop and mobile benchmarks typically complete under 20% of tasks, so the paper argues that the low ceiling comes less from LLM ability than from the interaction interface.
  • Because generated code is created on demand, tasks with no GUI representation become expressible as code, and states unreachable through pre-defined APIs become reachable.
  • Successful action code can be wrapped into a reusable API, so JiT-Codegen naturally bootstraps an API layer over time rather than conflicting with API-based design.
  • The approach's adoption hinges on four un-implemented safeguards: sandboxing, static analysis, audit logs and rollback, and rule-enforced LLM generation, which the paper states as its next step.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper leaves implicit that the same runtime access enabling one-step task completion also converts any mis-prompted or adversarially injected instruction into privileged code execution, so the practical test is whether a safety rule set blocks harmful actions without lowering the 8-of-10-level completion rate.
  • Because released software codebases are mostly static, a near-term extension is to pre-index each app's functions and data structures as a graph and retrieve only task-relevant slices before generation, a concrete test of the paper's accuracy proposal.
  • The multi-agent vision introduces a natural-language permission-delegation surface: one app agent can ask another to perform an action it lacks permission for, so auditing cross-agent requests is a direct stress test of the claimed digital-world scenario.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. The paper proposes a new software-agent paradigm, JiT-Codegen, in which an LLM-powered agent is given access to a program's source code and runtime state and is allowed to inject generated code directly into the running software. The authors argue that this whitebox approach is more accurate, more ubiquitous, and more efficient than API-based and GUI-based agents, and they support the proposal with a conceptual comparison (state-machine view and Table 1), a minimal Electron-based prototype, and a case study on two open-source desktop apps (marktext and YesPlayMusic) using five LLMs across ten tasks (Table 2). The paper concludes with challenges and future directions, with particular emphasis on safety, accuracy, deployment, and multi-software-agent systems.

Significance. If the approach can be made safe and its empirical claims substantiated, JiT-Codegen would be a meaningful reframing of software agents: it replaces many-step GUI manipulation with a single code-level action that can reach arbitrary runtime states, and it is complementary to existing API and GUI methods rather than mutually exclusive. The paper is candid about its status as a blueprint and gives concrete, reproducible details about the prototype size, the tested applications, and per-model task outcomes. Its strengths are transparency in reporting the 10-task results, a worked multi-round error-feedback example (Listing 1), and explicit recognition of safety as the primary obstacle. However, the evidence as presented supports feasibility, not the stronger claim of superiority over existing approaches: there are no same-task baselines, no repeated trials, no statistical analysis, and no implemented safety mechanisms.

major comments (4)
  1. [§4, Table 2] The central claim that JiT-Codegen achieves higher accuracy and efficiency than API-based and GUI-based agents is not tested comparatively. Table 2 reports success counts for five LLMs on ten self-selected tasks, but no API-based or GUI-based agent was run on the same tasks. The 80% success rate for GPT-4o therefore has no baseline against which superiority can be established. The manuscript should either run an existing GUI/API agent on the identical task set or clearly re-scope the claims to feasibility rather than superiority.
  2. [§4, Table 2 and Listing 2] Each task-model combination is run once, with no repeated trials, no error bars, and no statistical test. Given the stochasticity of LLM code generation and the manual judgment of success, the reported 8/10 figure is fragile. In addition, the manuscript states that failures are often 'salient' and that the agent stops unless the user gives extra feedback, yet Listing 2 shows that with manual prompting a workable solution for 'show my listening history' eventually appears for o1 after the table marks it as failed for all models. The evaluation protocol (how success is judged, whether extra prompting is allowed, and how the table and Listing 2 are reconciled) must be specified.
  3. [§4, Listing 1 vs. §3] The evidence does not isolate the proposed novelty. Listing 1 shows that success in 'increase the volume slightly' arises from the agent reacting to runtime error messages (TypeError, component-not-found) over multiple rounds. That learning signal is available to any code-execution agent, including a blackbox GUI-less agent, and does not by itself demonstrate that whitebox source-code access or in-runtime injection is the enabling factor. The authors should add an ablation that withholds source code and/or runtime-context access while keeping code execution and error feedback, or otherwise discuss why the current results cannot distinguish these mechanisms.
  4. [§5.1] Safety is acknowledged as the primary challenge, but the four proposed safeguarding techniques (sandboxing, static analysis, audit logs, and rule-enforced generation) are only discussed, not implemented or evaluated. The paper itself states that action code 'could do harmful and irreversible damage' and that flexibility comes at a safety cost. Because the practical viability and deployability of JiT-Codegen depend on containing such code, at least a minimal feasibility demonstration of one safeguarding mechanism, or an explicit statement that safety is outside the scope of this blue-print, is needed before the broader 'fundamentally overturn' claim can be taken as supported.
minor comments (6)
  1. [Abstract and title page] There is a typo in the author affiliation: 'Beiing University' should be 'Beijing University'.
  2. [Table 1] The checkmark scales (✓, ✓✓, ✗, ✗✗) are not defined; the reader cannot tell whether ✓✓ is stronger than ✓ or whether ✗✗ is weaker than ✗. Please add a legend or use a consistent ordinal scale.
  3. [Figure 4] The marks on the code snippets are visually ambiguous and appear inconsistent with the model labels (for example, the symbols next to GPT-4o and o3-mini are not clearly aligned with the reported success/failure outcomes). Please redraw the figure with explicit pass/fail labels.
  4. [§4] The prompt templates, source-code retrieval method, and sandbox implementation are not described in enough detail for reproduction. The paper reports the prototype line counts but does not provide a code repository or appendix; adding these would substantially strengthen the reproducibility of the case study.
  5. [§5.2] The statement that 'most unmanaged programming language like C/Rust do not allow catching runtime error' is imprecise: C and Rust do not have runtime exceptions in the same way as managed languages, but they do provide signal handling, panic hooks, and other error mechanisms. Please rephrase.
  6. [§4] The sentence attributing the lower performance of o1 and o3-mini to 'ineffective prompt engineering' is speculative; no evidence is given that prompt engineering, rather than model behavior or task selection, caused the difference.

Circularity Check

0 steps flagged · score 0.0 of 10

No circular derivation; the case study is an unbaselined empirical demonstration, and the self-citations are contextual rather than load-bearing.

full rationale

The paper makes no quantitative derivation or fitted-parameter prediction that could reduce to its inputs. Its central proposal, JiT-Codegen, is a design thesis; the case study reports direct empirical counts in Table 2 (e.g., GPT-4o solving 8/10 tasks), with no model fitting, no learned parameters, and no closed-form relation that would make a measured outcome true by construction. The self-citations (e.g., [27] Llamatouch, [19] DroidCall, and the author's surveys) appear in background, motivation, and future-work discussion; the claim that existing agents have low completion rates is supported by cited benchmarks including external ones ([15], [18]), and the cited self-authored artifacts are benchmarks or surveys rather than results that presuppose JiT-Codegen. The absence of API/GUI baselines or an ablation is a real evidentiary limitation for the comparative claim, but absence of evidence is not circularity: the 80% figure is a measured outcome, not a consequence of defining JiT-Codegen to be accurate. No step in the paper's argument is equivalent to its inputs by definition, and no load-bearing premise is imported solely from the author's prior work. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim depends on three domain assumptions rather than free parameters or invented entities. No constants are fitted; the reported success rates are empirical counts. The safety assumption is the least supported part: none of the four safeguarding mechanisms in Section 5.1 is implemented or benchmarked.

assumptions (3)
  • domain assumption The agent has full access to the software's source code and runtime context.
    Stated in Section 3: "JiT-Codegen-based agents assume full access to the software's source code and should be deployed by developers." This is necessary for code generation against runtime internals like vueInstance.$store.
  • domain assumption Generated action code can be safely constrained so it does not cause harmful or irreversible consequences.
    Section 5.1 opens with "flexibility comes with cost on safety" and proposes sandboxing, static analysis, audit logs, and rule-enforced generation without implementing or evaluating any of them. The whole practical viability depends on this.
  • domain assumption LLM-generated action code, given source context and execution feedback, is accurate enough for task completion.
    The case study supports this for 7 to 8 of 10 hand-picked tasks, but the failures and the manual prompting needed for "show my listening history" show it is not guaranteed. This assumption underlies the accuracy claim in Table 1.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Every Software as an Agent: Blueprint and Case Study." pith.science (2026). https://pith.science/paper/P44ND3ZL

@misc{pith2026250204747,
  author       = {Pith},
  title        = {Pith review of: Every Software as an Agent: Blueprint and Case Study},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/P44ND3ZL}},
  note         = {Machine review of arXiv:2502.04747}
}
read the original abstract

The rise of (multimodal) large language models (LLMs) has shed light on software agent -- where software can understand and follow user instructions in natural language. However, existing approaches such as API-based and GUI-based agents are far from satisfactory at accuracy and efficiency aspects. Instead, we advocate to endow LLMs with access to the software internals (source code and runtime context) and the permission to dynamically inject generated code into software for execution. In such a whitebox setting, one may better leverage the software context and the coding ability of LLMs. We then present an overall design architecture and case studies on two popular web-based desktop applications. We also give in-depth discussion of the challenges and future directions. We deem that such a new paradigm has the potential to fundamentally overturn the existing software agent design, and finally creating a digital world in which software can comprehend, operate, collaborate, and even think to meet complex user needs.

Figures

Figures reproduced from arXiv: 2502.04747 by the authors.

Figure 1
Figure 1. An illustrative example showing the difference between GUI-based agent and our proposal [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. An overview of software agent using JiT-Codegen. 2 The Status Quo and Their Inadequacies While the efforts to enable natural language communication between users and software date back to the 1960s [8], the recent emergence of large language models (LLMs) has significantly advanced the practical application of this concept. This section provides an overview of the current practice to build software agents. API-based… view at source ↗
Figure 3
Figure 3. A comparison of software agents from the perspective of state machines. [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figures from the paper (2 more)
Figure 4
Figure 4. Figure 4: Examples of the action code generated by different LLMs. [PITH_FULL_IMAGE:figures/full_fig_p006_4.png]
Figure 5
Figure 5. Figure 5: An example showing how multiple software agents collectively solve a user task. [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Vibe Coding vs. Agentic Coding: Fundamentals and Practical Implications of Agentic AI

    cs.SE 2025-05 conditional novelty 3.0 of 10

    A qualitative taxonomy positions vibe coding and agentic coding as complementary paradigms rather than rivals in AI-assisted software development.

Reference graph

Works this paper leans on

27 extracted references · 8 canonical work pages · cited by 1 Pith paper

  1. [1]

    Anthropic computer use

    Anthropic. Anthropic computer use. https://docs.anthropic.com/en/docs/ build-with-claude/computer-use#how-computer-use-works , 2024

  2. [2]

    Prompting is programming: A query language for large language models

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Prompting is programming: A query language for large language models. Proceedings of the ACM on Programming Languages, 7(PLDI):1946–1969, 2023

  3. [3]

    Guiding llms the right way: Fast, non-invasive constrained generation

    Luca Beurer-Kellner, Marc Fischer, and Martin Vechev. Guiding llms the right way: Fast, non-invasive constrained generation. arXiv preprint arXiv:2403.06988, 2024

  4. [4]

    Tinyagent: Function calling at the edge

    Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. Tinyagent: Function calling at the edge. arXiv preprint arXiv:2409.00608, 2024

  5. [5]

    Permission re-delegation: Attacks and defenses

    Adrienne Porter Felt, Helen J Wang, Alexander Moshchuk, Steve Hanna, and Erika Chin. Permission re-delegation: Attacks and defenses. In USENIX security symposium, volume 30, page 88, 2011

  6. [6]

    Mobileviews: A large-scale mobile gui dataset

    Longxi Gao, Li Zhang, Shihe Wang, Shangguang Wang, Yuanchun Li, and Mengwei Xu. Mobileviews: A large-scale mobile gui dataset. arXiv preprint arXiv:2409.14337, 2024

  7. [7]

    Large language model based multi-agents: A survey of progress and challenges

    Taicheng Guo, Xiuying Chen, Yaqi Wang, Ruidi Chang, Shichao Pei, Nitesh V Chawla, Olaf Wiest, and Xiangliang Zhang. Large language model based multi-agents: A survey of progress and challenges. arXiv preprint arXiv:2402.01680, 2024

  8. [8]

    (1961) shoebox - ibm archives (78-013)

    IBM. (1961) shoebox - ibm archives (78-013). https://mediacenter.ibm.com/media/ (1961)+Shoebox+-+IBM+Archives+(78-013)/0_4m2ynnkk , 2024

Show all 27 references
  1. [9]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts. Dspy: Compiling declarative language model calls into self-impro...

  2. [10]

    Mobilegpt: Augmenting llm with human-like app memory for mobile task automation

    Sunjae Lee, Junyoung Choi, Jungjae Lee, Munim Hasan Wasi, Hojun Choi, Steve Ko, Sangeun Oh, and Insik Shin. Mobilegpt: Augmenting llm with human-like app memory for mobile task automation. In Proceedings of the 30th Annual International Conference on Mobile Computing and Netwo...

  3. [11]

    Personal llm agents: Insights and survey about the capability, efficiency and security

    Yuanchun Li, Hao Wen, Weijun Wang, Xiangyu Li, Yizhen Yuan, Guohong Liu, Jiacheng Liu, Wenxing Xu, Xiang Wang, Yi Sun, et al. Personal llm agents: Insights and survey about the capability, efficiency and security. arXiv preprint arXiv:2401.05459, 2024

  4. [12]

    Turn every application into an agent: Towards efficient human-agent-computer interaction with api-first llm-based agents

    Junting Lu, Zhiyang Zhang, Fangkai Yang, Jue Zhang, Lu Wang, Chao Du, Qingwei Lin, Saravan Rajmohan, Dongmei Zhang, and Qi Zhang. Turn every application into an agent: Towards efficient human-agent-computer interaction with api-first llm-based agents. arXiv preprint arXiv:2409...

  5. [13]

    Small language models: Survey, measurements, and insights

    Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D Lane, and Mengwei Xu. Small language models: Survey, measurements, and insights. arXiv preprint arXiv:2409.15790, 2024

  6. [14]

    Computer-using agent

    OpenAI. Computer-using agent. https://openai.com/index/computer-using-agent/ , 2024. 11

  7. [15]

    Androidworld: A dynamic benchmarking environment for autonomous agents

    Christopher Rawles, Sarah Clinckemaillie, Yifan Chang, Jonathan Waltz, Gabrielle Lau, Mary- beth Fair, Alice Li, William Bishop, Wei Li, Folawiyo Campbell-Ajala, et al. Androidworld: A dynamic benchmarking environment for autonomous agents. arXiv preprint arXiv:2405.14573, 2024

  8. [16]

    Autodroid: Llm-powered task automation in android

    Hao Wen, Yuanchun Li, Guohong Liu, Shanhui Zhao, Tao Yu, Toby Jia-Jun Li, Shiqi Jiang, Yunhao Liu, Yaqin Zhang, and Yunxin Liu. Autodroid: Llm-powered task automation in android. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking, pag...

  9. [17]

    Autodroid-v2: Boosting slm-based gui agents via code generation

    Hao Wen, Shizuo Tian, Borislav Pavlov, Wenjie Du, Yixuan Li, Ge Chang, Shanhui Zhao, Jiacheng Liu, Yunxin Liu, Ya-Qin Zhang, et al. Autodroid-v2: Boosting slm-based gui agents via code generation. arXiv preprint arXiv:2412.18116, 2024

  10. [18]

    Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments

    Tianbao Xie, Danyang Zhang, Jixuan Chen, Xiaochuan Li, Siheng Zhao, Ruisheng Cao, Toh Jing Hua, Zhoujun Cheng, Dongchan Shin, Fangyu Lei, et al. Osworld: Benchmarking multimodal agents for open-ended tasks in real computer environments. arXiv preprint arXiv:2404.07972, 2024

  11. [19]

    Droidcall: A dataset for llm-powered android intent invocation

    Weikai Xie, Li Zhang, Shihe Wang, Rongjie Yi, and Mengwei Xu. Droidcall: A dataset for llm-powered android intent invocation. arXiv preprint arXiv:2412.00402, 2024

  12. [20]

    A survey of resource-efficient llm and multimodal foundation models

    Mengwei Xu, Wangsong Yin, Dongqi Cai, Rongjie Yi, Daliang Xu, Qipeng Wang, Bingyang Wu, Yihao Zhao, Chen Yang, Shihe Wang, et al. A survey of resource-efficient llm and multimodal foundation models. arXiv preprint arXiv:2401.08092, 2024

  13. [21]

    Edgemoe: Fast on-device inference of moe-based large language models.arXiv preprint arXiv:2308.14352, 2023

    Rongjie Yi, Liwei Guo, Shiyun Wei, Ao Zhou, Shangguang Wang, and Mengwei Xu. Edgemoe: Fast on-device inference of moe-based large language models.arXiv preprint arXiv:2308.14352, 2023

  14. [22]

    Phonelm: an efficient and capable small language model family through principled pre-training

    Rongjie Yi, Xiang Li, Weikai Xie, Zhenyan Lu, Chenghua Wang, Ao Zhou, Shangguang Wang, Xiwen Zhang, and Mengwei Xu. Phonelm: an efficient and capable small language model family through principled pre-training. arXiv preprint arXiv:2411.05046, 2024

  15. [23]

    Llm as a system service on mobile devices

    Wangsong Yin, Mengwei Xu, Yuanchun Li, and Xuanzhe Liu. Llm as a system service on mobile devices. arXiv preprint arXiv:2403.11805, 2024

  16. [24]

    Mobile foundation model as firmware

    Jinliang Yuan, Chen Yang, Dongqi Cai, Shihe Wang, Xin Yuan, Zeling Zhang, Xiang Li, Dingge Zhang, Hanzi Mei, Xianqing Jia, et al. Mobile foundation model as firmware. In Proceedings of the 30th Annual International Conference on Mobile Computing and Networking , pages 279–295, 2024

  17. [25]

    Characterizing insecure javascript practices on the web

    Chuan Yue and Haining Wang. Characterizing insecure javascript practices on the web. In Proceedings of the 18th international conference on World wide web, pages 961–970, 2009

  18. [26]

    Large language model-brained gui agents: A survey

    Chaoyun Zhang, Shilin He, Jiaxu Qian, Bowen Li, Liqun Li, Si Qin, Yu Kang, Minghua Ma, Qingwei Lin, Saravan Rajmohan, et al. Large language model-brained gui agents: A survey. arXiv preprint arXiv:2411.18279, 2024

  19. [27]

    Llamatouch: A faithful and scalable testbed for mobile ui task automation

    Li Zhang, Shihe Wang, Xianqing Jia, Zhihan Zheng, Yunhe Yan, Longxi Gao, Yuanchun Li, and Mengwei Xu. Llamatouch: A faithful and scalable testbed for mobile ui task automation. In Proceedings of the 37th Annual ACM Symposium on User Interface Software and Technology, pages 1–1...

Pith tools

Reviewed August 8, 2026 · model on record in the stance chip above.