Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

A Fast, Reliable, and Secure Programming Language for LLM Agents with Code Actions

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Pith's one-line read Transpiling agent Python into Quasar preserves accuracy while cutting runtime 42%, approvals 52%, and adding conformal uncertainty.

desk verdict A useful integration of parallel dispatch, batched approval, and conformal semantics for LLM agent code, but the headline accuracy claim is weakened by a 6-point end-to-end drop and the purity assumption is asserted, not enforced. read the letter →

arxiv 2506.12202 v1 pith:QKQNAQCX submitted 2025-06-13 cs.PL cs.AIcs.CRcs.LG

classification cs.PLcs.AIcs.CRcs.LG
keywords LLMagentscodeactionstranspilationparallelexecutionconformalpredictionaccesscontrolvisualquestionansweringprogramminglanguagedesign
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces Quasar, a programming language for LLM agents that write code to call external tools. The key move is to transpile a subset of Python, the language LLMs actually generate, into Quasar, whose pure functional core isolates all side effects in external calls. Because internal rewrites are effect-free, the interpreter can run external calls in parallel, collect them for batch user approval, and execute programs over sets of values instead of single values. On the ViperGPT visual question answering agent applied to GQA, the authors report that Quasar programs retain Python-level accuracy while reducing execution time by 42% and approval interactions by 52% on tasks where those reductions are possible, and that conformal semantics hit a target 10% error rate.

What carries the argument

The machinery is a small functional core language with rewrite rules: a program is a sequence of single-assignment statements, external calls are dispatched as soon as all their arguments are values, and internal rules (alias, projection, if-true/false, fold) rewrite the program until a terminal result. Side effects live only in external calls, which are invoked through a single rule that spawns a thread and later substitutes the returned value. The interpreter's strategy is to run internal rewrites as far as possible, collect all dispatchable external calls, ask the user once to approve the batch, then launch them in parallel; the same structure supports conformal semantics by adding abstract values (sets, maybe-lists, joins) and matching rewrite rules, with externally supplied conformal abstractions for model calls. The transpiler from a Python subset (function calls, assignments, if/for/while, no early returns) converts imperative control flow into functional folds and expression-level conditionals.

What would settle it

Run an equivalent Python and Quasar program whose external calls include a function that appends to a list or file; if the transcript of external calls (order, arguments, or repetition) differs between the two executions, the purity assumption behind the semantics is violated.

Watch

Extended reading notes

Core claim

The central claim is that an LLM agent can get the reliability and security benefits of a purpose-built language without giving up the fluency it already has in Python: generate Python in a restricted subset, transpile it to Quasar, and execute under Quasar's rewrite semantics. Quasar's semantics split computation into internal rules, which are pure and can be applied in any order, and one external rule that dispatches calls to user-approved functions. This makes all external calls first-class, so the interpreter can batch approvals and dispatch them in parallel, and it lets conformal evaluations propagate sets of possible values through the program. The paper reports 90.6% successful execution for transpiled programs versus 99.7% for raw Python, with identical 70.6% VQA accuracy on successful programs; on the roughly 41% of tasks with exploitable parallelism, execution time drops 42% ± 22; on the 42% of tasks with batchable approvals, interaction count drops 52% ± 19; and the conformal run hits 9.1% ± 1.9 test error against a 10% target.

Load-bearing premise

The load-bearing premise is that every external call in an agent program is a pure function of its inputs, so reordering and parallelizing calls cannot change the program's effects.

Editorial extensions

If this is right

  • LLM agents that already write Python can adopt Quasar without learning a new language or changing prompts; the transpiler does the conversion.
  • Tasks with independent tool calls should see wall-clock time roughly halved, and approval round-trips collapse to one interaction per batch.
  • Conformal execution turns every model-backed call into a set-valued result, so an agent can report 'I am not sure' instead of a single hallucinated answer, with a user-set target coverage.
  • The security surface is exactly the set of external calls: internal pure computation cannot leak or mutate state, so validation can focus on the call list.
  • Transpiled generation makes far fewer translation errors than LLM-generated Quasar (6.9x to 7.6x fewer), so the bottleneck shifts to the LLM adhering to the Python subset.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Inference: the same rewrite semantics would work for any source language that can be lowered to the functional core, so the Python subset is a convenience for LLM fluency rather than a fundamental constraint.
  • Inference: if external calls violate purity (e.g., a logging API or random sampler), the 'same set of external calls' guarantee breaks; a natural extension is to make side-effecting calls explicit state tokens threaded through the dependency graph, restoring deterministic reordering.
  • Inference: the conformal semantics could be used to drive interactive clarification: when a prediction set spans multiple plausible answers, the agent could ask the user one targeted question instead of committing to a single label.
  • Inference: the reported gains are measured on tasks already parallelizable or batchable; expected end-to-end speedup in a new domain depends on the fraction of such tasks, not just on the 42% and 52% headline numbers.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper introduces Quasar, a functional programming language for LLM agents that execute code actions, in which side effects are isolated in external calls. LLMs generate a restricted subset of Python, which is transpiled to Quasar and executed by an interpreter that parallelizes external calls, batches user approval requests, and supports set-valued conformal execution. The evaluation on the ViperGPT visual question answering agent over GQA claims task accuracy comparable to Python, a 42% reduction in execution time on improvable tasks, a 52% reduction in user-approval interactions on improvable tasks, and a conformal prediction guarantee targeting a 0.1 error rate. The central soundness property is stated in Section 3.1 as a confluence result conditional on the assumption that each external call depends only on its inputs.

Significance. The paper proposes a promising direction for LLM agent code actions by separating pure internal computation from effectful external calls, transpiling a Python subset into a functional core, batching user approvals, and extending execution to set-valued conformal semantics. A clear strength is the generation strategy evaluation: transpilation produces 6.9x and 7.6x fewer erroneous programs than LLM-based translation and direct Quasar generation, respectively, and the implementation is publicly available. However, the headline performance, security, and reliability claims are conditional on an unenforced purity assumption, and the accuracy comparison is weakened by conditioning on successful execution. If the purity assumption can be enforced or the experiments redone with real concurrent execution, and if end-to-end accuracy is reported, the contributions would be significant for the programming-languages and LLM-agent communities.

major comments (5)
  1. [§3.1 (also §3.2–§3.4, §4.2)] The load-bearing 'same effects' property is stated as an assumption ('Assuming each external call only depends on its inputs'), but no proof is given and no enforcement mechanism is described. External functions are implemented in arbitrary Python (Section 3.1 explicitly allows effects such as printing or calling an LLM), and in the ViperGPT domain the external calls are stochastic neural models whose concurrent execution can change outputs through random sampling, shared GPU state, or rate-limited APIs. The parallel dispatch of Section 3.3, the batched-approval security argument of Section 3.2, and the conformal semantics of Section 3.4 all rely on this property. The paper should either prove the confluence result under the stated assumption, add an effect system or runtime check that verifies purity, or empirically test whether the assumption holds for the target external calls.
  2. [§4.1, Table 1] The claim that 'the accuracy of QUASAR programs is comparable to that of Python programs' is not supported by the table as reported. Successful execution drops from 99.7% for Python to 90.6% for the transpiled approach, and VQA Accuracy is computed only over successful programs. The end-to-end accuracy is therefore approximately 70.4% for Python versus 64.0% for the transpiled approach (0.997 × 0.706 vs. 0.906 × 0.706), a relative drop of about 9%. The paper should report end-to-end accuracy and discuss this trade-off explicitly.
  3. [§4.2] The performance experiment replays recorded external-call results and running times on both Python and QUASAR. This measures scheduling speedup under the purity assumption, not end-to-end wall-clock time with real concurrent execution; it does not account for resource contention, shared GPU state, or rate-limited APIs, and it assumes that executing calls concurrently does not change their outputs. The headline '42% reduction' should be qualified as a simulated scheduling improvement on improvable tasks, and an experiment with real parallel execution would be needed to support a running-time claim.
  4. [§4.4] The conformal evaluation selects thresholds on validation splits (the largest threshold with validation error below 0.1) and then reports test error. This is a model-selection procedure, not the conformal rescaling with a held-out calibration set described in Section 3.4, and it does not provide a distribution-free coverage guarantee. The reported 61.4% uncertainty rate also indicates that the prediction sets are large, so the practical utility of the reliability improvement is unclear. Please clarify the protocol and report the end-to-end accuracy achieved with set-valued outputs.
  5. [§3.2] The security evaluation measures a reduction in the number of user-approval interactions, which is a usability improvement, not a security guarantee. The access-control argument assumes that each external call has known, controllable effects; however, external functions are implemented as arbitrary Python (e.g., they may print or call an LLM) and are not sandboxed or statically checked. If an approved external function has hidden effects, batching approvals does not make execution more secure. The paper should either restrict external functions to a verified interface or state this limitation explicitly.
minor comments (6)
  1. [Appendix C, fold-abs rule] The text says 'if bi is False the element ci may or may not be in the list, whereas if bi is False, then ci is definitely in the list'; the second occurrence of 'False' should be 'True'.
  2. [§4.4, Figure 4d] The text refers to a 'target coverage rate of 0.1' and then reports 'empirical error'; the wording should distinguish coverage probability from error rate.
  3. [Figure 2 and §3.2] There are typographical errors: 'path2.simple_query' should be 'patch2.simple_query' in §3.2, and 'rools' in Figure 2 should be 'rules'.
  4. [Table 1] Adding a row for end-to-end accuracy (successful execution × VQA accuracy) would make the comparison transparent and would prevent misinterpretation of 'comparable' accuracy.
  5. [§3.1 and Algorithm 1] The text says external calls are executed 'as soon as all of their arguments are available,' but Algorithm 1 first queries the user before dispatching; the relationship between eager dispatch and user approval should be clarified.
  6. [References] Reference [8] is formatted inconsistently as 'e. a. Jason Wei'; the author list should be formatted in the same style as the other references.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Quasar's central claims are evaluated against external benchmarks and standard conformal calibration; self-citations are building blocks, not load-bearing reductions.

full rationale

The derivation chain is self-contained rather than circular. The accuracy claim (Section 4.1) is measured on the external GQA/ViperGPT benchmark rather than computed from Quasar's definition. The performance result (Section 4.2) is measured by replaying recorded external-call times on equivalent Python and Quasar programs; this is a scheduling measurement, not a fitted parameter renamed as a prediction. The security interaction reduction (Section 4.3) compares one-at-a-time approval against Quasar's batched approval, and the reported 22%/52% figures depend on the dataset's dependency structure, so they are not forced by the definition alone. The conformal coverage result (Section 4.4) uses a held-out validation set to select a threshold and then reports test error, following standard split-conformal methodology rather than defining the target in terms of the output. The self-citations to prior work on parallel lambda calculus [16] and compositional conformal prediction [18] are used as building-block techniques, but the paper's central benefit claims are not stated as consequences of those citations; the confluence property in Section 3.1 is asserted as an assumption with an unproved 'can be shown' and an unenforced purity condition, which is a correctness risk rather than a circular step. No equation or fitted quantity is reused as the claimed prediction, so no circularity is established.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces a language and runtime rather than a new physical entity. The empirical claims rest on conformal calibration parameters and on the purity and exchangeability assumptions. The subset-adherence assumption is tested empirically.

free parameters (2)
  • per-model conformal thresholds = not reported
    Section 4.4 says 'we optimize the thresholds for each individual model on an optimization set [12]'. These thresholds determine prediction set sizes and are fitted to data, affecting coverage and uncertainty rate.
  • scaling parameter tau = chosen via conformal calibration
    Section 3.4: 'we jointly rescale these thresholds using a single scaling parameter tau in R chosen using conformal prediction to satisfy a desired coverage guarantee [2, 31]'. tau is a free parameter fitted on a held-out calibration set.
assumptions (3)
  • domain assumption External calls depend only on their inputs (purity)
    Stated in Section 3.1 after Algorithm 1: 'Assuming each external call only depends on its inputs, then it can be shown that any sequence of rule applications results in the same set of external calls, and therefore the same effects.' This justifies parallel execution.
  • standard math Calibration and test data are exchangeable
    Conformal prediction's coverage guarantee requires exchangeability; the evaluation uses 100 random validation/test splits of GQA.
  • domain assumption LLMs can write code in the supported Python subset
    The transpilation strategy depends on the LLM following subset restrictions; the paper measures a 90.6% success rate, so this assumption is partially validated but not guaranteed for other agents.

how reviews work

0 comments
Cite this review

Pith. "Pith review of A Fast, Reliable, and Secure Programming Language for LLM Agents with Code Actions." pith.science (2026). https://pith.science/paper/QKQNAQCX

@misc{pith2026250612202,
  author       = {Pith},
  title        = {Pith review of: A Fast, Reliable, and Secure Programming Language for LLM Agents with Code Actions},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QKQNAQCX}},
  note         = {Machine review of arXiv:2506.12202}
}
read the original abstract

Modern large language models (LLMs) are often deployed as agents, calling external tools adaptively to solve tasks. Rather than directly calling tools, it can be more effective for LLMs to write code to perform the tool calls, enabling them to automatically generate complex control flow such as conditionals and loops. Such code actions are typically provided as Python code, since LLMs are quite proficient at it; however, Python may not be the ideal language due to limited built-in support for performance, security, and reliability. We propose a novel programming language for code actions, called Quasar, which has several benefits: (1) automated parallelization to improve performance, (2) uncertainty quantification to improve reliability and mitigate hallucinations, and (3) security features enabling the user to validate actions. LLMs can write code in a subset of Python, which is automatically transpiled to Quasar. We evaluate our approach on the ViperGPT visual question answering agent, applied to the GQA dataset, demonstrating that LLMs with Quasar actions instead of Python actions retain strong performance, while reducing execution time when possible by 42%, improving security by reducing user approval interactions when possible by 52%, and improving reliability by applying conformal prediction to achieve a desired target coverage level.

Figures

Figures reproduced from arXiv: 2506.12202 by the authors.

Figure 1
Figure 1. Illustrative example of an image and a natural language question about that image. We [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Given program P1 for the question in [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. An example of the same agent code, in both Python (a) and raw Q [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗
Figures from the paper (6 more)
Figure 4
Figure 4. Figure 4: An overview of the improvements (mean ± stddev) provided by QUASAR (a). Python vs QUASAR running time for the improvable tasks (b). Python vs QUASAR user interactions required for the improvable tasks (c). Using the conformal semantics and targeting 0.1 coverage, the d…
Figure 5
Figure 5. Figure 5: The grammar defining programs in QUASAR. A Full QUASAR Language As described in Section 3, QUASAR executes programs by transforming them with rewrite rules until they reach a result. The syntax of programs is given in [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]
Figure 6
Figure 6. Figure 6: The rewrite rules of the semantics of QUASAR (a), and the formal definition of the value function used by the “disp” rule (b). T[stmt] means a program P with some statement stmt in it; T[[stmts]] means that the statement stmt was replaced by the list of statements stmt…
Figure 7
Figure 7. Figure 7: An illustration of the translation of “if” statements, shown in Python syntax. Before, [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The additional operations in the grammar of Q [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: The additional rewrite rules in the semantics of [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The LLMbda Calculus: AI Agents, Conversations, and Information Flow

    cs.PL 2026-02 reject novelty 7.0 of 10

    A call-by-value lambda calculus with LLM generation and dynamic labels is shown to satisfy termination-insensitive noninterference on a restricted fragment, with a supposedly Lean-verified interpreter.

Reference graph

Works this paper leans on

32 extracted references · 29 canonical work pages · cited by 1 Pith paper

  1. [1]

    Andriushchenko, A

    M. Andriushchenko, A. Souly, M. Dziemian, D. Duenas, M. Lin, J. Wang, D. Hendrycks, A. Zou, Z. Kolter, M. Fredrikson, E. Winsor, J. Wynne, Y . Gal, and X. Davies. Agentharm: A benchmark for measuring harmfulness of llm agents, 2025

  2. [2]

    A. N. Angelopoulos, S. Bates, E. J. Candès, M. I. Jordan, and L. Lei. Learn then test: Calibrating predictive algorithms to achieve risk control, 2022

  3. [3]

    Claude’s extended thinking, 2025

    Anthropic. Claude’s extended thinking, 2025

  4. [4]

    Debenedetti, I

    E. Debenedetti, I. Shumailov, T. Fan, J. Hayes, N. Carlini, D. Fabian, C. Kern, C. Shi, A. Terzis, and F. Tramèr. Defeating prompt injections by design, 2025

  5. [5]

    F. He, T. Zhu, D. Ye, B. Liu, W. Zhou, and P. S. Yu. The emerged security and privacy of llm agent: A survey with case studies, 2024

  6. [6]

    Huang, J

    D. Huang, J. M. Zhang, M. Luck, Q. Bu, Y . Qing, and H. Cui. Agentcoder: Multi-agent-based code generation with iterative testing and optimisation, 2024

  7. [7]

    D. A. Hudson and C. D. Manning. Gqa: A new dataset for real-world visual reasoning and compositional question answering.Conference on Computer Vision and Pattern Recognition (CVPR), 2019

  8. [8]

    e. a. Jason Wei. Chain-of-thought prompting elicits reasoning in large language models, 2023

Show all 32 references
  1. [9]

    Karpukhin, B

    V . Karpukhin, B. O˘guz, S. Min, P. Lewis, L. Wu, S. Edunov, D. Chen, and W. tau Yih. Dense passage retrieval for open-domain question answering, 2020

  2. [10]

    Lewis, E

    P. Lewis, E. Perez, A. Piktus, F. Petroni, V . Karpukhin, N. Goyal, H. Küttler, M. Lewis, W. tau Yih, T. Rocktäschel, S. Riedel, and D. Kiela. Retrieval-augmented generation for knowledge- intensive nlp tasks, 2021

  3. [11]

    S. Li, A. Kan, L. Callot, B. Bhasker, M. S. Rashid, and T. B. Esler. Redo: Execution-free runtime error detection for coding agents, 2024

  4. [12]

    S. Li, S. Park, I. Lee, and O. Bastani. Traq: Trustworthy retrieval augmented question answering via conformal prediction, 2024

  5. [13]

    L. Liu, Y . Pan, X. Li, and G. Chen. Uncertainty estimation and quantification for llms: A simple supervised approach, 2024

  6. [14]

    N. Liu, L. Chen, X. Tian, W. Zou, K. Chen, and M. Cui. From llm to conversational agent: A memory enhanced architecture with fine-tuning of large language models, 2024

  7. [15]

    Maharana, D.-H

    A. Maharana, D.-H. Lee, S. Tulyakov, M. Bansal, F. Barbieri, and Y . Fang. Evaluating very long-term conversational memory of llm agents, 2024

  8. [16]

    S. Mell, K. Kallas, S. Zdancewic, and O. Bastani. Opportunistically parallel lambda calculus. or, lambda: The ultimate llm scripting language, 2025

  9. [17]

    R. Puri, D. S. Kung, G. Janssen, W. Zhang, G. Domeniconi, V . Zolotov, J. Dolby, J. Chen, M. Choudhury, L. Decker, V . Thost, L. Buratti, S. Pujar, S. Ramji, U. Finkler, S. Malaika, and F. Reiss. Codenet: A large-scale ai for code dataset for learning a diversity of coding tasks, 2021

  10. [18]

    Ramalingam, S

    R. Ramalingam, S. Park, and O. Bastani. Uncertainty quantification for neurosymbolic programs via compositional conformal prediction, 2024

  11. [19]

    Sandhu and P

    R. Sandhu and P. Samarati. Access control: principle and practice.IEEE Communications Magazine, 32(9):40–48, 1994

  12. [20]

    R. S. Sandhu. Role-based access control11portions of this chapter have been published earlier in sandhu et al. (1996), sandhu (1996), sandhu and bhamidipati (1997), sandhu et al. (1997) and sandhu and feinstein (1994). volume 46 ofAdvances in Computers, pages 237–286. Elsevier...

  13. [21]

    Shypula, S

    A. Shypula, S. Li, B. Zhang, V . Padmakumar, K. Yin, and O. Bastani. Evaluating the diversity and quality of llm generated content, 2025

  14. [22]

    Shypula, A

    A. Shypula, A. Madaan, Y . Zeng, U. Alon, J. Gardner, M. Hashemi, G. Neubig, P. Ranganathan, O. Bastani, and A. Yazdanbakhsh. Learning performance-improving code edits, 2024

  15. [23]

    Surís, S

    D. Surís, S. Menon, and C. V ondrick. Vipergpt: Visual inference via python execution for reasoning, 2023

  16. [24]

    Trivedi, T

    H. Trivedi, T. Khot, M. Hartmann, R. Manku, V . Dong, E. Li, S. Gupta, A. Sabharwal, and N. Balasubramanian. Appworld: A controllable world of apps and people for benchmarking interactive coding agents, 2024

  17. [25]

    V ovk, A

    V . V ovk, A. Gammerman, and G. Shafer.Algorithmic learning in a random world, volume 29. Springer, 2005

  18. [26]

    L. Wang, C. Ma, X. Feng, Z. Zhang, H. Yang, J. Zhang, Z. Chen, J. Tang, X. Chen, Y . Lin, W. X. Zhao, Z. Wei, and J. Wen. A survey on large language model based autonomous agents. Frontiers of Computer Science, 18(6), Mar. 2024

  19. [27]

    X. Wang, Y . Chen, L. Yuan, Y . Zhang, Y . Li, H. Peng, and H. Ji. Executable code actions elicit better llm agents, 2024

  20. [28]

    J. Yang, C. E. Jimenez, A. Wettig, K. Lieret, S. Yao, K. Narasimhan, and O. Press. Swe-agent: Agent-computer interfaces enable automated software engineering, 2024

  21. [29]

    S. Yao, D. Yu, J. Zhao, I. Shafran, T. L. Griffiths, Y . Cao, and K. Narasimhan. Tree of thoughts: Deliberate problem solving with large language models, 2023

  22. [30]

    S. Yao, J. Zhao, D. Yu, N. Du, I. Shafran, K. Narasimhan, and Y . Cao. React: Synergizing reasoning and acting in language models, 2023

  23. [31]

    Zhang, S

    B. Zhang, S. Li, and O. Bastani. Conformal structured prediction, 2025

  24. [32]

    alias” removes a statement y←x , replacing it with nothing, but renaming all occurences of y in the program to x; “proj

    T. Y . Zhuo, M. C. Vu, J. Chim, H. Hu, W. Yu, R. Widyasari, I. N. B. Yusuf, H. Zhan, J. He, I. Paul, S. Brunner, C. Gong, T. Hoang, A. R. Zebaze, X. Hong, W.-D. Li, J. Kaddour, M. Xu, Z. Zhang, P. Yadav, N. Jain, A. Gu, Z. Cheng, J. Liu, Q. Liu, Z. Wang, B. Hui, N. Muennighoff...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.