{"id":"beb4ffb3-4d33-4a3f-b888-39b5c5e70813","arxiv_id":"2506.12202","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Quasar transpiles LLM-generated Python into a functional language that parallelizes external calls, batches security approvals, and supports conformal prediction, with mixed evidence on task performance parity.","lead":"The paper presents Quasar, a programming language designed for LLM agents that write code actions, together with a transpiler from a subset of Python. On a visual question answering benchmark, Quasar claims to keep task accuracy comparable to Python while cutting execution time by 42% and user approvals by 52% on improvable tasks, and adding conformal uncertainty guarantees.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Purity of external calls is assumed but never enforced; without it the confluence, speedup, security, and conformal claims are unsupported.","rationale":"The reader's weakest_assumption identifies the purity of external calls as the load-bearing premise, and I agree this is the single most critical point. The paper's own semantics and all three claimed benefits—performance, security, and reliability—depend on the confluence property that follows from purity. Yet the language provides no mechanism to ensure that an external function is pure; external functions are arbitrary Python and may print, mutate state, or depend on randomness. The paper even acknowledges that LLM calls are external functions, and LLM calls are stochastic. Thus, when Quasar dispatches external calls in parallel (Section 3.3), the actual behavior of a program can differ from sequential Python execution, and the guarantees of Section 3.1 fail. This is not a merely theoretical edge case; it directly affects the benchmark domain, since ViperGPT's detectors and LLMs are not deterministic pure functions. The replay-based speedup measurement in Section 4.2 is therefore an upper bound that assumes determinism and independence; it does not measure real parallel execution of the live system. The security batching of Section 4.3 similarly assumes that a batch of concurrently executed calls has the same effects as if approved and executed sequentially, which is only true for pure calls. The conformal guarantee of Section 4.4 relies on abstracting each external call as a function of its inputs; stateful calls break the composition argument. A secondary concern is the 9 percentage point drop in successful execution (Table 1), which the reader also noted, but that is an implementation/prompting issue that could be improved; the purity problem is foundational. I therefore keep the verdict as CONDITIONAL (UNCHANGED): the authors should either enforce the purity assumption (e.g., via a type system or sandbox) or demonstrate experimentally that their target external calls are pure, and they should provide the missing proof of confluence under the stated assumption.","tokens_in":13468,"tokens_out":7762,"duration_ms":96973,"concrete_test":"Add a stateful external function `increment_and_read()` that increments a shared counter and returns its new value. Run a Quasar program that makes two parallel calls to this function (or one call to it and one dependent call) via the eager dispatcher, and run the equivalent sequential Python program. If the final answers or counter values differ, the Section 3.1 confluence guarantee is violated; this would directly falsify the 'same set of external calls, same effects' claim and invalidate the parallel speedup and batching security arguments for stateful tools.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central soundness property stated in Section 3.1—'Assuming each external call only depends on its inputs, then it can be shown that any sequence of rule applications results in the same set of external calls, and therefore the same effects'—is the load-bearing foundation for parallel dispatch (Section 3.3), batched user approval (Section 3.2), and conformal semantics (Section 3.4). However, the paper never enforces this assumption: external functions are implemented in arbitrary Python (Section 3.1 explicitly allows 'printing a value or calling an LLM'), so they may have side effects or hidden state. No type system, static check, or runtime guard verifies purity. The claimed theorem is also not proven in the text. In the target ViperGPT domain, the external calls are stochastic neural models; running them concurrently can change outputs through internal randomness, shared GPU state, or rate-limited APIs, so the 'same effects' guarantee does not hold for actual executions. Consequently, the 42% speedup measured in Section 4.2 by replaying recorded results on both Python and QUASAR only demonstrates scheduling benefits under a purity assumption; it does not show real wall-clock speedup when calls are impure. Similarly, the security and conformal guarantees are conditional on a property the system does not ensure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Quasar, a functional programming language for LLM agents that execute code actions, in which side effects are isolated in external calls. LLMs generate a restricted subset of Python, which is transpiled to Quasar and executed by an interpreter that parallelizes external calls, batches user approval requests, and supports set-valued conformal execution. The evaluation on the ViperGPT visual question answering agent over GQA claims task accuracy comparable to Python, a 42% reduction in execution time on improvable tasks, a 52% reduction in user-approval interactions on improvable tasks, and a conformal prediction guarantee targeting a 0.1 error rate. The central soundness property is stated in Section 3.1 as a confluence result conditional on the assumption that each external call depends only on its inputs.","tokens_in":13663,"tokens_out":6539,"duration_ms":67539,"significance":"The paper proposes a promising direction for LLM agent code actions by separating pure internal computation from effectful external calls, transpiling a Python subset into a functional core, batching user approvals, and extending execution to set-valued conformal semantics. A clear strength is the generation strategy evaluation: transpilation produces 6.9x and 7.6x fewer erroneous programs than LLM-based translation and direct Quasar generation, respectively, and the implementation is publicly available. However, the headline performance, security, and reliability claims are conditional on an unenforced purity assumption, and the accuracy comparison is weakened by conditioning on successful execution. If the purity assumption can be enforced or the experiments redone with real concurrent execution, and if end-to-end accuracy is reported, the contributions would be significant for the programming-languages and LLM-agent communities.","major_comments":[{"comment":"The load-bearing 'same effects' property is stated as an assumption ('Assuming each external call only depends on its inputs'), but no proof is given and no enforcement mechanism is described. External functions are implemented in arbitrary Python (Section 3.1 explicitly allows effects such as printing or calling an LLM), and in the ViperGPT domain the external calls are stochastic neural models whose concurrent execution can change outputs through random sampling, shared GPU state, or rate-limited APIs. The parallel dispatch of Section 3.3, the batched-approval security argument of Section 3.2, and the conformal semantics of Section 3.4 all rely on this property. The paper should either prove the confluence result under the stated assumption, add an effect system or runtime check that verifies purity, or empirically test whether the assumption holds for the target external calls.","section":"§3.1 (also §3.2–§3.4, §4.2)"},{"comment":"The claim that 'the accuracy of QUASAR programs is comparable to that of Python programs' is not supported by the table as reported. Successful execution drops from 99.7% for Python to 90.6% for the transpiled approach, and VQA Accuracy is computed only over successful programs. The end-to-end accuracy is therefore approximately 70.4% for Python versus 64.0% for the transpiled approach (0.997 × 0.706 vs. 0.906 × 0.706), a relative drop of about 9%. The paper should report end-to-end accuracy and discuss this trade-off explicitly.","section":"§4.1, Table 1"},{"comment":"The performance experiment replays recorded external-call results and running times on both Python and QUASAR. This measures scheduling speedup under the purity assumption, not end-to-end wall-clock time with real concurrent execution; it does not account for resource contention, shared GPU state, or rate-limited APIs, and it assumes that executing calls concurrently does not change their outputs. The headline '42% reduction' should be qualified as a simulated scheduling improvement on improvable tasks, and an experiment with real parallel execution would be needed to support a running-time claim.","section":"§4.2"},{"comment":"The conformal evaluation selects thresholds on validation splits (the largest threshold with validation error below 0.1) and then reports test error. This is a model-selection procedure, not the conformal rescaling with a held-out calibration set described in Section 3.4, and it does not provide a distribution-free coverage guarantee. The reported 61.4% uncertainty rate also indicates that the prediction sets are large, so the practical utility of the reliability improvement is unclear. Please clarify the protocol and report the end-to-end accuracy achieved with set-valued outputs.","section":"§4.4"},{"comment":"The security evaluation measures a reduction in the number of user-approval interactions, which is a usability improvement, not a security guarantee. The access-control argument assumes that each external call has known, controllable effects; however, external functions are implemented as arbitrary Python (e.g., they may print or call an LLM) and are not sandboxed or statically checked. If an approved external function has hidden effects, batching approvals does not make execution more secure. The paper should either restrict external functions to a verified interface or state this limitation explicitly.","section":"§3.2"}],"minor_comments":[{"comment":"The text says 'if bi is False the element ci may or may not be in the list, whereas if bi is False, then ci is definitely in the list'; the second occurrence of 'False' should be 'True'.","section":"Appendix C, fold-abs rule"},{"comment":"The text refers to a 'target coverage rate of 0.1' and then reports 'empirical error'; the wording should distinguish coverage probability from error rate.","section":"§4.4, Figure 4d"},{"comment":"There are typographical errors: 'path2.simple_query' should be 'patch2.simple_query' in §3.2, and 'rools' in Figure 2 should be 'rules'.","section":"Figure 2 and §3.2"},{"comment":"Adding a row for end-to-end accuracy (successful execution × VQA accuracy) would make the comparison transparent and would prevent misinterpretation of 'comparable' accuracy.","section":"Table 1"},{"comment":"The text says external calls are executed 'as soon as all of their arguments are available,' but Algorithm 1 first queries the user before dispatching; the relationship between eager dispatch and user approval should be clarified.","section":"§3.1 and Algorithm 1"},{"comment":"Reference [8] is formatted inconsistently as 'e. a. Jason Wei'; the author list should be formatted in the same style as the other references.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about the unenforced purity assumption is legitimate and is the main barrier to accepting the paper's performance and reliability claims. The authors should be asked to either enforce the assumption or clearly frame the evaluation as a simulation. The paper is within the journal's scope, and the self-citations to earlier work are appropriate building blocks rather than a circularity concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about this paper because it is a serious attempt to give LLM agents a domain-specific language instead of just running generated Python. The new thing is the combination: Quasar bundles automatic parallel dispatch, batched user approval, and conformal set-based execution into one interpreter, and it includes a transpiler from a practical subset of Python so LLMs do not have to learn a new syntax. That integration is genuinely useful and the system is real.\n\nWhat it does well: the evaluation on ViperGPT/GQA is a real benchmark with 1000 tasks, and they compare against two sensible baselines (LLM-translated Quasar and direct Quasar generation), showing the transpiler is far more reliable. The conformal calibration is standard and the coverage results (9.1% mean error against a 10% target) look correct. The paper also explicitly states its central assumption—external calls depend only on their inputs—which is more than many papers do.\n\nThe soft spots are real. Most importantly, Table 1 reports \"VQA Accuracy\" only on programs that execute successfully. Transpiled programs succeed 90.6% of the time versus 99.7% for Python; the conditional accuracy is 70.6% in both cases, so end-to-end accuracy is roughly 64% versus 70%. Calling that \"comparable\" is misleading and undercuts the headline claim. Second, the confluence property that justifies parallel execution and batched approval is asserted (\"it can be shown\") but not proved, and nothing in the type system or runtime enforces purity. The stress-test note is right that this is load-bearing: if an external call writes state or reads the clock, reordering changes the set and ordering of effects. The replay-based performance evaluation measures scheduling gains under fixed call times; that is a legitimate controlled experiment, but it does not demonstrate real wall-clock speedup when calls are impure and contend for resources. One caveat to the stress-test: the conformal guarantees do not actually depend on purity, because they go through abstract versions that overapproximate outputs; the purity assumption matters for performance and security, not for the coverage result.\n\nBottom line: this is a solid systems paper with a genuine contribution, but the accuracy reporting needs fixing and the formal foundations need tightening. It is not a desk reject. I would send it to review, with a request to report end-to-end accuracy and to either prove the confluence claim under explicit conditions or weaken it accordingly.","headline":"A useful integration of parallel dispatch, batched approval, and conformal semantics for LLM agent code, but the headline accuracy claim is weakened by a 6-point end-to-end drop and the purity assumption is asserted, not enforced.","tokens_in":14255,"tokens_out":4778,"would_cite":true,"duration_ms":56433,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Transpiling agent Python into Quasar preserves accuracy while cutting runtime 42%, approvals 52%, and adding conformal uncertainty.","keywords":["LLM agents","code actions","transpilation","parallel execution","conformal prediction","access control","visual question answering","programming language design"],"falsifier":"Run an equivalent Python and Quasar program whose external calls include a function that appends to a list or file; if the transcript of external calls (order, arguments, or repetition) differs between the two executions, the purity assumption behind the semantics is violated.","tokens_in":13227,"feed_emoji":"⚡","tokens_out":6384,"duration_ms":142107,"temperature":0.7,"pith_summary":"The paper introduces Quasar, a programming language for LLM agents that write code to call external tools. The key move is to transpile a subset of Python, the language LLMs actually generate, into Quasar, whose pure functional core isolates all side effects in external calls. Because internal rewrites are effect-free, the interpreter can run external calls in parallel, collect them for batch user approval, and execute programs over sets of values instead of single values. On the ViperGPT visual question answering agent applied to GQA, the authors report that Quasar programs retain Python-level accuracy while reducing execution time by 42% and approval interactions by 52% on tasks where those reductions are possible, and that conformal semantics hit a target 10% error rate.","feed_headline":"Agent code in Quasar runs 42% faster and needs 52% fewer approvals","feed_subtitle":"Same Python agents, same accuracy; Quasar adds parallelism, batched approvals, and conformal uncertainty.","key_machinery":"The machinery is a small functional core language with rewrite rules: a program is a sequence of single-assignment statements, external calls are dispatched as soon as all their arguments are values, and internal rules (alias, projection, if-true/false, fold) rewrite the program until a terminal result. Side effects live only in external calls, which are invoked through a single rule that spawns a thread and later substitutes the returned value. The interpreter's strategy is to run internal rewrites as far as possible, collect all dispatchable external calls, ask the user once to approve the batch, then launch them in parallel; the same structure supports conformal semantics by adding abstract values (sets, maybe-lists, joins) and matching rewrite rules, with externally supplied conformal abstractions for model calls. The transpiler from a Python subset (function calls, assignments, if/for/while, no early returns) converts imperative control flow into functional folds and expression-level conditionals.","core_discovery":"The central claim is that an LLM agent can get the reliability and security benefits of a purpose-built language without giving up the fluency it already has in Python: generate Python in a restricted subset, transpile it to Quasar, and execute under Quasar's rewrite semantics. Quasar's semantics split computation into internal rules, which are pure and can be applied in any order, and one external rule that dispatches calls to user-approved functions. This makes all external calls first-class, so the interpreter can batch approvals and dispatch them in parallel, and it lets conformal evaluations propagate sets of possible values through the program. The paper reports 90.6% successful execution for transpiled programs versus 99.7% for raw Python, with identical 70.6% VQA accuracy on successful programs; on the roughly 41% of tasks with exploitable parallelism, execution time drops 42% ± 22; on the 42% of tasks with batchable approvals, interaction count drops 52% ± 19; and the conformal run hits 9.1% ± 1.9 test error against a 10% target.","pith_inferences":["Inference: the same rewrite semantics would work for any source language that can be lowered to the functional core, so the Python subset is a convenience for LLM fluency rather than a fundamental constraint.","Inference: if external calls violate purity (e.g., a logging API or random sampler), the 'same set of external calls' guarantee breaks; a natural extension is to make side-effecting calls explicit state tokens threaded through the dependency graph, restoring deterministic reordering.","Inference: the conformal semantics could be used to drive interactive clarification: when a prediction set spans multiple plausible answers, the agent could ask the user one targeted question instead of committing to a single label.","Inference: the reported gains are measured on tasks already parallelizable or batchable; expected end-to-end speedup in a new domain depends on the fraction of such tasks, not just on the 42% and 52% headline numbers."],"forward_implications":["LLM agents that already write Python can adopt Quasar without learning a new language or changing prompts; the transpiler does the conversion.","Tasks with independent tool calls should see wall-clock time roughly halved, and approval round-trips collapse to one interaction per batch.","Conformal execution turns every model-backed call into a set-valued result, so an agent can report 'I am not sure' instead of a single hallucinated answer, with a user-set target coverage.","The security surface is exactly the set of external calls: internal pure computation cannot leak or mutate state, so validation can focus on the call list.","Transpiled generation makes far fewer translation errors than LLM-generated Quasar (6.9x to 7.6x fewer), so the bottleneck shifts to the LLM adhering to the Python subset."],"supporting_citations":[{"why":"ViperGPT is the visual question answering agent whose generated Python programs are transpiled to Quasar and used as the evaluation testbed.","marker":"[23]"},{"why":"GQA provides the 1000 visual reasoning tasks on which the experiments measure accuracy, runtime, approvals, and conformal coverage.","marker":"[7]"},{"why":"Supplies the opportunistic parallel evaluation strategy for pure functional programs that Quasar adapts to dispatch external calls as soon as arguments are available.","marker":"[16]"},{"why":"Provides the compositional conformal prediction approach for neurosymbolic programs that Quasar extends to set-valued execution over whole agent programs.","marker":"[18]"},{"why":"Foundational conformal prediction guarantee that justifies the target coverage level used in the reliability evaluation.","marker":"[25]"},{"why":"Learn-then-test calibration used to choose the conformal threshold that meets the target error rate on the validation split.","marker":"[2]"},{"why":"Conformal structured prediction used for jointly rescaling thresholds across individual models during calibration.","marker":"[31]"}],"fun_headline_variants":["Python agents, Quasar speed: 42% faster, 52% fewer approvals","LLM agents code in Python, run Quasar: 42% speedup, 52% fewer prompts","Quasar transpiles agent Python for 42% faster, 52% safer actions","Agent code: Python fluency, Quasar gains—42% less time, 52% fewer approvals","Write Python, run Quasar: LLM agents get 42% speed, 52% approval cut"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every external call in an agent program is a pure function of its inputs, so reordering and parallelizing calls cannot change the program's effects.","fun_headline_variants_meta":{"raw":{"variants":["Python agents, Quasar speed: 42% faster, 52% fewer approvals","LLM agents code in Python, run Quasar: 42% speedup, 52% fewer prompts","Quasar transpiles agent Python for 42% faster, 52% safer actions","Agent code: Python fluency, Quasar gains—42% less time, 52% fewer approvals","Write Python, run Quasar: LLM agents get 42% speed, 52% approval cut"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000233,"raw_usage":{"total_tokens":1522,"prompt_tokens":1004,"completion_tokens":518,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":395}},"tokens_in":620,"tokens_out":518,"duration_ms":6716,"temperature":1.0,"reasoning_tokens":395,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:55:45.618796+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run an equivalent Python and Quasar program whose external calls include a function that appends to a list or file; if the transcript of external calls (order, arguments, or repetition) differs between the two executions, the purity assumption behind the semantics is violated.","supporting_citations":[{"cited_title":"Surís, S","cited_arxiv_id":null,"evidence_quote":"ViperGPT is the visual question answering agent whose generated Python programs are transpiled to Quasar and used as the evaluation testbed."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GQA provides the 1000 visual reasoning tasks on which the experiments measure accuracy, runtime, approvals, and conformal coverage."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the opportunistic parallel evaluation strategy for pure functional programs that Quasar adapts to dispatch external calls as soon as arguments are available."},{"cited_title":"Ramalingam, S","cited_arxiv_id":null,"evidence_quote":"Provides the compositional conformal prediction approach for neurosymbolic programs that Quasar extends to set-valued execution over whole agent programs."},{"cited_title":"V ovk, A","cited_arxiv_id":null,"evidence_quote":"Foundational conformal prediction guarantee that justifies the target coverage level used in the reliability evaluation."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Learn-then-test calibration used to choose the conformal threshold that meets the target error rate on the validation split."},{"cited_title":"Zhang, S","cited_arxiv_id":null,"evidence_quote":"Conformal structured prediction used for jointly rescaling thresholds across individual models during calibration."}],"review_version":1}