{"id":"6a495f73-343c-4a92-ac7f-c961e795ae01","arxiv_id":"2607.04542","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"AUTO compiles witnessed-deterministic LLM-agent spans into verified WASM cognition binaries and recompiles on deopt, cutting cost 6.4× at 96.9% parity on a 300-item shifted stream.","lead":"AUTO records live LLM-agent runs, measures which spans are deterministic, and compiles them into verified WebAssembly binaries that run for near-zero marginal cost, recompiling when the world shifts. If the approach holds on real traffic, agent operators could cut interpretation spend by large factors while keeping answers contract-checked.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection beyond the reader's already-identified production-transfer assumption; the designed-corpus caveat is correctly scoped and does not undermine the measured claims.","rationale":"The strongest claim is carefully scoped to AUTO-BENCH numbers and the designed stream; the paper never asserts that 87.1% or 6.4× hold for arbitrary production agents. The reader correctly flags the designed-corpus limitation as the weakest assumption and already assigns a CONDITIONAL verdict with medium correctness risk. My second-pass read finds no deeper load-bearing flaw (e.g., the CEGIS checker catching the constant-output cheat, physical WASM import confinement, and three-valued INCONCLUSIVE blocking emission are all sound within the stated design). Therefore the verdict remains CONDITIONAL; no adjustment is warranted. The concrete test simply operationalizes the experiment the authors themselves declare matters.","tokens_in":10666,"tokens_out":511,"duration_ms":5078,"concrete_test":"Re-execute the frozen AUTO-BENCH protocol (H1–H4, same contracts, witness rule, and spend ledger) on a production trace store of ≥500 distinct real agent spans drawn from live ticket/routing traffic; if the witnessed-deterministic fraction falls below ~60% or the closed-loop end-to-end cost reduction collapses below 2× under the same three-shift schedule, the production-transfer claim fails and the CONDITIONAL verdict should tighten.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reader's weakest_assumption already isolates the single load-bearing condition: that the 87.1% witnessed-determinism census (Table 1, 560 spans) and the 6.4\times ratchet curve (Figure 3, 300-item stream with three scheduled shifts) transfer beyond the designed corpora (20–40 inputs per family, 56 distinct ticket texts, repetition rates chosen to exercise the ratchet). The paper itself states this in §6 and treats the real-traffic re-run as the decisive experiment. Inside the pre-registered protocol the measurements are internally consistent: three-valued gate verdicts, differential replay, conformal α, and explicit failure modes (48.9% silent wrongness under loose calibration; gate-refused recompilation under unfaithful reference) are reported with equal weight. No additional internal inconsistency, hidden assumption in the IR/extraction/gate, or unacknowledged circularity appears that would further weaken the strongest claim as stated.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper presents AUTO, a compilation toolchain and tiered runtime for LLM agent behavior. Live agent runs are recorded via SDK shims; spans are censused for witnessed determinism; behavior is lowered to an effect-typed task-graph IR; symbolic parts are extracted by enumerative and LLM-guided CEGIS and the residue is distilled; candidates pass a differential-replay verification gate (exact, statistical, or LLM-judged) before emission as signed WebAssembly “cognition binaries” whose manifests report only measured numbers and whose declared capabilities are physically confined by the sandbox. A conformally calibrated guard routes in-distribution inputs to the compiled tier and deopts novelty to the reference agent, with captured traces recompiled (the “ratchet”). On the pre-registered AUTO-BENCH, 87.1% of 560 frontier-agent spans are witnessed-deterministic (Table 1); a 300-item stream with three scheduled shifts yields three gate-passed artifact generations, 6.4× end-to-end cost reduction, 96.9% parity on witnessed inputs, and zero errors (Figure 3), while Table 3 quantifies silent wrongness under loose calibration and gate-refused recompilation under an unfaithful deopt reference.","tokens_in":10913,"tokens_out":1428,"duration_ms":29299,"significance":"If the measured claims hold under the stated protocol, the work is a concrete systems contribution to agent serving: it treats verification and capability confinement as first-class products rather than prompt-level optimizations, closes a record–compile–deopt–recompile loop, and reports failure modes with equal weight to successes. Strengths that raise the bar include a pre-registered benchmark protocol, three-valued gate verdicts that refuse unmeasurable contracts, physical WASM import confinement, spend ledgers and evaluation-run identifiers attached to reported numbers, open code, and explicit quantification of silent mislabeling (48.9%) and gate refusal under unfaithful references. The narrow, testable definition of “AGI compiler” (experience → permanent verified near-free skill with measured ignorance) is useful even if the title is ambitious. The main significance risk is external validity: results are on designed corpora with high repetition, so the 87.1% and 6.4× figures are existence and amortization results on AUTO-BENCH, not yet estimates of production traffic.","major_comments":[{"comment":"The abstract and §1 lead with 87.1% witnessed-determinism and 6.4× end-to-end savings without the designed-corpus qualifier that §6 correctly identifies as decisive (“The corpora are designed, not production traffic… that re-run is the experiment that matters”). Given the title and the “AGI compiler” framing, this scoping is load-bearing for how readers interpret the headline numbers. Please state in the abstract and introduction that the census and ratchet curve are measured on designed AUTO-BENCH families (20–40 recorded inputs per family; 56 distinct stream texts; repetition chosen to exercise the ratchet), and that transfer to production traffic is untested.","section":"Abstract, §1, §6"},{"comment":"H4 reports F4 as “perfectly calibrated” from five held-out in-distribution and five disjoint-vocabulary OOD probes. That probe set is too small to support strong calibration language; the more informative evidence is already in the paper—the stream-level silent-wrongness rate under loose α (Table 3, Leg A: 48.9%) and the lexical-cousin admission of shift-3 fraud phrasing (Figure 3 / §5.3). Reframe H4 around the stream exposure and treat the static probes as illustrative, or enlarge the probe sets and report confidence intervals.","section":"§5.4, Table 3"},{"comment":"Related work correctly distinguishes AUTO from FrugalGPT, RouteLLM, GPTCache, Voyager, and distillation, but AUTO-BENCH reports no quantitative head-to-head of the full ratchet against a semantic cache and against pure distillation (same frozen 300-item stream, same parity/contract gate, same spend ledger). Without that comparison, the claim that “verification, not synthesis, is the product” and that compilation beats memoization remains partly qualitative. Adding these two baselines on the existing stream would substantially strengthen H1/H3.","section":"§2, §5.2–§5.3"}],"minor_comments":[{"comment":"Figure 1 and Figure 3 both convey cumulative/marginal spend; a single cross-reference in the Figure 1 caption to the per-window tier-1 fraction in Figure 3 (bottom) would help readers connect the flat stretches to compiled coverage.","section":"Figure 1, Figure 3"},{"comment":"Table 2’s dagger note (“per full 3-call run”) is easy to miss when comparing F2 frontier cost to single-call families; consider a column that normalizes cost per decision or an explicit “unit” row.","section":"Table 2"},{"comment":"The 17-operation DSL used by enumerative extraction is mentioned but not listed; a short appendix table of operators would make the F3 refusal (no object constructor) fully checkable without reading the repository.","section":"§3, §5.2"},{"comment":"Clarify early that “parity on witnessed inputs” (96.9%) is agreement with the reference agent’s prior recordings, not agreement with stream design labels—the three-label vs six-label distinction in §5.3 is important and currently easy to miss on first read.","section":"§5.3"},{"comment":"Minor consistency: the abstract writes “6.4x” and “2 micro-dollars”; the body uses “6.4×” and “µ$”. Pick one style.","section":"Abstract vs §5"}],"recommendation":"minor_revision","confidential_remarks":"Technically this is a careful agent-serving systems paper with unusually honest failure-mode reporting and reproducible artifacts. The “AGI compiler” title and framing will polarize some cs.LG referees; the narrow definition in the abstract is defensible, but venue fit depends on appetite for systems+agents work versus pure learning theory. I do not see internal inconsistency or circular evaluation; the designed-corpus caveat is the main external-validity issue and is already acknowledged in §6. Code and spend ledgers are real strengths."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The thing worth knowing: this is a complete record-to-WASM stack for agent behavior, with a witnessed-determinism census, a contract gate that actually refuses, and a live recompile loop that cuts cost on a shifting stream while measuring when cheap goes wrong. That closed loop is the contribution; the pieces (CEGIS, distillation, conformal guards, Voyager-style skills, caches) already exist separately.\n\nWhat they did well is the measurement discipline. Table 1’s 87.1% on 560 spans is a real census, not a claim about all agents. Table 2 reports refusals with the same weight as passes (F2 priority exact-contract fail, F3 JSON extraction refused at every rung, F5 judged fail). Figure 3 and Table 3 show the ratchet working and then show the two failure modes that matter: loose α → 48.9% silent wrongness, unfaithful deopt reference → gate-refused recompiles. Calibration and reference fidelity, not model size, decide whether cheap stays correct. That is the right lesson, and they put it in the abstract. Code and spend ledgers are shipped; the protocol was frozen before runs.\n\nSoft spots are real but scoped. Corpora are designed (20–40 inputs per family, 56 distinct texts, repetition chosen to exercise the ratchet). Guards are lexical, so cousins get in and paraphrases trip. Generative residue does not compile at the scale they tried. The “AGI compiler” title is a narrow definition they state up front; treat it as branding, not a claim about general intelligence. None of that undoes the measured stack inside the protocol.\n\nThis is for people building production agents who care about cost, confinement, and not re-deriving the same routing decision forever. It is not a theory paper. I would bring it to reading group, cite the census and the failure-mode table, and send it to peer review. Real-traffic re-run is the next experiment, not a reason to desk-reject.","headline":"A real closed-loop agent compiler with pre-registered measurements and honest failure modes; the AGI framing is marketing, the systems result is not.","tokens_in":11564,"tokens_out":511,"would_cite":true,"duration_ms":5110,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Most LLM-agent work is secretly deterministic, and a closed compiler loop can turn it into verified near-free skill.","keywords":["LLM agents","compilation","determinism census","WebAssembly","conformal prediction","knowledge distillation","agent runtime","cost amortization"],"falsifier":"Re-run the frozen AUTO-BENCH protocol unchanged on real production agent traffic: if witnessed-deterministic fractions fall far below ~87%, or the closed loop fails to amortize cost under genuine distribution shifts without silent mislabeling or gate refusals, the central claim fails.","tokens_in":11495,"feed_emoji":"⚙️","tokens_out":713,"duration_ms":5382,"temperature":0.7,"pith_summary":"LLM agents pay a frontier model to re-derive the same decisions over and over. This paper argues that a large share of that behavior is witnessed-deterministic—same input, same output when observed more than once—and therefore can be compiled out of interpretation. AUTO records live agent runs, measures which spans are deterministic, extracts them into verified programs or small specialists, and ships WebAssembly cognition binaries whose guarantees are measured and whose capabilities are physically confined by the sandbox. A tiered runtime runs the compiled path behind a calibrated guard; out-of-distribution inputs fall back to the reference agent, the new trace is captured, and the system recompiles, so novelty is paid for once. On a pre-registered benchmark, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic, and on a 300-item stream with three distribution shifts the closed loop cuts end-to-end cost by 6.4× while holding 96.9% parity on witnessed inputs. The same measurements show that loose guards and unfaithful fallbacks make cheap execution silently wrong, so calibration and reference fidelity decide whether the savings stay correct.","feed_headline":"87% of agent work is deterministic—compile it once","feed_subtitle":"A closed loop cuts LLM-agent cost 6.4× while measuring what it still does not know","key_machinery":"The ratchet: a closed record–lower–compile–gate–ship–run loop that emits cognition binaries only through differential verification against a declared behavioral contract, then executes them behind conformally calibrated guards that deopt novelty back to the reference agent for recompilation.","core_discovery":"A large fraction of real frontier-agent behavior is witnessed-deterministic and can be compiled into verified, capability-confined WebAssembly artifacts that a tiered runtime executes for near-zero marginal cost, deopting and recompiling when the world drifts. On AUTO-BENCH, 87.1% of 560 spans are witnessed-deterministic, and a closed loop on a 300-item stream with three shifts drives cost from 59 to 2 micro-dollars per item at 96.9% parity on witnessed inputs with zero errors; calibration and reference fidelity, not model capability, decide whether cheap stays correct.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["87% of agent spans are witnessed-deterministic—compile them","Auto extracts verified WASM from live agent traces once","Closed loop drops agent cost 6.4× to 2 µ$/item at 96.9%","Compile deterministic agent work; deopt and recompile on drift","Calibration decides if cheap compiled agents stay correct"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The measured determinism rates and cost curves on designed, repetitive task corpora will transfer to real production traffic whose spans may be less deterministic, more generative, or shift in ways lexical guards miss.","fun_headline_variants_meta":{"raw":{"variants":["87% of agent spans are witnessed-deterministic—compile them","Auto extracts verified WASM from live agent traces once","Closed loop drops agent cost 6.4× to 2 µ$/item at 96.9%","Compile deterministic agent work; deopt and recompile on drift","Calibration decides if cheap compiled agents stay correct"]},"model":"grok-4.5","effort":"low","cost_usd":0.004754,"raw_usage":{"total_tokens":1461,"prompt_tokens":907,"num_sources_used":0,"completion_tokens":75,"cost_in_usd_ticks":47540000,"prompt_tokens_details":{"text_tokens":907,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":479,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":907,"tokens_out":75,"duration_ms":4114,"temperature":1.0,"reasoning_tokens":479,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-11T17:39:19.542157+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Re-run the frozen AUTO-BENCH protocol unchanged on real production agent traffic: if witnessed-deterministic fractions fall far below ~87%, or the closed loop fails to amortize cost under genuine distribution shifts without silent mislabeling or gate refusals, the central claim fails.","supporting_citations":[],"review_version":1}