Pith. sign in

REVIEW 3 major objections 5 minor 15 references

Most LLM-agent work is secretly deterministic, and a closed compiler loop can turn it into verified near-free skill.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 17:39 UTC pith:WQYAAEOX

load-bearing objection A real closed-loop agent compiler with pre-registered measurements and honest failure modes; the AGI framing is marketing, the systems result is not. the 3 major comments →

arxiv 2607.04542 v1 pith:WQYAAEOX submitted 2026-07-05 cs.LG cs.AIcs.SE

Auto: The AGI Compiler

Jaber Jaber , Osama Jaber This is my paper
classification cs.LG cs.AIcs.SE
keywords LLM agentscompilationdeterminism censusWebAssemblyconformal predictionknowledge distillationagent runtimecost amortization
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM agents pay a frontier model to re-derive the same decisions over and over. This paper argues that a large share of that behavior is witnessed-deterministic—same input, same output when observed more than once—and therefore can be compiled out of interpretation. AUTO records live agent runs, measures which spans are deterministic, extracts them into verified programs or small specialists, and ships WebAssembly cognition binaries whose guarantees are measured and whose capabilities are physically confined by the sandbox. A tiered runtime runs the compiled path behind a calibrated guard; out-of-distribution inputs fall back to the reference agent, the new trace is captured, and the system recompiles, so novelty is paid for once. On a pre-registered benchmark, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic, and on a 300-item stream with three distribution shifts the closed loop cuts end-to-end cost by 6.4× while holding 96.9% parity on witnessed inputs. The same measurements show that loose guards and unfaithful fallbacks make cheap execution silently wrong, so calibration and reference fidelity decide whether the savings stay correct.

Core claim

A large fraction of real frontier-agent behavior is witnessed-deterministic and can be compiled into verified, capability-confined WebAssembly artifacts that a tiered runtime executes for near-zero marginal cost, deopting and recompiling when the world drifts. On AUTO-BENCH, 87.1% of 560 spans are witnessed-deterministic, and a closed loop on a 300-item stream with three shifts drives cost from 59 to 2 micro-dollars per item at 96.9% parity on witnessed inputs with zero errors; calibration and reference fidelity, not model capability, decide whether cheap stays correct.

What carries the argument

The ratchet: a closed record–lower–compile–gate–ship–run loop that emits cognition binaries only through differential verification against a declared behavioral contract, then executes them behind conformally calibrated guards that deopt novelty back to the reference agent for recompilation.

Load-bearing premise

The measured determinism rates and cost curves on designed, repetitive task corpora will transfer to real production traffic whose spans may be less deterministic, more generative, or shift in ways lexical guards miss.

What would settle it

Re-run the frozen AUTO-BENCH protocol unchanged on real production agent traffic: if witnessed-deterministic fractions fall far below ~87%, or the closed loop fails to amortize cost under genuine distribution shifts without silent mislabeling or gate refusals, the central claim fails.

Watch this falsifier — get emailed when new claim-graph text bears on it.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper presents AUTO, a compilation toolchain and tiered runtime for LLM agent behavior. Live agent runs are recorded via SDK shims; spans are censused for witnessed determinism; behavior is lowered to an effect-typed task-graph IR; symbolic parts are extracted by enumerative and LLM-guided CEGIS and the residue is distilled; candidates pass a differential-replay verification gate (exact, statistical, or LLM-judged) before emission as signed WebAssembly “cognition binaries” whose manifests report only measured numbers and whose declared capabilities are physically confined by the sandbox. A conformally calibrated guard routes in-distribution inputs to the compiled tier and deopts novelty to the reference agent, with captured traces recompiled (the “ratchet”). On the pre-registered AUTO-BENCH, 87.1% of 560 frontier-agent spans are witnessed-deterministic (Table 1); a 300-item stream with three scheduled shifts yields three gate-passed artifact generations, 6.4× end-to-end cost reduction, 96.9% parity on witnessed inputs, and zero errors (Figure 3), while Table 3 quantifies silent wrongness under loose calibration and gate-refused recompilation under an unfaithful deopt reference.

Significance. If the measured claims hold under the stated protocol, the work is a concrete systems contribution to agent serving: it treats verification and capability confinement as first-class products rather than prompt-level optimizations, closes a record–compile–deopt–recompile loop, and reports failure modes with equal weight to successes. Strengths that raise the bar include a pre-registered benchmark protocol, three-valued gate verdicts that refuse unmeasurable contracts, physical WASM import confinement, spend ledgers and evaluation-run identifiers attached to reported numbers, open code, and explicit quantification of silent mislabeling (48.9%) and gate refusal under unfaithful references. The narrow, testable definition of “AGI compiler” (experience → permanent verified near-free skill with measured ignorance) is useful even if the title is ambitious. The main significance risk is external validity: results are on designed corpora with high repetition, so the 87.1% and 6.4× figures are existence and amortization results on AUTO-BENCH, not yet estimates of production traffic.

major comments (3)
  1. [Abstract, §1, §6] The abstract and §1 lead with 87.1% witnessed-determinism and 6.4× end-to-end savings without the designed-corpus qualifier that §6 correctly identifies as decisive (“The corpora are designed, not production traffic… that re-run is the experiment that matters”). Given the title and the “AGI compiler” framing, this scoping is load-bearing for how readers interpret the headline numbers. Please state in the abstract and introduction that the census and ratchet curve are measured on designed AUTO-BENCH families (20–40 recorded inputs per family; 56 distinct stream texts; repetition chosen to exercise the ratchet), and that transfer to production traffic is untested.
  2. [§5.4, Table 3] H4 reports F4 as “perfectly calibrated” from five held-out in-distribution and five disjoint-vocabulary OOD probes. That probe set is too small to support strong calibration language; the more informative evidence is already in the paper—the stream-level silent-wrongness rate under loose α (Table 3, Leg A: 48.9%) and the lexical-cousin admission of shift-3 fraud phrasing (Figure 3 / §5.3). Reframe H4 around the stream exposure and treat the static probes as illustrative, or enlarge the probe sets and report confidence intervals.
  3. [§2, §5.2–§5.3] Related work correctly distinguishes AUTO from FrugalGPT, RouteLLM, GPTCache, Voyager, and distillation, but AUTO-BENCH reports no quantitative head-to-head of the full ratchet against a semantic cache and against pure distillation (same frozen 300-item stream, same parity/contract gate, same spend ledger). Without that comparison, the claim that “verification, not synthesis, is the product” and that compilation beats memoization remains partly qualitative. Adding these two baselines on the existing stream would substantially strengthen H1/H3.
minor comments (5)
  1. [Figure 1, Figure 3] Figure 1 and Figure 3 both convey cumulative/marginal spend; a single cross-reference in the Figure 1 caption to the per-window tier-1 fraction in Figure 3 (bottom) would help readers connect the flat stretches to compiled coverage.
  2. [Table 2] Table 2’s dagger note (“per full 3-call run”) is easy to miss when comparing F2 frontier cost to single-call families; consider a column that normalizes cost per decision or an explicit “unit” row.
  3. [§3, §5.2] The 17-operation DSL used by enumerative extraction is mentioned but not listed; a short appendix table of operators would make the F3 refusal (no object constructor) fully checkable without reading the repository.
  4. [§5.3] Clarify early that “parity on witnessed inputs” (96.9%) is agreement with the reference agent’s prior recordings, not agreement with stream design labels—the three-label vs six-label distinction in §5.3 is important and currently easy to miss on first read.
  5. [Abstract vs §5] Minor consistency: the abstract writes “6.4x” and “2 micro-dollars”; the body uses “6.4×” and “µ$”. Pick one style.

Circularity Check

0 steps flagged

No significant circularity: empirical systems measurements against pre-registered contracts and differential replay, not definitional or fitted predictions.

full rationale

AUTO is a systems paper whose load-bearing claims are measured quantities (witnessed-determinism census on 560 spans; cost/parity on a 300-item stream with scheduled shifts; silent-wrongness and gate-refusal failure modes), not first-principles derivations. Witnessed-determinism is an operational definition (observed ≥2 times, byte-equal, no errors) applied to recorded traces, not a quantity fitted then re-predicted. Contracts, α, K, and agreement thresholds are frozen before compile and never retuned after verdicts; the gate uses three-valued differential replay (exact/statistical/LLM-judged) against recorded reality, so acceptance is not forced by construction. There are no self-citations of prior author theorems, no uniqueness imports, no ansatz smuggled via citation, and no renaming of a known closed-form result as a new derivation. The designed-corpus limitation is scoped in §6 as external transfer risk, not internal circularity. The derivation chain (record → census → IR → extract/distill → contract gate → confined binary → conformal deopt/recompile) is self-contained and falsifiable by the released protocol and ledgers.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 3 invented entities

The central empirical claims rest on operational choices (witness rule ≥2, conformal α, recompile stride K, contract thresholds, lexical guard geometry) and domain assumptions (partial determinism of agent spans at default temperature; WASM import confinement; differential replay as ground truth). No new physical entities; invented software artifacts are engineering constructs with falsifiable runtime behavior. Free parameters are declared operating points, not hidden fits that define the headline metrics by construction.

free parameters (4)
  • conformal miscoverage α
    Abstention threshold for the guard; Table 3 shows α=0.10 vs 0.40 changes silent-wrong rate from 48.9% to lower damage and changes recompile behavior. Declared, not fitted to maximize the 6.4× claim.
  • recompile stride K
    K=8 new distinct ingested inputs triggers compile in the measured stream (Figure 3). Controls generation timing.
  • contract agreement thresholds
    Declared pre-compile thresholds such as ≥950‰ (F1), ≥900‰ (F4), 800‰ judged (F5), and exact match for some spans. Acceptance is relative to these fixed bars.
  • lexical guard geometry (trigram Jaccard / hashed-cosine)
    Choice of sketch and similarity metric sets the OOD admission rate; paper measures 2/5–3/5 OOD probe leakage and 125/253 tier-1 answers on never-witnessed inputs under leg C.
axioms (5)
  • ad hoc to paper A span observed ≥2 times with identical byte output and no errors is witnessed-deterministic; fractions are never extrapolated beyond witnessed spans.
    Defines the H2 census (§3, Table 1). Standard for the paper’s measurement, not a theorem from prior literature.
  • domain assumption Differential replay against recorded reality at pre-declared thresholds is the acceptance criterion for emission (three-valued PASS/FAIL/INCONCLUSIVE).
    Verification-as-type-system (§4). Grounds all compression and ratchet claims.
  • domain assumption Split-conformal calibration over leave-one-out guard scores controls miscoverage at declared α under exchangeability of scores.
    Invokes conformal prediction (Angelopoulos & Bates 2021) for guards (§3–§5.4). Validity assumes score exchangeability that may fail under non-stationary production drift.
  • domain assumption WebAssembly modules with zero imports (pure) or exactly one audited host function cannot exceed declared capabilities.
    Physical confinement claim (§4); depends on loader and sandbox correctness.
  • domain assumption Default-temperature frontier agent behavior on the designed task families is representative enough for the census and cost claims to be informative.
    Setup §5 and Limitations §6; single model snapshot gpt-5.4-mini.
invented entities (3)
  • cognition binary (.cbin) no independent evidence
    purpose: Signed, content-addressed WASM artifact plus measured-only manifest carrying parity and capability claims.
    Engineering artifact of the compiler back end; independent evidence is runtime behavior and gate logs, not an external physical prediction.
  • the ratchet independent evidence
    purpose: Closed loop of tier-1 serve / deopt / capture / recompile so novelty is paid once and folded into the next artifact generation.
    Named systems pattern (Algorithm 1); measured on the 300-item stream rather than postulated as a new natural law.
  • AUTO-BENCH independent evidence
    purpose: Pre-registered protocol measuring determinism census, parity-gated compression, ratchet curve, and calibrated ignorance.
    New benchmark suite introduced by the paper; value is the protocol and frozen thresholds, not a latent variable.

pith-pipeline@v1.1.0-grok45 · 14657 in / 3860 out tokens · 41067 ms · 2026-07-11T17:39:19.542157+00:00 · methodology

0 comments
read the original abstract

Every LLM agent run re-derives its behavior token by token on a frontier model: brilliant, expensive, slow, and unbounded. We present Auto, a compiler that records live agent behavior, measures which parts are secretly deterministic, extracts them into verified programs or distilled specialists, and emits cognition binaries: WebAssembly artifacts whose manifests carry measured guarantees and whose declared capabilities are physically enforced by the sandbox. A tiered runtime executes compiled behavior behind conformally calibrated guards; guard trips deopt to the reference agent, and the captured trace recompiles back down, so nothing is figured out twice. We use "AGI compiler" in one narrow, testable sense: a system that autonomously converts novel experience into permanent, verified, near-free skill while measuring what it does not know. On AUTO-BENCH, a benchmark we introduce and pre-register, 87.1% of 560 recorded frontier-agent spans are witnessed-deterministic (three of the four censused task families measure 100.0%). On a 300-item stream with three scheduled distribution shifts, the closed loop compiles three artifact generations and drives marginal cost from 59 to 2 micro-dollars per item (6.4x end-to-end) at 96.9% parity on witnessed inputs with zero errors. The same stream also quantifies the failure modes: a loose guard silently mislabels 48.9% of compiled answers, and an unfaithful deopt reference causes the verification gate to refuse recompilation. Calibration and reference fidelity, not model capability, decide whether cheap stays correct. Code: https://github.com/RightNow-AI/auto

Figures

Figures reproduced from arXiv: 2607.04542 by Jaber Jaber, Osama Jaber.

Figure 1
Figure 1. Figure 1: The central result in its simplest form: cumulative spend over the same 300 live tasks. The [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: The AUTO pipeline. Recorded behavior is measured, lowered to an effect-typed IR, extracted or distilled, and emitted only through the verification gate; a failing or unmeasurable contract never emits (red). The runtime splits execution across tiers behind a conformally calibrated guard, and the ratchet (green) recompiles every captured deopt, so nothing is figured out twice. Older roots. The ratchet is a m… view at source ↗
Figure 3
Figure 3. Figure 3: The ratchet curve, measured on a 300-item stream with three scheduled distribution shifts [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

15 extracted references · 15 linked inside Pith

  1. [1]

    Angelopoulos and Stephen Bates

    Anastasios N. Angelopoulos and Stephen Bates. A gentle introduction to conformal prediction and distribution-free uncertainty quantification.arXiv:2107.07511,

    Pith/arXiv arXiv

  2. [2]

    Adnan Darwiche and Pierre Marquis

    arXiv:2305.05176. Adnan Darwiche and Pierre Marquis. A knowledge compilation map.Journal of Artificial Intelligence Research, 17:229–264,

    Pith/arXiv arXiv

  3. [3]

    Language model cascades.arXiv:2207.10342,

    David Dohan et al. Language model cascades.arXiv:2207.10342,

    Pith/arXiv arXiv

  4. [4]

    Distilling the knowledge in a neural network

    Geoffrey Hinton, Oriol Vinyals, and Jeff Dean. Distilling the knowledge in a neural network. arXiv:1503.02531,

    Pith/arXiv arXiv

  5. [5]

    Carlos E

    arXiv:2305.02301. Carlos E. Jimenez et al. SWE-bench: Can language models resolve real-world GitHub issues? In International Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  6. [6]

    Omar Khattab et al

    arXiv:2310.06770. Omar Khattab et al. DSPy: Compiling declarative language model calls into self-improving pipelines. arXiv:2310.03714,

    Pith/arXiv arXiv

  7. [7]

    arXiv:2312.04511. John E. Laird, Allen Newell, and Paul S. Rosenbloom. SOAR: An architecture for general intelligence. Artificial Intelligence, 33(1):1–64,

    Pith/arXiv arXiv

  8. [8]

    Xiao Liu et al

    arXiv:2211.17192. Xiao Liu et al. AgentBench: Evaluating LLMs as agents. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  9. [9]

    arXiv:2308.03688. Tom M. Mitchell, Richard M. Keller, and Smadar T. Kedar-Cabelli. Explanation-based generalization: A unifying view.Machine Learning, 1(1):47–80,

    Pith/arXiv arXiv

  10. [10]

    RouteLLM: Learning to route LLMs with preference data.arXiv:2406.18665,

    Isaac Ong et al. RouteLLM: Learning to route LLMs with preference data.arXiv:2406.18665,

    Pith/arXiv arXiv

  11. [11]

    MemGPT: Towards LLMs as operating systems.arXiv:2310.08560,

    Charles Packer et al. MemGPT: Towards LLMs as operating systems.arXiv:2310.08560,

    Pith/arXiv arXiv

  12. [12]

    9 Timo Schick et al

    arXiv:2305.15334. 9 Timo Schick et al. Toolformer: Language models can teach themselves to use tools. InAdvances in Neural Information Processing Systems (NeurIPS),

    Pith/arXiv arXiv

  13. [13]

    Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat

    arXiv:2302.04761. Armando Solar-Lezama, Liviu Tancau, Rastislav Bodik, Sanjit Seshia, and Vijay Saraswat. Com- binatorial sketching for finite programs. InProceedings of the 12th International Conference on Architectural Support for Programming Languages and Operating Systems (ASPLOS),

    Pith/arXiv arXiv

  14. [14]

    Shunyu Yao et al

    arXiv:2305.16291. Shunyu Yao et al. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations (ICLR),

    Pith/arXiv arXiv

  15. [15]

    arXiv:2210.03629. 10

    Pith/arXiv arXiv