Pith. sign in

REVIEW 4 major objections 4 minor 36 references

Proof-or-Stop Lifecycle Control claims that an autonomous coding loop should advance a task from plan to done only when fresh, source-bound, mechanically verifiable evidence backs each lifecycle claim, and that enforcing this rule sharply c

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 00:46 UTC pith:VEA6UPTO

load-bearing objection Solid formalization, honest evaluation, but the 'model-agnostic' claim exceeds a same-family, single-task-concentrated ablation. the 4 major comments →

arxiv 2607.14890 v1 pith:VEA6UPTO submitted 2026-07-16 cs.AI cs.SE

Proof-or-Stop: Don't Trust the Agent, Trust the Evidence -- Loop Engineering for Verifiable Evidence-Gated Lifecycle Control

Jek Huang , Jeffery Hsia , Jiayi Sun , Freddie Shi , Wei Huang , Ian H. White This is my paper
classification cs.AI cs.SE
keywords autonomous agentslifecycle controlevidence gatingclaim admissibilityagent-as-claimprovenancesoftware engineeringreproducibility
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper is trying to establish a control principle for autonomous coding agents: treat everything an agent says as a claim, not as lifecycle state, and let a software gate advance the work only when structured evidence — fresh, bound to the exact tracked source state, authenticated, and produced by an authorized actor — supports that claim. It argues that current pipelines, durable-execution systems, and review protocols route or persist claims but do not decide whether a claim is admissible against the code about to be merged. The load-bearing statement is a formal equivalence: a lifecycle transition is permitted if and only if every required claim has admissible evidence. If true, autonomous software lifecycles could be made to act on evidence rather than on agent self-reports, which matters because agents can retry until visible checks pass and then narrate completion while a hidden defect survives. The empirical centerpiece is a 9,240-cell ablation in which the gated loop amplified 2 of 1,800 injected visible-pass/hidden-fail cases, versus 31 of 1,800 for the pre-registered compute-budgeted naive loop.

Core claim

The paper's central claim is Eq. (3): Advance(φi→φi+1,H) ⇐⇒ ∀c∈Ci ∃Ec : Provides(Ec,c) ∧ Admissible(Ec,c,H). A natural-language report from an agent is not an Ec, so self-report has no term in the gate. Evidence is admissible only if it is fresh against the current tracked source hashes, complete, integrity-verified, producer-authorized, execution-attested, supportive of the claim, and with an accepted outcome. The paper's discovery is that this rule can be instantiated in a working lifecycle, and that enforcing the review signal as a gate — rather than adding a reviewer as advice — is what reduces amplified wrong artifacts: the near-compute review-only arm amplified 14 of 1,800 injected cel

What carries the argument

The evidence-admission predicate of Eq. (2), combined with the transition rule of Eq. (3). Gate-admissible evidence is a structured receipt bound by content hashes of the tracked source tree (materialHash/headHash/storyFilesHash), plus policy and command-set hashes, an execution receipt identity (command, arguments, working directory, exit code, output digest), and producer/signature identity. The freshness conjunct rejects evidence the instant the source tree changes, so a reviewer verdict or test receipt from an earlier state cannot advance a later state. Eq. (3) is the control abstraction that converts actor output into a claim and makes lifecycle advancement a claim-admissibility decisio

Load-bearing premise

The gate's real-world value rests on the reviewer lane catching a meaningful share of visible-pass/hidden-fail defects; if the reviewer is no better than the author's passing tests, the gate simply passes the defect through.

What would settle it

Run the powered A4-versus-A2' comparison on tasks where the independent reviewer is deliberately blinded to the injected defect class (for example, a reviewer that checks only style and formatting). If the gated loop's amplified rate stays near or above the naive loop's rate, the effect is reviewer skill, not the gating mechanism. The paper's own task-concentration sensitivity is a concrete place to look for fragility: excluding the du-duration task changes A4 from 2/1800 to 0/1725 and A2' from 31/1800 to 2/1725 amplified cells.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • If the rule holds, 'reviewed', 'tested', and 'DONE' become machine-checkable certificates tied to a specific source revision, not statements from an agent.
  • Enforcing the review signal as a gate rather than as advice reduces visible-pass/hidden-fail amplification (A4 at 2/1800 versus A3 at 14/1800 and the compute-budgeted naive A2' at 31/1800).
  • Merge-readiness can be decided by a certificate that repository tooling consumes, with a protected compare-and-swap so a changed source commit invalidates the certificate before merge.
  • The method composes with existing CI, durable execution, and agent interoperability layers; it adds an assurance layer rather than replacing models or protocols.
  • On clean, well-specified tasks the gate is pure overhead, so its value is specifically as risk mitigation under green-but-wrong failures.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The gate's field value likely depends on the real-work base rate of visible-pass/hidden-fail defects; the paper measures risk mitigation conditional on injection, not that base rate, so cost-benefit decisions in practice remain open.
  • If cross-vendor review is the main source of signal, a single-vendor deployment of the gate may inherit the same blind spots as its reviewer; running the gate with deliberately degraded or same-family reviewers would quantify how much of the benefit comes from gating per se versus reviewer diversity.
  • The same evidence-gating abstraction could transfer to non-code agent lifecycles (research claims, data pipelines, operational workflows), but each domain needs its own evidence packages; the included PINN/quantum schema tests are smoke tests, not evidence for those domains.
  • A testable extension: run the gated loop with a reviewer given only hidden-oracle-failing artifacts and measure the catch rate needed to reproduce the 2/1800 result, separating mechanism strength from task difficulty.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 4 minor

Summary. The paper proposes Proof-or-Stop Lifecycle Control, an evidence-gating method for autonomous coding lifecycles. Agent outputs are treated as claims, and a lifecycle transition is permitted only when every required claim is backed by admissible evidence, formalized in Eqs. (2)–(3) with freshness, source-state binding, authenticated integrity, and receipt identity. The authors instantiate this in an open-source engine and evaluate it via mechanism tests, a pre-registered five-arm powered ablation (9,240 cells), a recovery pilot, a self-application corpus (565 stories, 1,007 findings), and a cross-vendor exhibit. The headline empirical result is that the gated loop A4 reduces visible-pass/hidden-fail amplification from 31/1,800 (A2′, compute-budgeted naive) to 2/1,800, with H1 not-amplified +1.6pp, 95% CI [0.8, 2.5]. The paper also claims the method is a 'model-agnostic, host-neutral control layer.'

Significance. The paper has real strengths. The powered ablation is pre-registered, uses an objective hidden-oracle scorer, reports a matched-cell invariant, gives a cluster-bootstrap CI, and releases records and analysis scripts. The mechanism tests (10/10 loop contract, 18 tamper classes rejected) are useful engineering evidence. The paper is unusually explicit about boundaries, including the task concentration of the primary effect, the null result on clean tasks, and the underpowered experience-reuse study. If the central claim were fully supported, this would be a valuable assurance layer for agentic software engineering. However, the headline 'model-agnostic, host-neutral' claim outreaches the evidence: all ablation cells use one model family with a same-family reviewer, and the primary effect is almost entirely concentrated in a single task. The central mechanism is defensible, but the load-bearing empirical support needs substantial qualification and additional analysis before the advertised claims can stand.

major comments (4)
  1. [Abstract; §13; §7] The abstract and conclusion claim Proof-or-Stop is a 'model-agnostic, host-neutral control layer,' but the powered ablation uses a single Sonnet model family with a same-family reviewer, the cross-vendor review is selectively invoked, and §7 explicitly gates the strong cross-host quorum claim. The evidence supports a control layer instantiated for one model family, plus a verified local git-native handoff mechanism. The headline should be reworded to state the actual scope, or a multi-family / cross-vendor powered run should be provided. This is load-bearing because C1 and C3 are stated as the paper's contributions.
  2. [§5.2, Table 13] The primary H1 contrast is almost entirely driven by the du-duration task: 29 of A2′'s 31 amplifications and both of A4's 2 amplifications occur there. Excluding du-duration, the contrast is 2/1,725 vs 0/1,725 — two events. The cluster-bootstrap CI [0.8, 2.5] and the aggregate '31→2' framing therefore rest on a single trap-active task. The paper acknowledges this in Table 13 but still presents the aggregate as the primary readout. Please add a sensitivity analysis that reports the H1 estimate and CI excluding du-duration, and explicitly frame the result as task-specific mechanism evidence rather than a general effect.
  3. [§5.2; §6; Fig. 9] For review claims, the evidence admitted under Eq. (2) is an LLM verdict. In the powered ablation the reviewer is from the same provider family as the generator (Sonnet), so the 31→2 reduction is not attributable to the gating mechanism alone; it is the gate plus a particular reviewer whose blind spots may be correlated with the generator's. Fig. 9 shows same-vendor Claude lanes passing critical defects (B-086b, B-4c65) that a Codex host caught. A concrete test would be to run the powered ablation with a different-vendor reviewer lane, or at least report per-lane calibration on the injected cells. Without this, the paper cannot support the 'model-agnostic' language.
  4. [§5.2, 'What the powered ablation instantiates'; §A] The powered ablation does not exercise the full Eq. (2) evidence predicate: the released records leave headHash/materialHash null, and the text states the powered harness 'does not material-hash-gate experiment cells.' Thus the primary contrast tests enforcement of a review-and-redo gate, not the freshness/source-state-binding machinery that is the formal centerpiece of the method. This distinction is acknowledged in a caveat, but it should be a prominent part of the claims: the formal lifecycle binding and the empirical control-policy result are different evidence objects, and the paper should not imply that the ablation validates Eqs. (1)–(3) in full.
minor comments (4)
  1. [§5.2, evaluation setup table] The exact dated provider snapshot is not recorded. This is a reproducibility limitation for a paper that emphasizes mechanical reproducibility; please state whether the model label alone is sufficient to re-run the released records.
  2. [Appendix A] The ablation is described as 'pre-registered,' but no external registry or timestamped protocol identifier is given. If a public pre-registration exists, please cite it; otherwise describe the in-house frozen-protocol procedure more precisely.
  3. [Table 11] A2′ (compute-budgeted naive) amplifies 31/1,800 while raw A2 amplifies 18/1,800, so the pre-registered primary control is weaker than the un-budgeted naive loop. The paper notes this, but the text around H1 should clarify that the comparison is A4 vs a deliberately budget-degraded control, not vs a natural naive baseline.
  4. [§8, Fig. 9] The phrase 'same-vendor lanes passed' is strong; since cross-vendor review was invoked selectively, it is more precise to say 'the same-vendor lanes did not report these findings in the selected stories.' Consider softening the caption to match the stated observational scope.

Circularity Check

0 steps flagged

No circular derivation: Eq. (3) is an operational definition, and the empirical contrasts are externally scored by hidden ground truth.

full rationale

The load-bearing formal step, Advance(φi→φi+1,H) ⇔ ∀c∈Ci ∃Ec : Provides(Ec,c) ∧ Admissible(Ec,c,H), is an explicit definition of the proposed control rule rather than a prediction derived from itself; the paper states that 'proof' is used operationally to mean gate-admissible evidence under a stated trust model, not semantic correctness. The empirical claim that the gate reduces visible-pass/hidden-fail amplification is tested in a powered ablation whose outcomes are read objectively from git history, process-event logs, and an independently-authored, known-good-validated acceptance script, with a hidden ground-truth oracle kept out of the agent prompt. No fitted parameter is renamed as a prediction: A2′ is a compute-budgeted control calibrated to A4's pilot median, and the paper explicitly says this 'cannot be read as a clean equal-spend causal estimate.' The not-amplified endpoint is the complement of the amplified count for the primary contrast, but that is an endpoint definition, not a derived result. The self-hosted corpus and same-family reviewer are acknowledged validity limitations ('The evaluation is limited to one model family, 24 ablation tasks, and a self-hosted corpus'; 'because cross-vendor review was invoked selectively, this is an existence and soundness exhibit, not an unbiased marginal-rate estimate'), and they weaken external generalization rather than making any result equivalent to its input by construction. No load-bearing self-citation chain is present; the closest-work citations [9] and [30] are independent papers.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

No physical entities are introduced; the paper's creations are software artifacts (receipt bundles, evidence gates, materialHash bindings, advisory memory layer). These are engineering constructs, not unobserved natural entities, and they carry their own verification. The main assumed burden is the local-key trust model and the hidden-oracle correctness.

free parameters (4)
  • review floor (R,K) = (3,2)
    Chosen for the high-risk DONE gate (Eq. 4); R=3 rounds, K=2 independent passing verdicts. Not derived from first principles.
  • A2' compute-budget band (±20% of A4's pilot median) = token+wall-clock band
    The pre-registered control A2' is budgeted to A4's pilot median spend with ±20% band; realized matching is coarse and the primary contrast depends on this calibration.
  • retry/reflection budgets (R=3 naive retry; two-review-round cap)
    Bound the repair loop; chosen by hand, not optimized.
  • severity round-grading (medium advisory round≥2, high advisory round≥3)
    Controls when findings block; hand-set admission rules.
axioms (5)
  • domain assumption The hidden correctness oracle for the powered ablation is itself correct and independently authored.
    All amplified/not-amplified labels in §5.2/Table 11 are judged against this oracle; if the oracle is wrong, the headline H1 is meaningless.
  • domain assumption Local-key signed receipts prove the attested execution to the required degree.
    Eq. (2) trusts local-key signatures; paper states it does not defeat a compromised runner, so the trust model is an axiom.
  • domain assumption A single LLM family's review signal (with the same provider model) can serve as an 'independent reviewer' for gate purposes.
    The powered ablation A3/A4 uses A4's reviewer, same model family as generator; cross-vendor is selective, so the gate's power rests on this untested independence assumption.
  • standard math Git hash canonicalization (materialHash excludes lifecycle metadata) cleanly captures the relevant source identity.
    Eq. (1) assumes SHA-256 over canonical git ls-tree view is a collision-resistant, stable binding; standard crypto assumption.
  • ad hoc to paper Pre-registration was genuine and complete; no protocol changes were made after seeing results.
    The paper asserts pre-registration (§5.2, Appendix A) but provides no external timestamp or registry; reproducibility of the inferential claim depends on this.

pith-pipeline@v1.3.0-alltime-deepseek · 36260 in / 11561 out tokens · 105884 ms · 2026-08-02T00:46:07.633305+00:00 · methodology

0 comments
read the original abstract

Autonomous coding agents increasingly execute multi-step software work, but lifecycle states such as reviewed, tested, DONE, and ready-to-merge remain claims unless supported by current evidence. We present Proof-or-Stop Lifecycle Control, a method that permits lifecycle transitions only when fresh, tracked-source-state-bound, mechanically verifiable evidence satisfies the relevant gate. The method treats agent outputs as claims rather than lifecycle state, and uses proof operationally to mean gate-admissible evidence under a stated trust model, not semantic program correctness. We evaluate an open-source implementation through mechanism tests, a powered control-policy ablation, and operated self-application evidence. The unattended-loop engine passed 10 of 10 scenarios with zero false-DONE, and local-key receipt bundles rejected 18 tamper classes with zero false accepts. In a 9,240-cell ablation, the pre-registered A4 versus A2-prime comparison reduced visible-pass/hidden-fail amplification from 31 of 1,800 injected cells under a compute-budgeted naive loop to 2 of 1,800 under the gated loop, a 1.6 percentage-point improvement in not-amplified rate with a 95 percent confidence interval of [0.8, 2.5]. A near-compute A3 versus A4 comparison, 14 of 1,800 versus 2 of 1,800, indicates that the gain is associated with enforcing review as a lifecycle gate rather than merely adding a reviewer. The self-application corpus contains 565 stories and 1,007 review findings, with 94.8 percent resolved, plus a 68-row high/critical cross-vendor exhibit. These results support Proof-or-Stop as a model-agnostic, host-neutral control layer for deciding which autonomous-agent claims a lifecycle may act on. The evaluation is limited to one model family, 24 ablation tasks, and a self-hosted corpus.

Figures

Figures reproduced from arXiv: 2607.14890 by Freddie Shi, Ian H. White, Jeffery Hsia, Jek Huang, Jiayi Sun, Wei Huang.

Figure 1
Figure 1. Figure 1: Loop Engineering at a glance. The unattended develop loop runs plan→execute→review→bounded reflect→gate→done; every consequential transition is de￾cided by an evidence gate (the monospace questions), not by the agent. The loop advances only on admissible, code-bound evidence, loops back for a bounded retry when evidence is short, and does not convert unsupported claims into done. One loop iteration is show… view at source ↗
Figure 2
Figure 2. Figure 2: The evidence-gating spine. The agent’s work is reduced to structured evidence bound to code identity; the gate decides the transition by checking Eq. (2). Self-report (dashed) is not admitted as gate evidence [PITH_FULL_IMAGE:figures/full_fig_p012_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Operational evidence flow. Commands produce structured artifacts, artifacts carry code and policy bindings, and gate consumers decide whether to advance, block, or honestly degrade. • plan→dev: an adversarial plan review must persist a structured result (not a “looks good”); scope is frozen into a contract bound to declared file paths. • dev→review: a scope-contract check verifies the diff touches only con… view at source ↗
Figure 4
Figure 4. Figure 4: The unattended evidence-gated loop. Each stage couples what the agent does (black) with the evidence the gate verifies (green); the gate advances only on admissible, code-bound proof (Eq. (2)), loops back for a bounded retry when evidence is short, and fails closed to safe-stop or escalation rather than unsupported done — the engine contract of [PITH_FULL_IMAGE:figures/full_fig_p015_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Amplification-rate ladder for the powered ablation (computed over injected B1–B15 cells only; lower is better). The middle column marks whether the arm uses the Proof-or-Stop control layer: A1–A3 are None, while A4 is Proof-or-Stop. Bars report visible-test-passing but hidden-failing artifacts per 1,000 injected cells. The pre-registered primary contrast is A4 Proof-or-Stop loop versus A2′ compute-budgeted… view at source ↗
Figure 6
Figure 6. Figure 6: Verified loop-fidelity gradient on visible-test-passing wrong artifacts across three tasks and five task-injection cells (15 runs per arm). A2′ has only the visible test and ships the wrong artifact; A4-C adds one independent gate and refuses to ship; A4b-B feeds the review finding back through bounded reflection and repairs before shipping [PITH_FULL_IMAGE:figures/full_fig_p023_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: Git-native handoff. Work transfers as git deltas plus a small JSON; the worktree is reconstructed locally, so “the worktree is too big” never arises. Recovery granularity is the last push. stale-phase warning (HN-9); a handoff to a host missing a capability surfaces a typed capability delta (HN-5); and with no real second host the system records degraded-single-host as local-only evidence rather than spoof… view at source ↗
Figure 8
Figure 8. Figure 8: Of 28 curated deep-set review findings, 26 (93%) were filed while the author’s own tests were passing. Most are coverage/doc-accuracy findings; six behavior-changing defects are itemized separately. Review catches what the author’s tests miss. The headline measurement: on a curated deep set of 12 stories with 28 review findings, 26 of 28 (93%) are smoke_would_miss — the author’s own smoke/test was passing … view at source ↗
Figure 9
Figure 9. Figure 9: Cross-vendor host-2 review caught load-bearing defects after same-vendor lanes [PITH_FULL_IMAGE:figures/full_fig_p032_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: Defect-class recurrence over the corpus. Recurrence is real and the trend is rising (Spearman ρ ≈ 0.72 [22]), but novelty per window stays positive — new defect classes keep appearing, so recall has a ceiling below 1. The causal claim is a separate ON/OFF A/B [17]: δ = outcomeoff − outcomeon, (9) where positive δ means the hinted arm repeats fewer known mistakes. The pilot executed 90 arm-runs with zero s… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

36 extracted references · 11 linked inside Pith

  1. [1]

    Model context protocol (mcp).https://modelcontextprotocol.io, 2024

    Anthropic. Model context protocol (mcp).https://modelcontextprotocol.io, 2024. Ac- cessed: 2 July 2026

    2024

  2. [2]

    Fitting linear mixed-effects models using lme4.Journal of Statistical Software, 67(1):1–48, 2015

    Douglas Bates, Martin Mächler, Ben Bolker, and Steve Walker. Fitting linear mixed-effects models using lme4.Journal of Statistical Software, 67(1):1–48, 2015

    2015

  3. [3]

    Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B, 57(1):289–300, 1995

    Yoav Benjamini and Yosef Hochberg. Controlling the false discovery rate: A practical and powerful approach to multiple testing.Journal of the Royal Statistical Society: Series B, 57(1):289–300, 1995

    1995

  4. [4]

    Bootstrap methods: Another look at the jackknife.The Annals of Statistics, 7(1):1–26, 1979

    Bradley Efron. Bootstrap methods: Another look at the jackknife.The Annals of Statistics, 7(1):1–26, 1979

    1979

  5. [5]

    Ronald A. Fisher. On the interpretation of Chi-Square from contingency tables, and the calculation of P.Journal of the Royal Statistical Society, 85(1):87–94, 1922

    1922

  6. [6]

    Agent2agent (a2a) protocol.https://a2a-protocol.org, 2025

    Google and Linux Foundation. Agent2agent (a2a) protocol.https://a2a-protocol.org, 2025. Cross-vendor agent interoperability. Google announcement: 9 April 2025; Linux Foundation project launch: 23 June 2025. Accessed: 2 July 2026. 39

    2025

  7. [7]

    An overview of the runtime verification tool Java PathEx- plorer.Formal Methods in System Design, 24(2):189–215, 2004

    Klaus Havelund and Grigore Roşu. An overview of the runtime verification tool Java PathEx- plorer.Formal Methods in System Design, 24(2):189–215, 2004

    2004

  8. [8]

    Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan

    Carlos E. Jimenez, John Yang, Alexander Wettig, Shunyu Yao, Kexin Pei, Ofir Press, and Karthik Narasimhan. SWE-bench: Can language models resolve real-world github issues? InInternational Conference on Learning Representations, 2024. arXiv:2310.06770. https: //arxiv.org/abs/2310.06770

    Pith/arXiv arXiv 2024

  9. [9]

    Agentic agile-v: From vibe coding to verified engineering in software and hardware development.arXiv preprint arXiv:2605.20456, 2026

    Christopher Koch. Agentic agile-v: From vibe coding to verified engineering in software and hardware development.arXiv preprint arXiv:2605.20456, 2026. https://arxiv.org/abs/ 2605.20456

    Pith/arXiv arXiv 2026

  10. [10]

    Langgraph: Stateful, resumable agent graphs.https://langchain-ai.github

    LangChain. Langgraph: Stateful, resumable agent graphs.https://langchain-ai.github. io/langgraph/, 2024. Accessed: 2 July 2026

    2024

  11. [11]

    Self- refine: Iterative refinement with self-feedback

    Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, Shashank Gupta, Bodhisattwa Prasad Majumder, Katherine Hermann, Sean Welleck, Amir Yazdanbakhsh, and Peter Clark. Self- refine: Iterative refinement with self-feedback. InAdvances in Neural Information Processing Sy...

    Pith/arXiv arXiv 2023

  12. [12]

    LLM critics help catch LLM bugs

    Nat McAleese, Rai Michael Pokorny, Juan Felipe Ceron Uribe, Evgenia Nitishinskaya, Maja Trebacz, and Jan Leike. LLM critics help catch LLM bugs. arXiv preprint arXiv:2407.00215. https://arxiv.org/abs/2407.00215, 2024

    Pith/arXiv arXiv 2024

  13. [13]

    Ablation studies in artificial neural networks.arXiv preprint arXiv:1901.08644, 2019.https://arxiv

    Richard Meyes, Melanie Lu, Constantin Waubert de Puiseau, and Tobias Meisen. Ablation studies in artificial neural networks.arXiv preprint arXiv:1901.08644, 2019.https://arxiv. org/abs/1901.08644

    Pith/arXiv arXiv 1901

  14. [14]

    Microsoft agent framework.https://github.com/microsoft/agent-framework,

    Microsoft. Microsoft agent framework.https://github.com/microsoft/agent-framework,

  15. [15]

    George C. Necula. Proof-carrying code. InProceedings of the 24th ACM SIGPLAN-SIGACT Symposium on Principles of Programming Languages, pages 106–119, 1997

    1997

  16. [16]

    Openai agents sdk

    OpenAI. Openai agents sdk. https://openai.github.io/openai-agents-python/, 2025. Accessed: 2 July 2026

    2025

  17. [17]

    Donald B. Rubin. Estimating causal effects of treatments in randomized and nonrandomized studies.Journal of Educational Psychology, 66(5):688–701, 1974

    1974

  18. [18]

    Goal structuring notation community standard version

    SCSC Assurance Case Working Group. Goal structuring notation community standard version

  19. [19]

    Reflexion: Language agents with verbal reinforcement learning

    Noah Shinn, Federico Cassano, Edward Berman, Ashwin Gopinath, Karthik Narasimhan, and Shunyu Yao. Reflexion: Language agents with verbal reinforcement learning. InAdvances in Neural Information Processing Systems, 2023. arXiv:2303.11366.https://arxiv.org/abs/ 2303.11366

    Pith/arXiv arXiv 2023

  20. [20]

    GSN Community Standard SCSC-141C, Safety-Critical Systems Club, 2021

    2021

  21. [21]

    in-toto and SLSA.https://slsa.dev/blog/2023/05/in-toto-and-slsa, 2023

    SLSA. in-toto and SLSA.https://slsa.dev/blog/2023/05/in-toto-and-slsa, 2023. Ac- cessed: 2 July 2026

    2023

  22. [22]

    DBOS: A 40 DBMS-oriented operating system

    Athinagoras Skiadopoulos, Qian Li, Peter Kraft, Kostis Kaffes, Daniel Hong, Shana Mathew, David Bestor, Michael Cafarella, Vijay Gadepally, Goetz Graefe, Jeremy Kepner, Christos Kozyrakis, Tim Kraska, Michael Stonebraker, Lalith Suresh, and Matei Zaharia. DBOS: A 40 DBMS-oriented operating system. InProceedings of the VLDB Endowment, volume 15, pages 21–3...

    2022

  23. [23]

    SLSA provenance.https://slsa.dev/provenance,

    Supply-chain Levels for Software Artifacts. SLSA provenance.https://slsa.dev/provenance,

  24. [24]

    The proof and measurement of association between two things.The American Journal of Psychology, 15(1):72–101, 1904

    Charles Spearman. The proof and measurement of association between two things.The American Journal of Psychology, 15(1):72–101, 1904

    1904

  25. [25]

    in-toto: Providing farm-to-table guarantees for bits and bytes

    Santiago Torres-Arias, Hammad Afzali, Trishank Karthik Kuppusamy, Reza Curtmola, and Justin Cappos. in-toto: Providing farm-to-table guarantees for bits and bytes. In28th USENIX Security Symposium, pages 1393–1410, 2019.https://www.usenix.org/conference/ usenixsecurity19/presentation/torres-arias

    2019

  26. [26]

    Voyager: An open-ended embodied agent with large language models

    Guanzhi Wang, Yuqi Xie, Yunfan Jiang, Ajay Mandlekar, Chaowei Xiao, Yuke Zhu, Linxi Fan, and Anima Anandkumar. Voyager: An open-ended embodied agent with large language models. arXiv preprint arXiv:2305.16291, 2023.https://arxiv.org/abs/2305.16291

    Pith/arXiv arXiv 2023

  27. [27]

    Temporal: Durable execution for workflows.https://temporal.io,

    Temporal Technologies. Temporal: Durable execution for workflows.https://temporal.io,

  28. [28]

    PROV-overview: An overview of the PROV family of documents

    World Wide Web Consortium. PROV-overview: An overview of the PROV family of documents. https://www.w3.org/TR/prov-overview/, 2013. W3C Working Group Note. Accessed: 2 July 2026

    2013

  29. [29]

    White, Doug Burger, and Chi Wang

    Qingyun Wu, Gagan Bansal, Jieyu Zhang, Yiran Wu, Beibin Li, Erkang Zhu, Li Jiang, Xiaoyun Zhang, Shaokun Zhang, Jiale Liu, Ahmed Hassan Awadallah, Ryen W. White, Doug Burger, and Chi Wang. AutoGen: Enabling next-gen LLM applications via multi- agent conversation. InConference on Language Modeling, 2024. arXiv:2308.08155.https: //arxiv.org/abs/2308.08155

    Pith/arXiv arXiv 2024

  30. [30]

    Researchloop: An evidence-gated control plane for ai-assisted research.arXiv preprint arXiv:2605.28282, 2026.https://arxiv.org/abs/2605.28282

    Yihan Xia and Taotao Wang. Researchloop: An evidence-gated control plane for ai-assisted research.arXiv preprint arXiv:2605.28282, 2026.https://arxiv.org/abs/2605.28282

    Pith/arXiv arXiv 2026

  31. [31]

    Edwin B. Wilson. Probable inference, the law of succession, and statistical inference.Journal of the American Statistical Association, 22(158):209–212, 1927

    1927

  32. [32]

    not amplified

    Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. ReAct: Synergizing reasoning and acting in language models. InInternational Conference on Learning Representations, 2023. arXiv:2210.03629.https://arxiv.org/abs/2210.03629. 41 A Pre-registered protocols Ablation (§5.2).Five arms — A1 prompt-only; A2 naive-retry (R=...

    Pith/arXiv arXiv 2023

  33. [35]

    Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press

    John Yang, Carlos E. Jimenez, Alexander Wettig, Kilian Lieret, Shunyu Yao, Karthik Narasimhan, and Ofir Press. SWE-agent: Agent-computer interfaces enable automated software engineering. InAdvances in Neural Information Processing Systems, 2024. arXiv:2405.15793. https://arxiv.org/abs/2405.15793

    Pith/arXiv arXiv 2024

  34. [2024]

    Accessed: 2 July 2026

    Durable / resumable workflow execution. Accessed: 2 July 2026

    2026

  35. [2025]

    Accessed: 2 July 2026

    Durable, multi-agent orchestration. Accessed: 2 July 2026

    2026

  36. [2026]

    Accessed: 2 July 2026

    2026