{"id":"559d6a4a-4cca-46e9-bd20-a02801ef4788","arxiv_id":"2609.29921","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SpecHarness separates agent proposals from authoritative state, committing task completion only from validated evidence against spec-derived obligations, and it raises official pass rates from 61.1% to 73.1% on SkillsBench and 86.2% to 91.3% on GuideBench.","lead":"The paper shows that LLM agents routinely claim completion they have not earned: across seven models, completion-claim rates exceed official evaluator pass rates by 28.7 to 37.9 percentage points, and only 79.6 to 86.4 percent of 509 source-grounded task directions are actually satisfied. It proposes SpecHarness, a runtime that compiles written specifications into verifiable obligations and commits completion only when qualified evidence satisfies them.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek/deepseek-v4-flash","headline":"The preventive no-bypass guarantee (Eq. 17) rests on an audited, not verified, enumeration of effect-equivalent paths; one reachable bypass on any closure-qualified surface would falsify the preventive half of the central claim, and only 44% of SkillsBench obligations (0% of GuideBench) even…","rationale":"The reader's weakest_assumption is the completeness of the closure audit's capability graph underpinning the no-bypass guarantee (Eq. 7–8, Eq. 17, Appendix B.2), and my read reaches the same point: it is the most claim-specific, least-secure condition on which the preventive half of 'agent proposes, SpecHarness commits' rests. An effect-equivalent path not represented in AvailPaths(Hexec) would let an agent action establish specification-governed state without validation, directly violating the authority principle. The paper handles this honestly — it states the conditionality in Appendix B.2 (b274, b277, b91), runs 1,628 stress cases, and excludes all 11 detected violations from closure-qualified surfaces — which is why this is a scoped, auditable risk rather than a soundness failure. What remains unresolved is that the enumeration is self-audited and fixed-set; the equivalence relation is semantic and not formally verifiable, and no independent second-party audit exists. I additionally sharpen the reader's point with a scope observation: preventiveness covers only 44% of SkillsBench hard obligations and none of GuideBench (Table 7), so the guarantee protects a minority footprint, while the universal authority mechanism — the finalization gate (Eq. 12) requiring all fresh mandatory obligations satisfied — is bypass-robust and depends instead on validator soundness, which is the standard verification risk. I weighed the attribution confound (Raw→SpecHarness bundles feedback, repair, and realized compute; b262, b314, b335) and judged it acknowledged and partially mitigated by the architecture ablation (Table 5), where removing commitment produces the largest S–A degradation; it weakens the empirical superiority claim but does not directly falsify the authority invariant, so I do not elevate it above the bypass completeness. Given the acknowledged-but-unverified audit condition, the reader's CONDITIONAL verdict stands; the missing pieces remain an independent closure audit and a matched-compute ablation, both of which the concrete test addresses for the former.","tokens_in":24762,"tokens_out":7310,"duration_ms":52126,"concrete_test":"Run an independent adversarial closure audit: a second team with system access but blind to the authors' capability graph enumerates reachable effect-equivalent paths on 3–5 closure-qualified SkillsBench mediated surfaces (e.g., the data-to-d3 file-write surface in Appendix E.2), probing (i) symlink and relative-path escape from authorized output paths, (ii) direct subprocess spawn that invokes a protected tool without the wrapper, (iii) TOCTOU windows between the authorization check and controlled execution, and (iv) malformed-proposal normalization collisions that map to a privileged canonical action. Any path that reaches the mediated effect without passing Auth falsifies the no-bypass claim on that surface; zero findings across an independently constructed enumeration would be the first external support for the guarantee's completeness.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — 'agent proposes, SpecHarness commits' — has two load-bearing pillars: the state-write invariant (Eq. 9–10), which is an implementation property with no formal verification, and the preventive no-bypass guarantee (Eq. 7–8, 17, Appendix B.2), whose completeness of AvailPaths(Hexec) is the least secure condition. The paper's own stress suite (1,628 cases, Table 11) found 11 violations, and honestly excluded their surfaces; that is good practice, but the enumeration is a fixed adversarial set over a semantic equivalence relation (π∼eff a), which is undecidable in general and has zero independent verification. The authors explicitly scope the guarantee to the audited boundary and threat model (b274, b277, b91, b53), so this is an acknowledged conditionality, not a hidden flaw — but nothing in the paper reduces the residual risk that a reachable but unenumerated path (wrapper bypass via direct subprocess, symlink/relative-path alias, TOCTOU between authorization and controlled execution, or a normalization-collision proposal) establishes authoritative state without validation on a mediated surface. That would falsify the preventive half of the claim. Note also the empirical thinness of the preventive layer: mediate-and-commit covers only 44% of SkillsBench hard obligations and 0% of GuideBench (Table 7, b161), so the guarantee protects a minority footprint; the universal authority mechanism is the finalization gate (Eq. 12), which is robust to bypasses but depends on validator soundness, a generic verification risk rather than an authority-specific one. The attribution confound (Raw→SpecHarness bundles feedback, repair, and compute; b262, b314, b335) is real but acknowledged, and Table 5's ablation suggests commitment carries the largest S–A effect; I do not find it more load-bearing than the bypass completeness for the stated strongest claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek/deepseek-v4-flash","summary":"The paper identifies two structural failures in specification-following LLM agents—the understanding–execution gap (understood requirements not satisfied during execution) and the state–authority gap (agent-reported completion is not verified completion)—and proposes SpecHarness, a runtime that separates agent proposals from authoritative state. Visible specifications are compiled into source-linked obligations; evidence from qualified validators commits versioned obligation state; finalization is allowed only when all fresh mandatory obligations are satisfied (Eq. 12). On all 87 SkillsBench tasks the official-verifier pass rate rises from 61.1% to 73.1% macro across seven task-agent models (GuideBench: 86.2%→91.3%), with paired task-bootstrap 95% CIs excluding zero and per-model exact McNemar tests significant after Holm correction (Tables 16–17). The compiler is selected on a task-disjoint development set and frozen before evaluation; held-out official verifiers are used only for final pass measurement and post-hoc alignment. The paper is explicit that preventive mediation covers only closure-audited surfaces (44% of SkillsBench hard obligations, 0% of GuideBench, Table 7), that the no-bypass guarantee is conditional on an audited capability graph (b274, b277), and that the end-to-end comparison does not isolate the authority mechanism from online validation, feedback, and repair (b314, b335).","tokens_in":25152,"tokens_out":7752,"duration_ms":60318,"significance":"If the results hold, the paper is a substantial contribution: it gives an explicit authority boundary for specification-following agents and evaluates it with an externally grounded protocol. Concrete strengths: (i) the primary Pass metric is measured by held-out official verifiers that never enter obligation construction or runtime feedback (b56); (ii) the compiler is frozen on a task-disjoint development set (Appendix A.1); (iii) the statistics are paired and multiplicity-corrected (Tables 16–17); (iv) the paper is unusually candid, scoping the guarantee to grounded mandatory obligations (b100, b161) and reporting the 11 stress-test violations it excluded from mediated claims (Table 11). The residual correctness risk is concentrated in the preventive half of the central claim: the no-bypass guarantee rests on a finite, audited enumeration of effect-equivalent paths over an undecidable equivalence relation, and the mediated footprint is a minority of the obligation surface. These are acknowledged conditions, so the overall claim is defensible as stated, but the presentation should be rebalanced so readers cannot over-read the preventive guarantee.","major_comments":[{"comment":"The no-bypass guarantee is conditional on every reachable effect-equivalent path being represented in the audited capability graph, and the authors state this conditionality explicitly (b91, b274, b277), so this is not a hidden flaw. What remains unresolved is any argument for completeness of AvailPaths(Hexec): it is a fixed adversarial enumeration over an effect-equivalence relation pi~_eff a that is undecidable in general, and the 1,628-case stress suite (Table 11) is a finite set that found 11 violations, honestly excluded. The paper gives no bound on residual bypass risk for an unenumerated but reachable path on a closure-qualified surface, which would falsify the preventive half of 'agent proposes, SpecHarness commits.' I recommend (a) stating at the level of the central claim that preventive mediation provides no completeness guarantee beyond the audited enumeration, and (b) phrasing Eq. (8)'s existence-negation conclusion in the audited-conditional form that the prose already uses.","section":"Appendix B.2, Eq. (17)","authors_acknowledge":"A preventive no-bypass claim applies only when ev- ery reachable effect-equivalent path is represented in the capability graph and mediated within the trusted bound- ary.","authors_acknowledge_p":0.64},{"comment":"The preventive layer is empirically thin: mediate-and-commit covers 44.0% of SkillsBench hard obligations and 0% of GuideBench, so the universal authority mechanism is the finalization gate (Eq. 12) operating over validate-and-commit evidence. The authors acknowledge this (b161), but the abstract's 'governing execution and finalization through versioned obligation state' overstates the preventive contribution, since for a majority of obligations (and all of GuideBench) the runtime validates and commits after the fact rather than mediating the action. The contribution should be rebalanced to present evidence-authorized commitment plus the finalization gate as the universal guarantee, with mediate-and-commit as a closure-audited special case; otherwise the reader will infer a preventive guarantee over the full obligation surface that the data do not support.","section":"Table 7, Runtime-Mode Allocation","authors_acknowledge":"Its agent-specific contribution is to compile visible specifications into source-linked obligations and use qual- ified evidence to govern versioned state and finalization; preventivemediationremainslimitedtoclosure-auditedsur- faces (Appendix B.2).","authors_acknowledge_p":0.83},{"comment":"The headline Pass gain (+12.0 pp, 95% CI [+9.1, +14.9]) is externally grounded and statistically solid. The causal attribution to the authority mechanism is weaker: the Raw-to-SpecHarness comparison bundles online validation, source-linked feedback, repair, and additional realized compute, which the authors repeatedly acknowledge (b148, b262, b314, b335), and the Table 5 ablations separate mechanisms within an interaction framework that does not match trajectories or compute. The directionality of the mechanisms is supported, but the magnitude of the isolated contribution of the authority boundary is not separately quantified. This is acknowledged, so I do not treat it as a defect; I recommend one sentence in the results section stating explicitly that the authority mechanism's isolated effect is not separately estimated and that the ablations are relative rather than matched-compute comparisons.","section":"Tables 1 and 5, Appendix D.1","authors_acknowledge":"Thus, validator definitions and measurement criteria are shared, whereas condition- specific acceptance rules, online feedback, repair trajecto- ries,andrealizedcomputearenotmatched.Thecomparison therefore evaluates the complete SpecHarness architecture rather than the isolated causal effect of commitment.","authors_acknowledge_p":0.95}],"minor_comments":[{"comment":"In the supplied full text the abstract reads as a single run-on token sequence ('Largelanguagemodelagentsoperateunder...'); please verify the production pipeline so the published abstract is word-spaced.","section":"Abstract (rendered text)","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The 'Normalized feedback time' row reports 1.00 versus 0.44 with no stated normalization base; please define the reference condition and unit in the caption or surrounding text.","section":"Table 23","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The notation Z^b_x \\ S^b_{m,c}(x) is dense; a one-line gloss clarifying that S^b is the subset of the frozen measurement surface supported by the shared validators, and that U-E aggregates over measurement units rather than tasks, would help readers who do not go to Appendix D.1.","section":"Eq. (13), Metrics","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The removal of four duplicate GuideBench rules is a de-duplication choice that acts as a free parameter; please add one sentence stating that the de-duplication is applied identically to all conditions and therefore cannot bias the paired comparisons.","section":"Experiments, GuideBench","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The replays are honestly described as 'audited protocol replays ... not verbatim logs'; please add one sentence on how the illustrative obligation was selected, to avoid the appearance of cherry-picking.","section":"Table 25","authors_acknowledge":null,"authors_acknowledge_p":null},{"comment":"The provider model identifiers are proprietary endpoints; a brief reproducibility note on version pinning and endpoint availability would strengthen the experimental protocol.","section":"Table 14","authors_acknowledge":null,"authors_acknowledge_p":null}],"recommendation":"minor_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek/deepseek-v4-flash","letter":"The thing to know: this is a genuine system contribution, and it is unusually honest about its limits. The obligation-evidence-commit architecture — versioned ledger, source-linked obligations, agent-proposes-runtime-commits — is new relative to the post-hoc verification, completion-gating, and runtime-enforcement work it cites. Those each cover one intervention stage; SpecHarness ties execution control, effect validation, and acceptance into a single authority boundary. The headline numbers (SkillsBench Pass 61.1 to 73.1, GuideBench 86.2 to 91.3) are statistically solid: paired task bootstrap, exact per-model McNemar with Holm correction, and — the part I value most — held-out official verifiers never enter obligation construction or runtime feedback. That is real external grounding for the central improvement claim.\n\nCredit where earned: compiler selection on a task-disjoint dev set, frozen before evaluation; validator qualification completed before task-agent execution; residual failures reported plainly (Tables 20-21); and the authors state their own main confound twice over (b262, b314, b335). They don't resolve it — Raw-to-SpecHarness bundles online validation, feedback, repair, and more compute — but they don't hide it, and Table 5's ablation points to commitment carrying the largest S-A effect, which is the direction the strongest claim needs.\n\nThe soft spots, in proportion. The attribution confound is the closest thing to load-bearing. If the claim is that the authority boundary itself causes the gain, the current comparison doesn't isolate it; a matched-compute ablation (commitment without repair feedback) is what closes that. I read the stress-test's bypass concern as somewhat over-weighted. The preventive no-bypass guarantee (Eq 17) does rest on audited enumeration of effect-equivalent paths, and that equivalence is undecidable in general, so the guarantee is conditional — but the authors say so themselves (b274, b277), and the universal mechanism is the finalization gate (Eq 12), which does not depend on bypass completeness at all. It requires fresh satisfied commitments, full stop. A reachable bypass would dent the preventive story, which already covers only 44% of SkillsBench and 0% of GuideBench hard obligations, but the central claim — completion requires validated evidence, not agent self-report — survives it. The stress test is right about the preventive half; it overstates the blast radius onto the claim as framed.\n\nVerdict: CONDITIONAL, as the reader said, resting on a matched-compute ablation and independent scrutiny of the compiler-extracted measurement surface. This deserves a serious referee. Bring it to reading group.","headline":"A real system contribution with honest, well-executed evaluation; the 12-point gain is statistically solid but conflates the authority boundary with feedback and repair, and the paper's own scoping does most of the referee's work.","tokens_in":25881,"tokens_out":2834,"duration_ms":22464},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek/deepseek-v4-flash","headline":"The paper claims that when the same language model both acts under a specification and declares its own completion, self-assessment cannot establish that the required state was reached; a runtime called SpecHarness should own acceptance…","keywords":["LLM agents","specification authority","State Authority Principle","obligation-evidence-commit","understanding-execution gap","state-authority gap","runtime enforcement","completion verification"],"falsifier":"Add a reachable, unenumerated effect-equivalent path to a mediated surface, such as an indirect tool call or subprocess the capability graph misses, and show an agent action committing authoritative state without passing the bound validator. The paper's own closure stress tests found 11 bypasses, all on surfaces then excluded from mediation; a bypass that remained on a mediated surface, or an official-verifier pass attributable to such a path, would refute the core authorization claim.","tokens_in":26236,"feed_emoji":"🔏","tokens_out":5135,"duration_ms":37269,"temperature":0.7,"pith_summary":"Large language model agents are usually handed task instructions and guidelines, then left to act and to declare themselves done under those same instructions, so the specification stays context for the very model that judges the outcome. The paper argues this collapse of proposal and acceptance produces two measurable failures: requirements that are understood but not satisfied, and completion claims that outrun verified completion. It then makes specifications the authority by construction: a runtime called SpecHarness compiles visible requirements into source-linked obligations, updates specification-governed state only from admissible evidence produced by qualified validators, and blocks finalization until every fresh mandatory obligation is satisfied. Across seven models this raises official pass rates from 61.1% to 73.1% on all 87 SkillsBench tasks and from 86.2% to 91.3% on GuideBench. If the results hold, a specification can serve as authority over what counts as correct execution and compliant completion rather than merely behavioral guidance.","feed_headline":"Spec-trusting runtime lifts AI pass rate from 61 to 73 percent","feed_subtitle":"Verify with evidence, not agent claims: SkillsBench and decision benchmarks both improve across all seven models.","key_machinery":"The load-bearing object is the obligation–evidence–commit runtime with versioned authoritative state, formalized in Eqs. (6)–(12). Visible specifications are compiled into a source-linked obligation IR whose hard obligations are mandatory, source-grounded, and bound to qualified validators, while ambiguous or subjective requirements remain advisory or abstained. An action proposal is authorized only if it matches applicable obligations, and mediation is preventive only on a closure-audited action surface where every reachable effect-equivalent path is governed or denied (Eqs. 7–8). Evidence is admissible only when it comes from a trusted provider, satisfies the obligation's criterion, and yields a passed or failed validator result (Eq. 9). Commitment is atomic and versioned, and a result counts as fresh only when its dependency digest matches current versions, so a bound mutation invalidates affected commitments and forces revalidation before finalization (Eqs. 11–12). The finalization gate then requires every fresh mandatory obligation to be satisfied before completion is accepted.","core_discovery":"The paper's central claim is a division of labor named the State Authority Principle: an agent may propose, plan, and act, but it cannot itself establish specification-governed state; only admissible evidence from qualified providers may commit that state. SpecHarness operationalizes this as an obligation–evidence–commit runtime, summarized by the paper's own slogan 'agents propose; SpecHarness commits.' Concretely, the runtime compiles agent-visible prompts, guidelines, skill files, and workspace schemas into source-linked obligations, each carrying provenance, authorization, effect, state, and control fields; it authorizes matched actions under closure-audited mediation or enforces validate-and-commit on isolated channels; it writes versioned ledger entries only when a trusted observation plus a qualified validator's passed or failed result are admissible under Eq. (9); and it allows finalization only when every fresh mandatory obligation is satisfied under Eq. (12). The measured consequence is that unverified self-assessment stops being a completion signal on artifact tasks and on guideline-driven decision tasks alike.","pith_inferences":["The versioned obligation ledger doubles as an audit trail, so a natural extension the paper leaves implicit is per-task evidence certificates usable for downstream compliance or accountability checks.","The measured 28.7–37.9 percentage-point gap between self-reported and verified completion suggests a cheap diagnostic for any agent benchmark: a large self-report delta flags the surfaces where authority boundaries are most needed.","Because ambiguous and subjective requirements stay advisory, the framework's promise is bounded to monitorable requirements; a credible extension would probe where the advisory boundary sits on open-ended creative or judgment tasks."],"forward_implications":["Agent self-assessment stops being a completion signal: acceptance requires admissible evidence from qualified validators, so unsupported completion claims cannot establish authoritative state.","Specifications govern covered execution, not just the terminal artifact: source-linked obligations gate matched actions and force effect validation, shrinking the understanding–execution gap.","The authority pattern transfers from artifact state to decision state: on GuideBench, rule-local decisions are governed by the same obligation–evidence–commit discipline.","Versioned freshness prevents stale completion: any bound dependency mutation invalidates affected commitments and forces revalidation, which the ablation shows restores roughly 96% of affected tasks after repair."],"supporting_citations":[{"why":"Supplies the 87-task SkillsBench benchmark whose official verifiers and pass rates are the primary measurement surface.","marker":"Li et al. 2026b"},{"why":"Supplies the 1,042-task GuideBench guideline-following benchmark that extends the results to decision state.","marker":"Diao et al. 2025"},{"why":"AgentSpec is the runtime-enforcement baseline SpecHarness must beat on SkillsBench.","marker":"Wang, Poskitt, and Sun 2025"},{"why":"VeriMAP is the completion-gating baseline compared on both benchmarks.","marker":"Xu et al. 2026b"},{"why":"The reference-monitor principle on which the specification authority boundary is built.","marker":"Saltzer and Schroeder 1975"},{"why":"The transactional-commit principle behind atomic ledger updates and versioned commitment.","marker":"Gray and Lamport 2004"},{"why":"Runtime verification, the basis for online obligation checking during execution.","marker":"Leucker and Schallhart 2009"},{"why":"SatLM is the solver-based computation baseline compared on GuideBench.","marker":"Ye et al. 2023"}],"fun_headline_variants":["Let specs sign off, not agents: 61% to 73% pass rate","SpecHarness: agents propose, evidence commits, AI score jumps","Say no to self-certified AI: runtime lifts pass rate 12 points","Obligations beat opinions: spec authority boosts AI to 73%","From agent claims to verified state: spec-led runtime wins"],"cache_read_input_tokens":0,"weakest_assumption_plain":"The no-bypass guarantee holds only if the audited capability graph lists every way an agent could produce the same effect as a mediated action; a reachable but unenumerated path would let an agent action commit authoritative state without validation, and the paper acknowledges this conditionality itself in Appendix B.2.","fun_headline_variants_meta":{"raw":{"variants":["Let specs sign off, not agents: 61% to 73% pass rate","SpecHarness: agents propose, evidence commits, AI score jumps","Say no to self-certified AI: runtime lifts pass rate 12 points","Obligations beat opinions: spec authority boosts AI to 73%","From agent claims to verified state: spec-led runtime wins"]},"model":"deepseek/deepseek-v4-flash","effort":"low","cost_usd":0.000162,"raw_usage":{"cost":0.000161868,"is_byok":false,"cost_details":{"upstream_inference_cost":0.000161868,"upstream_inference_prompt_cost":0.000114548,"upstream_inference_completions_cost":4.732e-05},"total_tokens":1192,"prompt_tokens":1023,"completion_tokens":169,"prompt_tokens_details":{"audio_tokens":0,"video_tokens":0,"cached_tokens":256,"cache_write_tokens":0},"completion_tokens_details":{"audio_tokens":0,"image_tokens":0,"reasoning_tokens":69}},"tokens_in":1023,"tokens_out":169,"cost_source":"openrouter_usage","duration_ms":3931,"image_pages":[],"temperature":1.0,"billed_attempts":1,"reasoning_tokens":69,"transport_provider":"openrouter","cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-09-25T05:59:14.538961+00:00","model_set":{"reader":"deepseek/deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.1-daily-deepseek-or-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Add a reachable, unenumerated effect-equivalent path to a mediated surface, such as an indirect tool call or subprocess the capability graph misses, and show an agent action committing authoritative state without passing the bound validator. The paper's own closure stress tests found 11 bypasses, all on surfaces then excluded from mediation; a bypass that remained on a mediated surface, or an official-verifier pass attributable to such a path, would refute the core authorization claim.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek/deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}