{"id":"29cd421c-72a8-4188-991b-a1b5f5e84228","arxiv_id":"2607.27309","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Compiling prose agent skills into executable harnesses raises mandated-step execution from 56% to 86% of steps and lifts full-procedure completion by 2.3×, independent of model capability.","lead":"AI agents increasingly get their jobs from plain-text skill files they are supposed to follow; this paper shows such instructions get followed only about half the time, and presents SIGIL, a compiler that turns the skill file into an executable program so required steps run as code. On 30 skills, compiled agents perform 86% of mandated steps versus 56%, finish whole procedures 2.3× more often, and use about 60% of the tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline AMC numbers rely on unseen, possibly compiler-influenced reference procedures; independent annotation is needed to confirm the 30-point gap.","rationale":"The reader's weakest_assumption precisely identifies the load-bearing concern: the reference procedures are the denominator for every AMC number, yet their construction is unreported and may be confounded with the compiler. This is not a minor detail; it determines whether the 30-point gap is a real effect of compilation or an artifact of how 'mandated steps' were defined. The paper's own robustness check (§5.7) only addresses gate credit, not the reference procedure itself, and the four self-authored skills heighten the risk. No other concern is as central: the internal inconsistencies (32 vs 30 skills, missing G2) are cosmetic, and the two-model capability axis is explicitly qualified. The correct verdict remains CONDITIONAL: the central claim is plausible and well-mechanized, but the measurement apparatus must be exposed and independently validated before the numbers can be taken at face value. The proposed test directly settles whether this concern lands.","tokens_in":15110,"tokens_out":4366,"duration_ms":37593,"concrete_test":"Recruit two annotators with no connection to the authors; give them the 30 SKILL.md files plus a written rubric for identifying MUST steps, with examples from unrelated skills. Ask them to independently produce reference procedures, blinded to the compiled artifacts and to the paper's results. Compute inter-annotator agreement (e.g., Cohen's kappa) on the mandate lists. Then score the existing runs (or a random subset of at least 10 skills) against each independent reference procedure using the same four-way judge protocol, and recompute the prose vs harness AMC gap. If the gap remains ≥20 points and kappa ≥0.6, the concern is resolved; if the gap drops below 10 points or agreement is low, the central claim is materially weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Every headline number—56% prose, 86% harness, 2.3× completion—is a fraction of a reference procedure of 'mandatory steps' that the authors distilled from each SKILL.md (§5.2). The paper never states who did this distillation, under what instructions, or whether the distillers saw the compiled harnesses, STRUCT-COV reports, or the study hypothesis. Four of the 30 skills were authored by the authors for this study (§5.1), and the compiler's own Extraction is a similar model-driven reading of the same prose. If the reference procedure was shaped—even implicitly—by what SIGIL can enforce (e.g., by aligning the mandate list with the code gates that STRUCT-COV reports, or by phrasing mandates in the deontic vocabulary the extractor consumes), then the harness is scored against a target it was built to hit, while prose is scored against the same possibly skewed target. The direction of bias is predictable: including only mechanically checkable mandates, or wording them as explicit tool invocations, advantages the harness. The §5.7 gate-credit check (77% vs 56% without automatic gate credit) addresses one scoring asymmetry but does not validate the ground-truth reference itself. A model-based judge with no reported reliability further compounds this. This is a ground-truth measurement threat, not a claim of fraud; it is the condition least secured by the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Skill Compilation, realized in SIGIL, a compiler that translates natural-language agent skills (e.g., SKILL.md files) into executable harnesses. The key design is AG-IR, a typed intermediate representation that separates model-owned cognition from code-owned mechanism, with provenance back to the source prose. Extraction grounds every rule in verbatim quotations and applies compile gates, while Lowering is deterministic and produces Jac/OSP modules. The evaluation reports that, across 30 skills and two OpenAI models, compiled harnesses perform 86% of mandated steps versus 56% for prose, complete full procedures 2.3× more often, and use 0.58× the tokens at the median, while remaining at 86% across model generations.","tokens_in":15328,"tokens_out":5115,"duration_ms":43657,"significance":"If the empirical claims hold, this is a valuable contribution to dependable AI-agent execution: it targets the actual performance of mandated procedural steps rather than artifact-level output checks, and it shows that compiled structure, rather than model capability, can carry procedure compliance. The design has several concrete strengths: deterministic lowering, compile gates such as G4 (the compile oracle) and G5 (STRUCT-COV), provenance from skill sentence to run trace, and an explicit, honest boundary of what can be enforced (Table 3: 37% of mandates lower to code gates). The paper also reports cost outliers and the residual 14% missed mandates, which is useful calibration. The main risk is measurement: all headline AMC numbers are fractions of author-distilled reference procedures, and the evaluation pipeline does not yet demonstrate that those procedures are independent of the compiler's own capabilities.","major_comments":[{"comment":"The AMC denominator and every headline number ('56% prose, 86% harness, 2.3× completion') depend on per-skill reference procedures of mandatory steps, but the manuscript does not state who constructed these procedures, under what instructions, or whether the distillers had access to the compiled harnesses, STRUCT-COV reports, or the study hypothesis. Four of the 30 skills were authored by the authors for this study (§5.1). Without an independent, blinded distillation protocol and inter-annotator agreement, the reported 30-point gap may partly reflect alignment between the reference procedures and what SIGIL can enforce. This is a ground-truth measurement threat, not resolved by the §5.7 gate-credit ablation, which addresses scoring asymmetry but not reference-procedure construction. I ask the authors to provide the annotation protocol, release the reference procedures, and report inter-a","section":"§5.2, Table 3"},{"comment":"AMC statuses are assigned by a model-based judge with no reported reliability. For the harness arm, the judge reads the node-path trace produced by the harness's own runtime; for the prose arm, it reads the token stream. The gate-credit robustness check ('77% vs 56% without automatic gate credit') is reassuring, but the prose arm's 'followed' statuses are still model judgments, and the harness's trace is the system-under-test's log. Please report human agreement on a sample (e.g., Cohen's kappa or a confusion matrix by status), and consider a blinded judge that is not told which arm a run came from.","section":"§5.7 / §5.2"},{"comment":"The claim that the guarantee is 'model-independent' is stronger than the evidence supports. The result is demonstrated on two models from a single provider (gpt-4o and gpt-5) at default sampling settings. Two points from one model family establish invariance across those two generations, not model-independence in general. The abstract and conclusion should be narrowed to 'invariant across the two OpenAI generations tested' or the evaluation extended to at least one additional model family/provider.","section":"Abstract / §5.4 / RQ2"},{"comment":"The cost analysis is reported 'Across the 32 skills for which both arms produced token accounting,' and the caption repeats '24 of 32,' but the skill suite is defined as 30 skills in §5.1. This internal inconsistency affects the RQ3 headline result (median 0.58× tokens). Please reconcile the count: if it is a typo, correct to 30; if token accounting covers a different or multiplerun set, state exactly what the 32 refers to.","section":"§5.5, Figure 7"}],"minor_comments":[{"comment":"The compile-gate table lists G1, G3, G4, G5, and G6, but no G2. If a gate was removed, renumber the table; if G2 exists, it should be described. As written, the 'six gates' claim in §4.1 is not directly checkable from the table.","section":"Table 2"},{"comment":"The RQ1 takeaway 'no skill is worse off' conflicts with the gpt-5 result in §5.4, where the harness loses on four skills. The 'no losses' statement appears to be specific to gpt-4o; please state that scope in the takeaway.","section":"§5.3 / §5.4"},{"comment":"The phrase 'each skill's prose is distilled into a reference procedure' should include an explicit statement that the distillation was fixed before any scoring was performed and, ideally, that the reference procedures are released as supplementary material. This would allow readers to audit the mandate lists independently.","section":"§5.2"},{"comment":"The definitions of the AG-IR node alphabet and the 'Owner test' are clear in prose, but a compact worked example showing two sentences from a skill mapped to Mind/Code, with their modality and cohesion attributes, would make the representation easier to reproduce.","section":"§3.2"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Jayanaka—quick read of arXiv:2607.27309. The short version: the paper is worth engaging seriously. The central idea—compile a prose skill into an executable harness via a typed IR, separating model-owned cognition from code-owned mechanism—is genuinely new as a layered system, and the implementation is substantial. The mechanism is transparent: code-owned abilities fire on node entry, so steps that are determined by their inputs execute because they are program structure, not because the model re-derived them. The authors also know their own limits: the threats section concedes the gate-credit objection and recomputes compliance with no automatic gate credit (77% vs 56%), which still shows a large effect. That honesty is real, and the STRUCT-COV diagnostic and provenance spine are good engineering.\n\nWhat I'd push back on is the measurement apparatus. Every headline number—86%, 56%, 2.3×—is a fraction of a reference procedure of mandated steps that the authors distilled from each skill. The paper never says who did that distillation, under what instructions, or whether distillers saw the compiled harnesses or STRUCT-COV reports. Four of the thirty skills were authored by the authors for this study. If the reference procedure was shaped, even implicitly, by what SIGIL can enforce, then the harness is scored against a target it was built to hit while prose is scored against the same possibly skewed target. The 77%-vs-56% gate-credit check addresses one scoring asymmetry but does not validate the reference itself. On top of that, the judge is a model with no reported reliability, and there is no inter-annotator check. These are not fatal to the core claim—the direction of the effect is mechanistically defensible and the paper's own numbers show the benefit is bounded (only 37% of mandates lower to code gates, 14% still missed in compiled runs)—but they mean the strength of the headline claim outruns the evidence.\n\nMinor mechanical issues: §5.5 says 32 skills where the analysis had 30, and the gate table lists six gates but enumerates G1, G3–G6 with no G2. Small, but sloppy.\n\nWho should read it: people working on LLM-agent reliability, skill authoring, and runtime supervision. It reframes the problem from 'watch the agent' to 'compile the procedure,' and that's worth a reading group. For peer review: yes, send it out, but ask for artifact release (compiler, traces, reference-procedure definitions), an inter-annotator study on the mandate lists, and judge-reliability numbers. The central claim will probably survive those checks—I'd bet on it—but the paper needs to give readers the tools to verify.\n\nRecommended action: serious referee, conditional acceptance path.","headline":"SIGIL is a genuine systems contribution—prose-to-harness compilation with an honest evaluation—but the headline gap rests on author-distilled ground truth, so treat it as 'substantially improved' rather than 'guaranteed'.","tokens_in":15997,"tokens_out":2312,"duration_ms":19168},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Compiling prose skills into programs lifts agent mandate compliance to 86%.","keywords":["AI agents","skill compilation","agent harnesses","intermediate representation","procedure compliance","LLM reliability","typed model calls","mandate execution"],"falsifier":"Take one of the 30 skills, have two independent annotators (blind to the compiled harness) each write the reference procedure of mandatory steps, then re-score all runs against both references. If the compiled harness's margin over prose shrinks substantially under an independent reference, or if a trace shows a code-gated step failing to execute when its inputs are present, the central claim is weakened; the paper itself reports 14% of mandates still missed in compiled runs, so such a case is concretely searchable.","tokens_in":14817,"feed_emoji":"⚙️","tokens_out":4361,"duration_ms":34073,"temperature":0.7,"pith_summary":"The paper claims that the standard way of giving AI agents skills—loading a natural-language procedure file into the model's context—loses the procedure: agents re-derive control flow on every run and skip a large share of steps the skill itself mandates. Across 30 skills, prose agents perform only 56% of mandated steps on a mid-tier model, even while producing plausible artifacts. The authors propose treating the skill as source code: compile it into an executable harness in which mandated steps are program structure, not prose advice. Compiled harnesses perform 86% of mandated steps, complete the full procedure 2.3 times as often, and use 58% of the tokens, with the compliance rate unchanged at 86% when the underlying model changes.","feed_headline":"Compiling prose skills lifts agent mandate compliance to 86%","feed_subtitle":"On 30 skills, harness-encoded procedures beat prose re-derivation and hold steady across two model generations.","key_machinery":"AG-IR is a typed graph over four primitive families—Mind, Flow, Boundary, Code—with each node assigned an owner and a modality. The load-bearing rule is the Owner test: if a step's output is a function of its inputs, code owns it and it lowers to unconditional structure; otherwise the model owns it as a typed slot. Modality becomes structure: mandatory rules become abilities that fire on node entry, forbidden rules become absent paths, and discretionary rules become typed verdicts consumed by surrounding code. Extraction is constrained by grounding every rule to a verbatim quote, six compile gates, and a deontic audit; Lowering is deterministic and contains no model call.","core_discovery":"The central claim is that a mandated step executes reliably when it is represented in program structure rather than when a model correctly interprets a prose instruction. To make that possible from a prose source, the paper introduces AG-IR, a typed intermediate representation that labels every node with an owner (code or model) and a modality (mandatory, forbidden, discretionary, etc.), plus provenance back to the sentence that mandated it. A deterministic lowering stage transpiles AG-IR into an executable agent harness, so code-owned steps cannot be skipped, reordered, or retargeted; only genuinely open-ended 'judgment' steps remain model-owned typed slots. The empirical claim is that this","pith_inferences":["An independent annotation study that varies the reference procedures of mandatory steps would reveal how much of the 30-point gap is compilation and how much is a scoring convention tied to the authors' distillation choices.","The flat 86% across two model generations suggests compilation could serve as capability insurance: organizations can lock in procedure adherence without waiting for stronger models, and the benefit should grow on small on-device models.","The Owner test is a determinism criterion that could be applied to arbitrary agent traces to identify which steps are safe to compile in other languages and runtimes, independent of the specific implementation here.","If the structural guarantee is real, a harness-compiled skill should be auditable post-hoc: every code-gated step leaves a node-trace record, so a run can be checked against the procedure without trusting the model's narration."],"forward_implications":["Agent skill execution moves from a per-run act of reconstruction to a fixed graph, so compliance stops depending on the model's ability to parse and obey the prose.","Teams using cheaper or smaller models gain the most: the gap is 30 points on the weaker model and 17 on the stronger one.","Reliability does not have to be paid for at the median: compiled harnesses use 0.58x the tokens of prose, though adaptive tool-use skills can cost more because the harness actually runs the loop the prose agent skips.","Artifact-level output checks are insufficient as a correctness oracle; procedure execution itself is the metric, since skipped steps can leave plausible outputs.","The approach's limit is explicit: only steps whose outcome is determined by their inputs can become code gates (37% of mandates in this suite), and judgment-heavy skills retain a model-owned residue."],"fun_headline_variants":["Compiling skills into harnesses boosts compliance to 86%","Harness-compiled skills hit 86% mandate adherence","From prose to program: 86% step compliance","Type-checked skill harnesses beat prose at 86%","Skill compilation lifts mandate execution to 86%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Every headline number is scored against a reference procedure of 'must' steps that the authors distilled from each skill's prose; if that distillation were different—or shaped by what the compiler can enforce—both arms' scores would shift.","fun_headline_variants_meta":{"raw":{"variants":["Compiling skills into harnesses boosts compliance to 86%","Harness-compiled skills hit 86% mandate adherence","From prose to program: 86% step compliance","Type-checked skill harnesses beat prose at 86%","Skill compilation lifts mandate execution to 86%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000519,"raw_usage":{"total_tokens":2345,"prompt_tokens":731,"completion_tokens":1614,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":475,"completion_tokens_details":{"reasoning_tokens":1534}},"tokens_in":475,"tokens_out":1614,"duration_ms":9713,"temperature":1.0,"reasoning_tokens":1534,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T09:46:53.063439+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.3.0-daily-deepseek","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Take one of the 30 skills, have two independent annotators (blind to the compiled harness) each write the reference procedure of mandatory steps, then re-score all runs against both references. If the compiled harness's margin over prose shrinks substantially under an independent reference, or if a trace shows a code-gated step failing to execute when its inputs are present, the central claim is weakened; the paper itself reports 14% of mandates still missed in compiled runs, so such a case is concretely searchable.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}