Software Engineering · arXiv:2607.27309 v3 · submitted 29 Jul 2026
SIGIL: Skill Compilation for Reliable and Efficient Agent Execution
The paper says
“Agent skills describe reusable procedures, but runtime models must still interpret their instructions and coordinate execution. We introduce Skill Compilation, which translates the procedure prescribed by an authored skill into an executable harness while preserving decisions left to the model.”
more of the abstract
Our compiler, SIGIL, translates skills and their resources into a typed intermediate representation, validates it, and deterministically generates the harness. We evaluate SIGIL on 11 compliance-critical skills, where following the prescribed procedure is part of correctness. SIGIL improves adherence across all four runtime models, achieving up to 100% measured mean Skill Adherence compared with 26.2-54.0% for direct SKILL.md execution. On this suite, SIGIL reduces total runtime tokens by 21-45% for three of the four models. On SkillsBench, SIGIL also improves task performance across all four models, with absolute gains of 3.5-35.7 percentage points. These findings suggest that compiling reusable skill procedures into executable harnesses can improve adherence, task completion, and runtime efficiency.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 1 Aug 2026 · of version 1 of the paper; version 3 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. SIGIL is a genuine systems contribution—prose-to-harness compilation with an honest evaluation—but the headline gap rests on author-distilled ground truth, so treat it as 'substantially improved' rather than 'guaranteed'. This line is wrong
4 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at §5.2, Table 3
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
The AMC denominator and every headline number ('56% prose, 86% harness, 2.3× completion') depend on per-skill reference procedures of mandatory steps, but the manuscript does not state who constructed these procedures, under what instructions, or whether the distillers had access to the compiled harnesses, STRUCT-COV reports, or the study hypothesis.
more of objection 1
Four of the 30 skills were authored by the authors for this study (§5.1). Without an independent, blinded distillation protocol and inter-annotator agreement, the reported 30-point gap may partly reflect alignment between the reference procedures and what SIGIL can enforce. This is a ground-truth measurement threat, not resolved by the §5.7 gate-credit ablation, which addresses scoring asymmetry but not reference-procedure construction. I ask the authors to provide the annotation protocol, release the reference procedures, and report inter-a
The paper says, at §5.7 / §5.2
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
AMC statuses are assigned by a model-based judge with no reported reliability. For the harness arm, the judge reads the node-path trace produced by the harness's own runtime; for the prose arm, it reads the token stream. The gate-credit robustness check ('77% vs 56% without automatic gate credit') is reassuring, but the prose arm's 'followed' statuses are still model judgments, and the harness's trace is the system-under-test's log.
more of objection 2
Please report human agreement on a sample (e.g., Cohen's kappa or a confusion matrix by status), and consider a blinded judge that is not told which arm a run came from.
The paper says, at Abstract / §5.4 / RQ2
The review did not quote this passage.
Pith says: Objection 3 of 4 · major
The claim that the guarantee is 'model-independent' is stronger than the evidence supports. The result is demonstrated on two models from a single provider (gpt-4o and gpt-5) at default sampling settings. Two points from one model family establish invariance across those two generations, not model-independence in general. The abstract and conclusion should be narrowed to 'invariant across the two OpenAI generations tested' or the evaluation extended to at least one additional model family/provider.
The paper says, at §5.5, Figure 7
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The cost analysis is reported 'Across the 32 skills for which both arms produced token accounting,' and the caption repeats '24 of 32,' but the skill suite is defined as 30 skills in §5.1. This internal inconsistency affects the RQ3 headline result (median 0.58× tokens). Please reconcile the count: if it is a typo, correct to 30; if token accounting covers a different or multiplerun set, state exactly what the 32 refers to.
4 minor objections
The paper says, at Table 2
Pith says: minor objection 1 of 4
The compile-gate table lists G1, G3, G4, G5, and G6, but no G2. If a gate was removed, renumber the table; if G2 exists, it should be described. As written, the 'six gates' claim in §4.1 is not directly checkable from the table.
The paper says, at §5.3 / §5.4
Pith says: minor objection 2 of 4
The RQ1 takeaway 'no skill is worse off' conflicts with the gpt-5 result in §5.4, where the harness loses on four skills. The 'no losses' statement appears to be specific to gpt-4o; please state that scope in the takeaway.
show 2 more
The paper says, at §5.2
Pith says: minor objection 3 of 4
The phrase 'each skill's prose is distilled into a reference procedure' should include an explicit statement that the distillation was fixed before any scoring was performed and, ideally, that the reference procedures are released as supplementary material. This would allow readers to audit the mandate lists independently.
The paper says, at §3.2
Pith says: minor objection 4 of 4
The definitions of the AG-IR node alphabet and the 'Owner test' are clear in prose, but a compact worked example showing two sentences from a skill mapped to Mind/Code, with their modality and cohesion attributes, would make the representation easier to reproduce.
What would settle it
Recruit two annotators with no connection to the authors; give them the 30 SKILL.md files plus a written rubric for identifying MUST steps, with examples from unrelated skills. Ask them to independently produce reference procedures, blinded to the compiled artifacts and to the paper's results. Compute inter-annotator agreement (e.g., Cohen's kappa) on the mandate lists. Then score the existing runs (or a random subset of at least 10 skills) against each independent reference procedure using the same four-way judge protocol, and recompute the prose vs harness AMC gap. If the gap remains ≥20 points and kappa ≥0.6, the concern is resolved; if the gap drops below 10 points or agreement is low, the central claim is materially weakened.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 1 Aug 2026
- Reader holds up, with conditions
- Skeptic agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Jayanaka—quick read of arXiv:2607.27309. The short version: the paper is worth engaging seriously. The central idea—compile a prose skill into an executable harness via a typed IR, separating model-owned cognition from code-owned mechanism—is genuinely new as a layered system, and the implementation is substantial. The mechanism is transparent: code-owned abilities fire on node entry, so steps that are determined by their inputs execute because they are program structure, not because the model re-derived them. The authors also know their own limits: the threats section concedes the gate-credit objection and recomputes compliance with no automatic gate credit (77% vs 56%), which still shows a large effect. That honesty is real, and the STRUCT-COV diagnostic and provenance spine are good engineering.
What I'd push back on is the measurement apparatus. Every headline number—86%, 56%, 2.3×—is a fraction of a reference procedure of mandated steps that the authors distilled from each skill. The paper never says who did that distillation, under what instructions, or whether distillers saw the compiled harnesses or STRUCT-COV reports. Four of the thirty skills were authored by the authors for this study. If the reference procedure was shaped, even implicitly, by what SIGIL can enforce, then the harness is scored against a target it was built to hit while prose is scored against the same possibly skewed target. The 77%-vs-56% gate-credit check addresses one scoring asymmetry but does not validate the reference itself. On top of that, the judge is a model with no reported reliability, and there is no inter-annotator check. These are not fatal to the core claim—the direction of the effect is mechanistically defensible and the paper's own numbers show the benefit is bounded (only 37% of mandates lower to code gates, 14% still missed in compiled runs)—but they mean the strength of the headline claim outruns the evidence.
Minor mechanical issues: §5.5 says 32 skills where the analysis had 30, and the gate table lists six gates but enumerates G1, G3–G6 with no G2. Small, but sloppy.
Who should read it: people working on LLM-agent reliability, skill authoring, and runtime supervision. It reframes the problem from 'watch the agent' to 'compile the procedure,' and that's worth a reading group. For peer review: yes, send it out, but ask for artifact release (compiler, traces, reference-procedure definitions), an inter-annotator study on the mandate lists, and judge-reliability numbers. The central claim will probably survive those checks—I'd bet on it—but the paper needs to give readers the tools to verify.
Recommended action: serious referee, conditional acceptance path.
The strongest objection
Every headline number—56% prose, 86% harness, 2.3× completion—is a fraction of a reference procedure of 'mandatory steps' that the authors distilled from each SKILL.md (§5.2). The paper never states who did this distillation, under what instructions, or whether the distillers saw the compiled harnesses, STRUCT-COV reports, or the study hypothesis. Four of the 30 skills were authored by the authors for this study (§5.1), and the compiler's own Extraction is a similar model-driven reading of the same prose. If the reference procedure was shaped—even implicitly—by what SIGIL can enforce (e.g., by aligning the mandate list with the code gates that STRUCT-COV reports, or by phrasing mandates in the deontic vocabulary the extractor consumes), then the harness is scored against a target it was built to hit, while prose is scored against the same possibly skewed target. The direction of bias is predictable: including only mechanically checkable mandates, or wording them as explicit tool invocations, advantages the harness. The §5.7 gate-credit check (77% vs 56% without automatic gate credit) addresses one scoring asymmetry but does not validate the ground-truth reference itself. A model-based judge with no reported reliability further compounds this. This is a ground-truth measurement threat, not a claim of fraud; it is the condition least secured by the paper.
The reader's weakest_assumption precisely identifies the load-bearing concern: the reference procedures are the denominator for every AMC number, yet their construction is unreported and may be confounded with the compiler. This is not a minor detail; it determines whether the 30-point gap is a real effect of compilation or an artifact of how 'mandated steps' were defined. The paper's own robustness check (§5.7) only addresses gate credit, not the reference procedure itself, and the four self-authored skills heighten the risk. No other concern is as central: the internal inconsistencies (32 vs 30 skills, missing G2) are cosmetic, and the two-model capability axis is explicitly qualified. The correct verdict remains CONDITIONAL: the central claim is plausible and well-mechanized, but the measurement apparatus must be exposed and independently validated before the numbers can be taken at face value. The proposed test directly settles whether this concern lands.
The referee's report
If the empirical claims hold, this is a valuable contribution to dependable AI-agent execution: it targets the actual performance of mandated procedural steps rather than artifact-level output checks, and it shows that compiled structure, rather than model capability, can carry procedure compliance. The design has several concrete strengths: deterministic lowering, compile gates such as G4 (the compile oracle) and G5 (STRUCT-COV), provenance from skill sentence to run trace, and an explicit, honest boundary of what can be enforced (Table 3: 37% of mandates lower to code gates). The paper also reports cost outliers and the residual 14% missed mandates, which is useful calibration. The main risk is measurement: all headline AMC numbers are fractions of author-distilled reference procedures, and the evaluation pipeline does not yet demonstrate that those procedures are independent of the compiler's own capabilities.
Pith's reading of the paper (a machine's summary, not the authors')
The paper claims that the standard way of giving AI agents skills—loading a natural-language procedure file into the model's context—loses the procedure: agents re-derive control flow on every run and skip a large share of steps the skill itself mandates. Across 30 skills, prose agents perform only 56% of mandated steps on a mid-tier model, even while producing plausible artifacts. The authors propose treating the skill as source code: compile it into an executable harness in which mandated steps are program structure, not prose advice. Compiled harnesses perform 86% of mandated steps, complete the full procedure 2.3 times as often, and use 58% of the tokens, with the compliance rate unchanged at 86% when the underlying model changes.
What carries the argument. AG-IR is a typed graph over four primitive families—Mind, Flow, Boundary, Code—with each node assigned an owner and a modality. The load-bearing rule is the Owner test: if a step's output is a function of its inputs, code owns it and it lowers to unconditional structure; otherwise the model owns it as a typed slot. Modality becomes structure: mandatory rules become abilities that fire on node entry, forbidden rules become absent paths, and discretionary rules become typed verdicts consumed by surrounding code. Extraction is constrained by grounding every rule to a verbatim quote, six compile gates, and a deontic audit; Lowering is deterministic and contains no model call.
Core claim. The central claim is that a mandated step executes reliably when it is represented in program structure rather than when a model correctly interprets a prose instruction. To make that possible from a prose source, the paper introduces AG-IR, a typed intermediate representation that labels every node with an owner (code or model) and a modality (mandatory, forbidden, discretionary, etc.), plus provenance back to the sentence that mandated it. A deterministic lowering stage transpiles AG-IR into an executable agent harness, so code-owned steps cannot be skipped, reordered, or retargeted; only genuinely open-ended 'judgment' steps remain model-owned typed slots. The empirical claim is that this
The premise it rests on. Every headline number is scored against a reference procedure of 'must' steps that the authors distilled from each skill's prose; if that distillation were different—or shaped by what the compiler can enforce—both arms' scores would shift.
If the paper is right (Pith's inferences, not the authors')
Agent skill execution moves from a per-run act of reconstruction to a fixed graph, so compliance stops depending on the model's ability to parse and obey the prose.
Teams using cheaper or smaller models gain the most: the gap is 30 points on the weaker model and 17 on the stronger one.
Reliability does not have to be paid for at the median: compiled harnesses use 0.58x the tokens of prose, though adaptive tool-use skills can cost more because the harness actually runs the loop the prose agent skips.
Artifact-level output checks are insufficient as a correctness oracle; procedure execution itself is the metric, since skipped steps can leave plausible outputs.
The approach's limit is explicit: only steps whose outcome is determined by their inputs can become code gates (37% of mandates in this suite), and judgment-heavy skills retain a model-owned residue.
An independent annotation study that varies the reference procedures of mandatory steps would reveal how much of the 30-point gap is compilation and how much is a scoring convention tied to the authors' distillation choices.
The flat 86% across two model generations suggests compilation could serve as capability insurance: organizations can lock in procedure adherence without waiting for stronger models, and the benefit should grow on small on-device models.
The Owner test is a determinism criterion that could be applied to arbitrary agent traces to identify which steps are safe to compile in other languages and runtimes, independent of the specific implementation here.
If the structural guarantee is real, a harness-compiled skill should be auditable post-hoc: every code-gated step leaves a node-trace record, so a run can be checked against the procedure without trusting the model's narration.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent skills describe reusable procedures, but runtime models must still interpret their instructions and coordinate execution. We introduce Skill Compilation, which translates the procedure prescribed by an authored skill into an executable harness while preserving decisions left to the model. Our compiler, SIGIL, translates skills and their resources into a typed intermediate representation, validates it, and deterministically generates the harness. We evaluate SIGIL on 11 compliance-critical skills, where following the prescribed procedure is part of correctness. SIGIL improves adherence across all four runtime models, achieving up to 100% measured mean Skill Adherence compared with 26.2-54.0% for direct SKILL.md execution. On this suite, SIGIL reduces total runtime tokens by 21-45% for three of the four models. On SkillsBench, SIGIL also improves task performance across all four models, with absolute gains of 3.5-35.7 percentage points. These findings suggest that compiling reusable skill procedures into executable harnesses can improve adherence, task completion, and runtime efficiency.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-23
The authors' main claim, as the classifier picked it out 1.00
“These results show that compiling procedural structure improves the reliability and efficiency of skill execution while retaining model judgment where it is needed.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.23 · human participants 0.05 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.95 · mentions code or data availability 0.06
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent skills provide a reusable way to specify multi-step agent behavior, but they remain natural-language specifications interpreted by the model at runtime. | background 0.99 | 0.08 | 0.03 | asserted 1.00 |
| As a result, required tool calls, ordering constraints, and checks may be skipped even when explicitly prescribed by the skill. | background 0.80 | 0.09 | 0.02 | hedged possibility 0.97 |
| We introduce Skill Compilation, a paradigm that translates natural-language skills into executable agent programs while preserving model judgment where semantic decisions are required. | contribution 0.99 | 0.81 | 0.13 | asserted 0.99 |
| We realize this idea in SIGIL. | contribution 0.96 | 0.62 | 0.02 | asserted 0.84 |
| SIGIL extracts source-grounded requirements, decomposes them using a closed Agent Instruction Set (AIS), composes them into AG-IR with explicit ownership, data flow, and control flow, and deterministically lowers validated AG-IR into executable code. | method 0.99 | 0.89 | 0.05 | asserted 0.98 |
| Across 33 publicly available SKILL.md files and three runtime models, SIGIL increases mean Applicable-Mandate Compliance (AMC), the fraction of applicable skill requirements satisfied during execution, from 66.0% with direct skill execution to 88.6%. | result 1.00 | 0.96 | 0.16 | asserted 1.00 |
| SIGIL also reduces total runtime token consumption by 2.40-5.95x. | result 1.00 | 0.95 | 0.06 | asserted 1.00 |
| These results show that compiling procedural structure improves the reliability and efficiency of skill execution while retaining model judgment where it is needed. | interpretation 1.00 | 0.91 | 0.93 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v2
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 6,249 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (8)
Showing 7 of the paper's 8 figures. Figures are the authors', reproduced from the paper.
Cited by 4 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- Guarding the Kingdom of Skill-Based Agents in the AI Dynasty of Computation: Control-Flow Integrity for Agent Skills cs.CR · Nov 2026
- Agent-Native by Design: A Programming Language for Agentic Applications cs.PL · Nov 2026
- Versioned Transitive Dependency-Closure Binding and Operation-Time Effect Governance for Agent Skills: ClosureBound cs.CR · Sep 2026
- Mining Agent Skills from Production Traces cs.AI · Oct 2026
How the literature uses it
What this paper does with the arXiv papers it cites
What it cites · 1 sentence without a confident label
Uses the method or tool of 1 paper
-
Each AG-IR primitive has a fixed translation into an Object-Spatial Programming (OSP) [ref] agent.
Object-Spatial Programming · Introduction · 0.90
Cites as background 1 paper
-
Prior work generally retains the prose execution model and adds runtime supervision, including formal monitors [ref] , action-boundary guards [ref] , and rule languages evaluated during each run [ref] .
Formal Methods Meet LLMs: Auditing, Monitoring, and Intervention for Compliance of Advanced AI Systems · Introduction · 0.95
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment