Artificial Intelligence · arXiv:2609.30123 v2 · submitted 24 Sep 2026
HEXIS: Compiling Agent Skills into Extended Finite State Machines
The paper says
“Agent skills provide instructions and reusable knowledge, yet agents have to frequently infer which action to take next. This skill's execution paradigm combines task reasoning with control decisions, which could cause the agent to act improperly or omit necessary steps.”
more of the abstract
To improve agent compliance for skills, we introduce HEXIS, which compiles agent skills into extended finite state machines (FSM) that separate knowledge from control flow. Skill knowledge is incorporated into local instructions that guide reasoning and generation within states. The machine records execution progress and intermediate results, while explicit transition conditions determine subsequent operations. Our incremental compiler first maps skill clauses and tool interfaces to state operations, local instructions, data bindings, and transitions. It then aligns development traces with existing states to identify missing operations and dependencies. These are incorporated by adding or reusing states and refining their connections. Updates are accepted only after static checks and replay of the current and all previously accepted traces. Across four benchmarks and four executors, HEXIS improves success over Skill + ReAct by 16.2 percentage points on average. Qwen3.8-27B reduces execution tokens by 38.4-88.9% across benchmarks.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 25 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up, with conditions. A solid engineering contribution with a genuinely new regression-style skill compiler, held back mainly by the absence of any uncertainty reporting on its headline numbers. This line is wrong
4 major objections · 5 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Table 1, §5.3
The review did not quote this passage.
Pith says: Objection 1 of 4 · major
Every number in Table 1 is a single run with no confidence intervals, standard errors, or significance tests, yet the central claim (b9, b166) is quantitative. With binary success outcomes and test sizes of n=51 (DABench, LongSeal), n=57 (SpreadsheetBench), and n=121 (LiveMath), the standard error per cell is roughly 5-7 pp.
more of objection 1
Several reported HEXIS-vs-Skill+ReAct margins are within 1-3 task flips of zero: DABench margins of 4.0, 4.0, 3.9, and 1.9 pp correspond to 1-2 extra successes on n=51, and the LongSeal margin of 5.9 pp on Qwen3.8-27B is 3 of 51 tasks, with an exact tie on Qwen3.5-9B. The 16.1-pp average is dominated by LiveMath (~34 pp) and SpreadsheetBench (~19 pp), with DABench contributing only ~3.5 pp. Baseline instability (AFlow on DABench: 80.4% with qwen3.6-flash vs 15.7% with Qwen3.8-27B; ReasoningBank on LiveMath: 8.3-41.3% across executors) shows the noise floor. I request per-cell binomial confidence intervals or significance tests and, where feasible, multiple seeds; the '15 of 16 settings' and 16.1-pp claims should be re-stated with uncertainty.
The paper says, at Eq. (10), §5.2; Appendix F
The review did not quote this passage.
Pith says: Objection 2 of 4 · major
The relationship between Table 1 'success' and the Eq. (10) metric is not clear, and Appendix F concedes a crack in what the machine actually certifies. The metric section states that checks cover 'file delivery, output format, and tool-use requirements independently of answer correctness,' while Table 1 reports 'Success rates.'
more of objection 2
Appendix F states that the terminal category 'verified' only certifies that the delivered file agrees with the machine's selected answer letter, 'not that the mathematical answer is correct.' For LiveMath in particular, the headline 31.4-38.0 pp gains may thus measure procedural compliance (producing a file containing the model's selected letter) rather than selecting the correct answer. The paper should state, per benchmark, what Table 1 measures and how it relates to answer correctness, and should temper the interpretation of the gains if they are compliance gains rather than correctness gains.
The paper says, at Appendix C, §C.2-C.3
The review did not quote this passage.
A classifier read this sentence of the paper as acknowledging the concern (probability 0.77):
“Semantic preservation is an assumption of this analysis; the implemented checks and replay directly maintain the replayability of accepted event paths.”
Pith says: Objection 3 of 4 · major
The convergence analysis depends on assumptions that presuppose the conclusion. The assumption that 'updates preserve semantic distinctions already represented' via a deterministic mapping g_n with R_n = g_n(R_{n+1}) (b435, b436), and the assumption that each surviving incorrect interpretation is eliminated with probability at least gamma > 0 each round (b444), are exactly the properties the appendix aims to derive.
more of objection 3
The authors honestly label these as assumptions (b440: 'Semantic preservation is an assumption of this analysis'), and the implementation only guarantees replayability of accepted event paths (b409), not inter-round semantic preservation or a fixed elimination probability. As a result, Eq. (C.4)'s claim that expected representation loss tends to zero is conditional on unverified idealization. I recommend reframing Appendix C explicitly as a formalization of ideal conditions under which accumulation would converge, with the assumptions stated as open conditions for the algorithm to satisfy, rather than presenting convergence as an established property of HEXIS.
The paper says, at §5.2, Appendix H, §5.3; Fig. 5
The review did not quote this passage.
Pith says: Objection 4 of 4 · major
The paper's value proposition includes cost, but the compilation and development cost is never reported. The machines are compiled by Claude Fable 5.1 (a frontier commercial model) over all qwen3.6-flash development trajectories, including 487 LiveMath tasks across 49 groups (Appendix H), with a retry limit on candidates and acceptance only after replay of the full archive.
more of objection 4
The reported token figures (Fig. 5) are execution tokens only, and the paper's efficiency claims (e.g., 38.4-88.9% reduction for Qwen3.8-27B; Table 2's 69k tokens) omit the compile-time inference cost, which can be substantial given the trace volume and the replay validation. Since 'lower inference cost' is part of the stated motivation, I request a reported compile cost (model calls, tokens, or wall-clock per skill) and a statement of whether the cost amortizes over the reusability claim.
5 minor objections
The paper says, at §5.1
Pith says: minor objection 1 of 5
The DABench split is described as 'stratified by difficulty' and LiveMath as 'stratified by month,' but the stratification procedure and the number of strata are not specified; a sentence describing how stratification was done would aid reproducibility.
The paper says, at §5.2
Pith says: minor objection 2 of 5
The baseline list spells the method 'A WM' (agent workflow memory) while Table 1 and Figure 4 use 'AWM'; the spelling should be made consistent.
show 3 more
The paper says, at Eq. (5), §3.2
Pith says: minor objection 3 of 5
The representational objective in Eq. (5) is presented as a minimization over M, but no optimization over M is actually performed; the compilation procedure is heuristic (draft, align, replay, commit). The paper notes this in passing, but a sentence stating that Eq. (5) is motivational and not optimized explicitly would prevent a misreading.
The paper says, at §5.2; Table 1
Pith says: minor objection 4 of 5
Table 1 does not report the number of runs per cell, and the paper nowhere states that all experiments were single-run; a note in the metric section confirming single-run reporting and the absence of variance would make the statistics transparent.
The paper says, at Fig. 4, Fig. 5
Pith says: minor objection 5 of 5
Figure 4 and Figure 5 each use different axis scales across panels, and the token figures are reported in thousands (k); the captions should state the units explicitly in the text, not only in the axis labels.
What would settle it
Regenerate Table 1 from per-task paired outcomes and report, for every cell, a McNemar test (or bootstrap 95% CI over tasks) comparing HEXIS vs Skill + ReAct, plus a bootstrap CI for the aggregate 16.1-pp average across the 16 cells (test sets: 51, 51, 57, 121). If DABench (margins 1.9-4.0 pp) and LongSeal (0-13.8 pp) gains fail to exclude zero and the 95% CI of the average does not exclude a ~6-8 pp floor, then the headline must be re-framed as 'consistently non-worse with significant gains on LiveMath and SpreadsheetBench,' and the '15 of 16' count should be replaced by a significance-filtered count.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 25 Sep 2026
- Reader holds up, with conditions
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 4 major objections, 5 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
Quick take: this is a genuinely useful engineering paper, and the central mechanism claim holds up better than the headline-number weakness suggests. The trace-guided incremental compiler with regression-style acceptance — an update commits only if static checks pass and the new trace plus every previously accepted trace replays — is a real contribution I don't see in the cited prior work. StateFlow uses predefined task-domain machines, TraceCompiler derives programs without skill-text initialization, Formal Skill needs hand-written programs. The evaluation is also more honestly grounded than much of this subfield: machines are compiled on qwen3.6-flash development traces and transferred unchanged to three other executors, which gives the cross-backbone claim independent grounding. And the Prompt-only ablation in Appendix G is the right kind of control — identical machine content rendered as a document, no programmatic enforcement — so the gains on GLM and Qwen3.8-27B with 28–94% fewer tokens aren't a compute-budget artifact.
The paper also states its limits plainly. b197 concedes that static checks and trace replay validate recorded paths but not in-state reasoning; Appendix F says "verified" only means the file read-back matches the selected letter, not that the math is right. Acknowledged limits, not hidden flaws.
Now the soft spots, in proportion. The stress-test concern lands: the 16.1-point average is stated to one decimal with no variance or significance anywhere, on test sets of 51–121 tasks. Binary outcomes on n≈51 give standard errors of 5–7 points. DABench margins are 1.9–4.0 points — one or two flipped tasks each — and LongSeal has an exact tie on Qwen3.5-9B. The baselines' own swings (AFlow on DABench goes from 80.4% to 15.7% just by swapping executors) show a noise floor wider than several claimed margins. The headline magnitude isn't supported at the stated precision; the direction is.
Two more omissions worth naming: StateFlow and Formal Skill — the closest related methods — are cited but never run, and compilation cost isn't amortized into the token-savings claims. Both are addressable.
The Appendix C convergence analysis I'd treat as decoration. The assumptions (semantic preservation, every distinguishing trace eliminated with fixed probability γ, traces always consistent with the original skill) guarantee the result, and b440 concedes the implementation only maintains replayability. Honest, but not evidence for the empirical claim.
Bottom line: solid, well-written engineering work with a real mechanism and an honestly held-out evaluation, weak on statistical reporting and with one theory appendix that overreaches. Send it to review with conditions: error bars, the two missing baselines if feasible, amortized compile cost, and a rewrite of Appendix C's claims to match what replay actually guarantees.
The strongest objection
The paper's central claim (b9, b166-b170) is quantitative: HEXIS improves success over Skill + ReAct by 16.1 pp on average and in 15 of 16 settings. No confidence interval, variance, or significance test appears anywhere in Table 1, and the test sets are small: SpreadsheetBench 57, DABench 51, LongSeal 51, LiveMath 121. Reconstructing the cells: DABench HEXIS-vs-S+R margins are 4.0, 4.0, 3.9, and 1.9 pp, i.e., 1-2 extra task successes each on n=51; LongSeal margins are 13.8, 11.8, 0.0 (exact tie on Qwen3.5-9B), and 5.9 pp, the last being 3 of 51 tasks. With binary per-task outcomes and n≈51-57, standard errors are roughly 5-7 pp, so several 'improvement' cells are within 1-2 flipped tasks of the baseline. The 16.1-pp average is dominated by LiveMath (~34 pp) and SpreadsheetBench (~19 pp); DABench contributes only ~3.5 pp. Baseline instability reveals the noise floor: AFlow on DABench is 80.4% with qwen3.6-flash but 15.7% with Qwen3.8-27B (a 64.7-pp swing from swapping only the executor), and ReasoningBank on LiveMath ranges 8.3-41.3% across executors. A handful of outcome flips would shrink '15 of 16' and cut the headline average by several points. The mechanism direction is better defended than the reader's weakest assumption suggests: Appendix G's Prompt-only ablation (identical machine content, no programmatic enforcement) isolates control-flow from content, and the large gains on GLM/Qwen3.8-27B with 28.4-94.5% fewer tokens argue against a compute-budget confound; the judge-label fidelity limit is explicitly conceded (b197, b300, Appendix F: 'verified' means only that the file read-back matches the selected letter). What is unaddressed is magnitude robustness: the claim is stated to one decimal place on data whose noise floor, demonstrated by the baselines' own swings, is several points wider than the margins being asserted.
The reader's CONDITIONAL verdict and weakest assumption (judge-label routing fidelity) capture a real, author-conceded limitation: the machine's guards route on model-written verdicts whose correctness the compiler cannot check (b197, b300, Appendix F), and Appendix G's Prompt-only control plus cross-backbone token reductions partially defend the mechanism claim. My stress-test pass elevates a different, unaddressed issue to primary status: the headline is a precise quantitative superiority claim (16.1 pp; 15 of 16 settings) supported only by single-run point estimates on 51-121-task test sets with no variance or significance reporting. Recomputing per-cell margins shows DABench gains of 1.9-4.0 pp (1-2 successes on n=51) and a LongSeal tie, while the same Table exhibits baseline swings of up to 64.7 pp (AFlow on DABench) from swapping only the executor, demonstrating that the noise floor is several points wide. The 16.1-pp average therefore overstates what the data can bear; the contribution is better characterized as positive-but-noisy, with the magnitude concentrated in LiveMath and SpreadsheetBench. This is an addressable omission rather than a fatal flaw, so the verdict remains CONDITIONAL (UNCHANGED), but the conditions the reader should impose must now include uncertainty reporting for every Table 1 cell and for the aggregate average, not only code release and amortized compilation cost. I agree with the reader's identification of judge-label fidelity as a genuine and load-bearing mechanism concern; I disagree only that it is the single most load-bearing one, since it is explicitly conceded by the authors and partially controlled, whereas the statistical fragility of the quantified central claim is neither conceded nor controlled.
The referee's report
If the central empirical claim holds, the contribution is real and useful: HEXIS provides a mechanism for turning free-text skills into executable structure that is reusable across backbones without target-model recompilation, with evidence of reduced inference cost on some executors. The design has genuine strengths that I want to credit explicitly. The evaluation uses held-out test tasks with machines compiled only on development traces, so the reported numbers are not a fitted reduction on the test set. Cross-model transfer (11 of 12 target-model settings) is independent grounding for the representation's generality, because the machines were never adapted to the target executors. The prompt-only ablation in Appendix G directly isolates programmatic enforcement from content changes, and the compiler-sensitivity check in Appendix D shows the gains are not an artifact of a specific construction model. The replay-based validation of archived traces is a concrete, machine-checkable property rather than a purely statistical claim.
Pith's reading of the paper (a machine's summary, not the authors')
The paper argues that agent skills fail less from missing knowledge than from coupling reasoning with control decisions: the model must repeatedly infer the next step from the skill document, so prescribed operations get skipped or misapplied. HEXIS compiles a skill document into an extended finite state machine that splits these apart — skill knowledge stays in per-state prompts, while explicit transitions over typed variables decide what runs next. A trace-guided compiler builds and refines the machine, accepting an update only when it passes static checks and replays every previously accepted trace. Across four benchmarks and four executors the method beats native Skill + ReAct by 16.1 percentage points on average, improves 15 of 16 settings, and cuts execution tokens by 38.4–88.9% on a mid-size local model. If correct, this gives a reusable, cross-backbone way to enforce prescribed skill steps at lower inference cost.
What carries the argument. The load-bearing object is the extended finite state machine $M = (Q,q_0,V,a,E,F,\tau)$ : states carry local prompts holding skill knowledge, operations with declared read and write variable sets, and tool argument templates; ordered outgoing edges carry guard conditions evaluated on variable values, with unconditional edges last; counters bound loop traversals; terminal states return an outcome category (verified, unverified, fallback). The selection rule — pick the first enabled edge $j^* = \min\{j : g_j(\nu^+) = \text{true}\}$ — is what removes next-step control from model inference. The compiler's acceptance condition — static checks of structure, variable dependencies, terminal evidence, and declared constraints, plus replay of the current and every archived trace — is what lets the machine grow without breaking previously accepted behaviour. Judge states are the bridge where model judgment (a label) meets machine routing (the guard), and the paper concedes that bridge is where reasoning quality the compiler cannot check still decides the outcome.
Core claim. The central claim is that prescribed skill steps are followed more reliably when the skill's control requirements are turned into explicit guarded transitions of a state machine, leaving the model only the in-state reasoning it is good at. The machine records the current stage, typed variables hold task inputs and intermediate results, and after each operation the runtime picks the first enabled edge to route execution, so choosing the next step no longer requires another model decision. Judge states emit a fixed label that guards use to branch to revision or verification. A compiler first maps skill clauses and tool interfaces to states, local prompts, data bindings, and transitions, then aligns development traces to add missing operations and dependencies, committing an update only when static checks and replay of the current and all archived traces pass. The claim is that this division — program-enforced control, model-enforced content — reduces the per-step deviation events the paper models as a product over a chain rule, and that the resulting machines transfer unchanged to other backbones without target-model recompilation.
The premise it rests on. The load-bearing premise is that the compiled machine's transition guards and judge-label routing faithfully encode the skill's required control flow, and that reaching the 'verified' terminal means acceptable task completion; the paper itself concedes that static checks and trace replay validate recorded paths, not in-state reasoning or unseen situations, and that 'verified' means the delivered file agrees with the model's chosen answer letter, not that the mathematical answer is correct.
If the paper is right (Pith's inferences, not the authors')
Machines compiled once from one executor's traces transfer to other backbones unchanged and still improve success in 11 of 12 target-model settings, so compilation cost need not be paid per model.
On Qwen3.8-27B execution tokens fall 38.4–88.9% across benchmarks, indicating that enforcing control in transitions can cut the context overhead of carrying the whole skill and interaction history.
Combining HEXIS with skill-text optimization reaches 84.2% success on SpreadsheetBench at 69k tokens per task, suggesting content quality and control structure are complementary levers.
The deviation model, which frames cumulative compliance as a product over per-step deviation probabilities, predicts that removing repeated next-step inference compounds, matching the pattern of gains on LiveMath.
The information-theoretic framing with mutual-information accumulation bounds says the machine's representation converges to the skill's semantics given enough distinguishing traces, under assumptions the authors state explicitly.
The weakest link is the judge label: if an in-state judge mislabels a failed check, the guard dutifully routes execution down the wrong branch and the machine's 'verified' terminal still reports success — compilation shifts failure from step-selection into verdict quality, so a test that isolates mislabel rates would separate those two sources of error.
Because replay uses recorded tool outputs and placeholder model results, it certifies recorded paths, not behaviour on variations the traces never exercised; the natural stress test is sweeping novel branches and adversarial task inputs, where compiled and native execution should diverge.
Since 'verified' only certifies that the delivered file agrees with the model's own selected answer letter rather than that the answer is correct, measuring congruence with ground truth would reveal how much of the compliance gain is procedural versus substantive.
Cross-backbone transfer suggests the compiled machine encodes model-agnostic structure; if so, the same machines might compress prompting further or feed cheap verifiers, though the paper does not test these extensions.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Agent skills provide instructions and reusable knowledge, yet agents have to frequently infer which action to take next. This skill's execution paradigm combines task reasoning with control decisions, which could cause the agent to act improperly or omit necessary steps. To improve agent compliance for skills, we introduce HEXIS, which compiles agent skills into extended finite state machines (FSM) that separate knowledge from control flow. Skill knowledge is incorporated into local instructions that guide reasoning and generation within states. The machine records execution progress and intermediate results, while explicit transition conditions determine subsequent operations. Our incremental compiler first maps skill clauses and tool interfaces to state operations, local instructions, data bindings, and transitions. It then aligns development traces with existing states to identify missing operations and dependencies. These are incorporated by adding or reusing states and refining their connections. Updates are accepted only after static checks and replay of the current and all previously accepted traces. Across four benchmarks and four executors, HEXIS improves success over Skill + ReAct by 16.2 percentage points on average. Qwen3.8-27B reduces execution tokens by 38.4-88.9% across benchmarks.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-25
The authors' main claim, as the classifier picked it out 0.97
“Across four benchmarks and four executors, HEXIS improves success over Skill + ReAct by 16.1 percentage points on average.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.11 · human participants 0.04 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Agent skills provide reusable knowledge and instructions, yet agents must repeatedly infer how to apply them and which operation should follow. | background 0.93 | 0.07 | 0.03 | asserted 1.00 |
| This couples task reasoning with control decisions, allowing prescribed steps to be omitted or applied incorrectly. | background 0.87 | 0.05 | 0.02 | asserted 0.98 |
| We introduce HEXIS, which compiles agent skills into extended finite state machines that separate knowledge from control flow. | contribution 1.00 | 0.89 | 0.27 | asserted 0.99 |
| Skill knowledge is incorporated into local instructions that guide reasoning and generation within states. | method 0.97 | 0.64 | 0.06 | asserted 1.00 |
| The machine records execution progress and intermediate results, while explicit transition conditions determine subsequent operations. | method 0.98 | 0.57 | 0.05 | asserted 1.00 |
| Our incremental compiler first maps skill clauses and tool interfaces to state operations, local instructions, data bindings, and transitions. | method 0.99 | 0.91 | 0.04 | asserted 1.00 |
| It then aligns development traces with existing states to identify missing operations and dependencies. | method 1.00 | 0.82 | 0.02 | asserted 0.98 |
| These are incorporated by adding or reusing states and refining their connections. | method 0.99 | 0.71 | 0.03 | asserted 0.98 |
| Updates are accepted only after static checks and replay of the current and all previously accepted traces. | method 1.00 | 0.80 | 0.03 | asserted 0.99 |
| Across four benchmarks and four executors, HEXIS improves success over Skill + ReAct by 16.1 percentage points on average. | result 1.00 | 0.95 | 0.56 | asserted 1.00 |
| Qwen3.8-27B reduces execution tokens by 38.4-88.9% across benchmarks. | result 1.00 | 0.85 | 0.15 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.95
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-25: 1 request, 7,913 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment