Artificial Intelligence · arXiv:2610.19148 v2 · submitted 13 Oct 2026
Compiling Expert Procedures into Verified Programs: When a Small Model Is Enough
The paper says
“In procedural tool-use domains, how much of the gap between base models is procedure rather than capability? We build an explicit program from a teacher's traces and run it in front of unchanged small models, including an 8B model.”
more of the abstract
The construction, PLVR , takes the reasoning traces of a teacher, Kimi-K3, clusters their segments to locate recurring operations, has a frontier model author one typed primitive per operation, each run by a small frozen model and assembles the program by A* search over a mined rule library, under a reference-action objective. At inference the program takes one of two forms: for troubleshooting and tool search it produces the chain of thought the student would otherwise generate for itself, and the student acts on it; for multi-turn tool matching the student proposes each action and the program checks, repairs, or retries it before it goes out. On the official tau2-Bench Retail and Telecom test sets and on LiveMCPBench, with three Nemotron base models and the same program for each, the answer depends on the domain. On Telecom, Granite-4.2-8B as the student goes from 43.0% to 91.2% under the program, and every Nemotron student reaches 93.3-94.4%, against at most 58.2% for fine-tuning on the same traces, prompt optimisation of the student, or two token-matched baselines. With Granite-4.2-8B as the student, the program is within 2 to 3 points of the Nemotron students on Telecom and within 3 on Retail, and 3 to 13 points below them on LiveMCPBench: where the domain is procedural an 8B student is enough, and on tool search the student still matters. On Retail and LiveMCPBench the program is competitive with the best baseline. The spread between base models shrinks from about 7 points to about 2, averaged over the suites, with no weights changed.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 15 Oct 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
Holds up. A clean empirical separation of procedure from capability in procedural tool-use domains, with a novel method that compiles teacher traces into a verifiable program. This line is wrong
1 major objection · 7 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at Abstract and §5, second bullet (b8, b33, b122)
The review did not quote this passage.
Pith says: Objection 1 of 1 · major
The paper claims that the spread between base models shrinks from 6.6 points to 1.9 points averaged over suites. However, the per-suite spreads reported in the same paragraph (15.8→3.5 on Retail, 15.0→1.1 on Telecom, 10.9→9.4 on LiveMCPBench) average to 13.9 (untreated) and 4.67 (under the program), not 6.6 and 1.9.
more of objection 1
The authors should clarify what metric defines 'spread' (e.g., average absolute deviation, standard deviation, or something else) and correct the numbers if needed. The qualitative conclusion that the spread narrows substantially stands, but the precise values should be consistent.
7 minor objections
The paper says, at §2, Related Work
Pith says: minor objection 1 of 7
Several citations appear as double question marks (e.g., '??' for Program-of-thought and PAL-style methods). These should be filled with proper references.
The paper says, at Table 2 footnote
Pith says: minor objection 2 of 7
The footnote says 'The program’s Telecom cells are means over three runs; all other cells ... are one run.' This is appropriate but should also note whether the standard deviation over those three runs is available, even if small, for completeness.
show 5 more
The paper says, at §3.2, primitive extraction
Pith says: minor objection 3 of 7
The text states that 'clustering involves no language model and no hand-picked segments,' yet frontier models (Claude 5 Opus) post-process clusters to author primitives. Clarifying that the clustering itself is automated but the primitive design is human-assisted through the frontier model would avoid confusion.
The paper says, at §3.4, program search
Pith says: minor objection 4 of 7
The search is described as A* with successor insertion of one rule or group of rules. The number of groups is not specified; listing the group structure would help reproducibility.
The paper says, at §5.6, ablation
Pith says: minor objection 5 of 7
The ablation that 'removes both things the traces give the method' still uses the hand-written seed, which itself encodes domain knowledge. The experiment is valuable but it would be clearer to state that even without traces, a frontier model can propose some procedures (75.9% on Telecom).
The paper says, at Appendix D.1.3, Table of clusters
Pith says: minor objection 6 of 7
The column 'Domains' lists 'tool-search, retail, telecom' but some clusters are marked as appearing in only one domain; consider using a consistent naming (e.g., 'M' for tool-search) to avoid confusion with the rule library IDs.
The paper says, at General
Pith says: minor objection 7 of 7
The paper uses 'Granite' to refer to Granite-4.2-8B but sometimes omits the size; specifying the full model name on first use helps clarity.
What would settle it
Compute the spread (max minus min student accuracy per suite) for every baseline method (SFT, GEPA, ReAct, sample-and-judge) in Table 2. If the program's averaged spread (1.9) is not substantially smaller than the smallest baseline spread (e.g., sample-and-judge on Retail likely has spread <3 points), the claim that the program uniquely shrinks the gap would weaken and require more careful qualification.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 15 Oct 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee minor revision: 1 major objection, 7 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
This paper does what a lot of work talks about doing: it cleanly separates how much of a performance gap is procedure versus capability, and it does so with a concrete method (PLVR) that compiles teacher traces into a program rather than into weights. The result on Telecom — an 8B model jumping from 43% to 91% under the same program, while larger models cluster within a point — is striking and well-supported.
The novelty is in the construction: clustering teacher trace segments to propose primitives, having a frontier model author those primitives with typed I/O and deterministic verifiers, and assembling a program via A* search over a mined rule library with a reference-action objective. This is not just another distillation or prompt optimization paper. The experiments are thorough: multiple baselines (SFT, GEPA, token-matched ReAct, sample-and-judge) and ablations (seed vs search, no-traces variant). The ablation shows the traces contribute more than the frontier model's prior knowledge. The authors honestly report where the method works (Telecom, Retail) and where it doesn't (LiveMCPBench/tool search). The limitations are stated upfront: single runs, shared domains, upper bound, cost.
The concerns are real but acknowledged. Single runs for most cells limit resolution, but the pattern across three models and three suites is consistent. The synthetic training set shares domains with test sets, but the authors verified no task overlap. The teacher traces are assumed optimal, but the authors call the estimate an upper bound and show that removing traces degrades performance. The inference cost is 5-10x vanilla, but for procedural tasks that might be acceptable. None of these are load-bearing flaws.
This paper is for anyone working on agentic systems, tool use, or neurosymbolic approaches. It deserves a serious referee and will likely influence deployment decisions. The central claim—that most of the gap in procedural domains is procedure—is well-supported by the data. I would move to accept.
The strongest objection
The paper is thorough and its central claim is well-supported by the data. The most concerning assumption—that teacher traces approximate optimal procedure—is explicitly acknowledged as yielding an upper-bound estimate, and the ablation in §5.6 shows that removing traces degrades performance, confirming the traces contribute substantive procedure beyond the frontier models' prior knowledge. The synthetic training set shares domains with the test sets, but this equally affects all baselines and the authors verified no task overlap. Single runs for most cells are honestly noted, and the pattern of results (dramatic gap closure on procedural Telecom and Retail, negligible change on tool-search LiveMCPBench) is consistent across domains and models. No additional load-bearing concern beyond those already discussed by the authors and the reader emerges from a careful read.
The reader identifies teacher-trace optimality as the weakest assumption. I agree this is a meaningful limitation, but the paper honestly calls their estimate an upper bound (§6) and provides an ablation showing traces contribute beyond frontier-model knowledge (§5.6). That concern does not undermine the central claim that procedure accounts for most of the gap in procedural domains—the dramatic performance lift on Telecom (43%→91% for an 8B model) and near-zero spread (1.1 points) among students under the same program are striking and robust. My primary check would be on whether the spread reduction is unique to the program compared to baselines (e.g., sample-and-judge may also reduce spread on Retail). If baselines also reduce spread substantially, the conclusion that the gap is 'procedure' rather than 'capability that baselines also capture' would need sharper framing. However, even in that scenario the program's absolute performance on Telecom remains far above all baselines, so the core finding stands. The paper is careful, well-scoped, and honestly limited. I find no reason to change the ACCEPT verdict.
The referee's report
If the result holds, it provides a crisp method for separating procedure from capability in procedural tool-use domains and shows that explicit, verifiable programs can substitute for model scaling in certain tasks. The empirical evidence is strong: the same program raises an 8B model from 43.0% to 91.2% on Telecom, and the spread between three much larger models collapses on the two most procedural suites. The methodology is transparent (all primitives, rules, and search details are given) and the limitations (single runs for most cells, shared domains, frontier model prior knowledge) are honestly discussed. The paper advances the understanding of when and why small models can suffice.
Pith's reading of the paper (a machine's summary, not the authors')
This paper asks how much of the performance gap between base models of different sizes in procedural tool-use tasks is due to missing procedure rather than missing capability. To answer, the authors build an explicit program (PLVR) from a teacher's traces by clustering segments, having a frontier model author typed primitives run by a frozen small model, and assembling them via A* search. This single program is run unchanged in front of several base models, including an 8B one. On telecom troubleshooting, the 8B model jumps from 43% to 91% under the program, every larger model reaches 93-94%, and no other baseline exceeds 58%. On retail multi-turn matching, the 8B model is within 3 points of the large ones; on tool search, the student still matters. The spread between base models shrinks from about 7 points to about 2 averaged over the suites.
What carries the argument. PLVR (Program Learning with Verifiable Rewards) constructs the program by clustering teacher trace segments into recurring operations, having a frontier model author one typed primitive per operation (each a call to a frozen small model with input-output schema, deterministic verifier, and fallback), and assembling primitives into a program by A* search over a mined rule library under a reference-action objective that minimises the distance between the program's actions and the teacher's actions. At inference, the program either replaces the student's chain-of-thought reasoning or guards the student's proposed actions by checking and repairing them before they execute.
Core claim. The paper discovers that in procedural tool-use domains, most of the performance gap between base models of different sizes can be eliminated by capturing the expert procedure in an explicit, verifiable program rather than by training a larger or better model. The single program constructed from teacher traces narrows the spread between three Nemotron models from 6.6 points to 1.9 averaged over three test suites. An 8B model under the program reaches 91.2% on telecom troubleshooting and 82.5% on retail, within 2-3 points of the best Nemotron students on those domains. The teacher's procedure, not the student's inference-time reasoning, supplies nearly all the performance on procedural tasks.
The premise it rests on. The teacher traces (from Kimi-K3) are assumed to represent optimal or near-optimal procedure for the tasks; if the teacher's procedure is suboptimal or the domain is mismatched, the compiled program encodes that same procedure, potentially limiting achievable performance.
If the paper is right (Pith's inferences, not the authors')
In domains with checkable intermediate outputs, a program compiled from expert traces can serve as a drop-in replacement for chain-of-thought reasoning, lifting small models to near the performance of much larger ones.
The method provides an upper bound on how much of a model gap is procedure: the proportion of the spread that a single program closes across base models, guiding decisions on whether to invest in a larger model or in better procedure capture.
Since the program is a readable, modifiable object rather than a weight update, it can be inspected, debugged, and refined one primitive at a time without retraining.
The technique could extend to other procedural domains with verifiable outputs, such as code generation with test cases, data processing pipelines, or multi-step planning in constrained environments.
The result suggests that for many enterprise tool-use tasks, the bottleneck is not model scale but the quality of the written procedure: an 8B model with a good procedure can match a 12B model without one, implying that organisations should prioritise documenting expert workflows over training larger models.
The method shows striking data efficiency: 539 teacher traces produce large gains through the program while fine-tuning on the same traces yields much smaller improvements, hinting that the structure of what operations occur matters more than the number of traces.
A testable extension would apply PLVR to a non-tool-use procedural domain (e.g., legal document drafting or medical triage) with checkable intermediate outputs; if the program similarly closes the model gap, the claim generalises beyond tool-use.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
In procedural tool-use domains, how much of the gap between base models is procedure rather than capability? We build an explicit program from a teacher's traces and run it in front of unchanged small models, including an 8B model. The construction, PLVR , takes the reasoning traces of a teacher, Kimi-K3, clusters their segments to locate recurring operations, has a frontier model author one typed primitive per operation, each run by a small frozen model and assembles the program by A* search over a mined rule library, under a reference-action objective. At inference the program takes one of two forms: for troubleshooting and tool search it produces the chain of thought the student would otherwise generate for itself, and the student acts on it; for multi-turn tool matching the student proposes each action and the program checks, repairs, or retries it before it goes out. On the official tau2-Bench Retail and Telecom test sets and on LiveMCPBench, with three Nemotron base models and the same program for each, the answer depends on the domain. On Telecom, Granite-4.2-8B as the student goes from 43.0% to 91.2% under the program, and every Nemotron student reaches 93.3-94.4%, against at most 58.2% for fine-tuning on the same traces, prompt optimisation of the student, or two token-matched baselines. With Granite-4.2-8B as the student, the program is within 2 to 3 points of the Nemotron students on Telecom and within 3 on Retail, and 3 to 13 points below them on LiveMCPBench: where the domain is procedural an 8B student is enough, and on tool search the student still matters. On Retail and LiveMCPBench the program is competitive with the best baseline. The spread between base models shrinks from about 7 points to about 2, averaged over the suites, with no weights changed.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-10-15
The authors' main claim, as the classifier picked it out 0.99
“With Granite-4.2-8B as the student, the program is within 2 to 3 points of the Nemotron students on Telecom and within 3 on Retail, and 3 to 13 points below them on LiveMCPBench: where the domain is procedural an 8B student is enough, and on tool search the student still matters.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.04 · states limitation 0.91 · human participants 0.09 · controlled comparison 0.96 · randomized assignment 0.02 · states quantitative result 0.99 · compares with prior results 0.98 · mentions code or data availability 0.04
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| In procedural tool-use domains, how much of the gap between base models is procedure rather than capability? | background 0.51 | 0.05 | 0.04 | unclear 0.99 |
| We build an explicit program from a teacher's traces and run it in front of unchanged small models, including an 8B model. | method 0.57 | 0.95 | 0.06 | asserted 0.97 |
| The construction, PLVR , takes the reasoning traces of a teacher, Kimi-K3, clusters their segments to locate recurring operations, has a frontier model author one typed primitive per operation, each run by a small frozen model and assembles the program by A* search over a mined rule library, under a reference-action objective. | method 0.99 | 0.92 | 0.04 | asserted 0.99 |
| At inference the program takes one of two forms: for troubleshooting and tool search it produces the chain of thought the student would otherwise generate for itself, and the student acts on it; for multi-turn tool matching the student proposes each action and the program checks, repairs, or retries it before it goes out. | method 0.99 | 0.82 | 0.05 | asserted 1.00 |
| On the official tau2-Bench Retail and Telecom test sets and on LiveMCPBench, with three Nemotron base models and the same program for each, the answer depends on the domain. | result 0.85 | 0.92 | 0.14 | asserted 0.98 |
| On Telecom, Granite-4.2-8B as the student goes from 43.0% to 91.2% under the program, and every Nemotron student reaches 93.3-94.4%, against at most 58.2% for fine-tuning on the same traces, prompt optimisation of the student, or two token-matched baselines. | result 1.00 | 0.95 | 0.22 | asserted 1.00 |
| With Granite-4.2-8B as the student, the program is within 2 to 3 points of the Nemotron students on Telecom and within 3 on Retail, and 3 to 13 points below them on LiveMCPBench: where the domain is procedural an 8B student is enough, and on tool search the student still matters. | result 0.68 | 0.95 | 0.74 | asserted 1.00 |
| On Retail and LiveMCPBench the program is competitive with the best baseline. | result 1.00 | 0.93 | 0.11 | asserted 1.00 |
| The spread between base models shrinks from about 7 points to about 2, averaged over the suites, with no weights changed. | result 0.97 | 0.94 | 0.28 | asserted 1.00 |
What kind of paper the classifier reports this is: method or tool 0.97 · evidence from observation or experiment 0.99
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-10-15: 1 request, 7,347 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (1)
Figures are the authors', reproduced from the paper.
Cited by 2 Pith papers
Reviewed papers on Pith that cite this one. The full citation record
- ReTrace: Distilling Tool-Using Agents into Planner-Free Workflows cs.AI · Nov 2026
- Bellman Language Models: Solving the Recurring Decisions of LLM Agents cs.AI · Oct 2026
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment