Artificial Intelligence · arXiv:2609.36630 v2 · submitted 29 Sep 2026
A Comprehensive Survey of Agent Distillation across Models, Artifacts, and Harnesses
The paper says
“Knowledge distillation provides a teacher-student framework for transferring capabilities at reduced cost and has been widely studied for large language models (LLMs).”
more of the abstract
As LLM-based agents move from response generation to sustained, closed-loop task solving, distillation expands beyond model parameters to knowledge retained in reusable resources and execution workflows. This shift from model-level to agent-level knowledge transfer supports efficient deployment, broader capability access, and continual self-evolution. However, existing surveys largely examine model distillation and agent system design separately, limiting comparison of transfer strategies across agent components. To bridge this gap, we formalize Agent Distillation as persistent knowledge transfer between teacher and student agents and present its first unified survey. Distinguishing experience used for transfer from knowledge retained for reuse, we develop a knowledge-substrate taxonomy based on where knowledge persists: models, artifacts, or harnesses. Building on this taxonomy, we analyze cross-substrate transfer paths and hybrid strategies. To assess the resulting capabilities, we review benchmarks and evaluation protocols for retention, causal attribution, and deployed utility without further teacher support. This survey provides a unified basis for studying agent knowledge transfer and developing efficient, adaptable, and self-evolving agents.
from the abstract
Pith says
Machine review · deepseek-v4-flash · 30 Sep 2026 · of version 1 of the paper; version 2 is newer · what was read was not recorded · not checked by a person
It can be wrong. Challenge this review or reply to it.
The passes disagreed. Reader: holds up. Skeptic: agrees with the reader. Referee: major revision.
A genuinely useful organizing survey for agent-level knowledge transfer, with one real soft spot: the primary-substrate classification rule is not fully operational for learned control components. This line is wrong
2 major objections · 4 minor
How often objections from reviews like this one are right: not yet measured
Not checked: figures, code, what the references say.
The paper says, at 3 vs. 5.4 and 6.2
The review did not quote this passage.
Pith says: Objection 1 of 2 · major
The substrate classification rule is applied inconsistently to learned control components, which is load-bearing for contribution (2). Section 3 states that 'knowledge that governs the sequencing and control of execution belongs to the harness' and that membership is determined by how knowledge participates in future task solving 'rather than by its physical format.'
more of objection 1
Yet 6.2 assigns a learned router (RouteLLM) to M_S because the knowledge is 'retained in the parameters,' reserving H_S for the runtime that queries it, and 5.4 assigns a learned retriever to M_S while placing the 'runtime policy that decides when to search' in H_S. Section 6.2 then states that for tool and artifact routing, 'the selection and invocation policy resides in H_S.' Under the 3 functional rule, a learned router or learned artifact-retrieval policy governs which component executes and therefore belongs to the harness; the 5.4/6.2 assignments instead privilege the retention locus (parameters), which is precisely the physical-format criterion 3 disavows. Moreover, the two sections disagree about where learned artifact-selection knowledge lives: 5.4 puts the learned retriever in M_S while 6.2 puts the artifact-selection policy in H_S. Since the 3x3 matrix (Eq 18) and the hybrid criterion (Eq 21) presuppose a reproducible primary-substrate assignment, the paper should state an explicit reconciliation rule (e.g., learned decision functions always count as parametric regardless of their control role, with only unlearned enforcement and sequencing logic in H) and then verify that the classifications in 4-7 are consistent under it.
The paper says, at 2.4
The review did not quote this passage.
Pith says: Objection 2 of 2 · major
The empirical basis of the taxonomy is the definition-guided corpus collection and per-study coding described in 2.4 (teacher-student configuration, knowledge form, updated components, mechanism, evaluation setting), but no coding instrument, inter-coder agreement measure, or inclusion/exclusion statistics are reported. Given that the survey's central organizing claim is that every core method admits a primary substrate under the 3 rule, the reproducibility of those assignments should be demonstrated rather than asserted.
more of objection 2
A short appendix containing the coding protocol, counts per substrate class, and one worked example for each of M, R, and H would make the taxonomy checkable and would also serve as the operationalization the field needs to apply the framework to new methods.
4 minor objections
The paper says, at 2.2
Pith says: minor objection 1 of 4
The phrase 'preserving the repository's intended behavior intended behavior' contains a duplicated phrase that should be removed.
The paper says, at Fig. 2 and 1.3
Pith says: minor objection 2 of 4
The word 'parameteric' appears in Fig. 2 and in 1.3 ('the parameteric model'); this is a typo for 'parametric.'
show 2 more
The paper says, at Table 1
Pith says: minor objection 3 of 4
The row label 'Ours2027' should be formatted consistently with the author-year style used for the other rows (e.g., 'Ours (2027)').
The paper says, at Table 3
Pith says: minor objection 4 of 4
Given that Eq 22 defines the claim tuple (source, destination, evidence, transformation, Z_S) as the unit of evaluation, Table 3 would be more actionable if its final column also recorded the strongest supported source-to-destination arrow for each benchmark rather than only a general caveat; several rows (e.g., APIBank, ToolSandbox) currently leave the substrate attribution question implicit.
What would settle it
Take roughly 30 named core methods spanning §4–§6 (e.g., FireAct, AgentBank, Agent Memory Distillation, SKILL-KD, FlowScout, RouteLLM, AFlow, GPTSwarm, Agent Harness Distillation, Kang et al. tool-using models). Give two independent annotators only the original cited papers plus the §3 rule and Eq 21, and ask each to assign a primary substrate (M/R/H) and hybrid status. Compute Cohen's kappa on primary substrate, then inspect whether disagreement concentrates on learned-router, learned-retriever, or learned-workflow methods. Independently, apply the §3 rule and §6.2's treatment to one learned router and show the two readings yield M versus H.
from the skeptic's reading
How the review went
4 readings by deepseek-v4-flash · 30 Sep 2026
- Reader holds up
- Skeptic partly agrees
- Editor wrote the verdict above
- Referee major revision: 2 major objections, 4 minor
Not run on this paper: the simulated authors' defence.
Pith did not check this review's objections against the paper.
The review in full
The review in brief, machine-written
You should read this one. It's the first survey I've seen that squarely names the gap between model-centered distillation surveys (which stop at the student's parameters) and system-centered agent surveys (which don't organize around teacher-student transfer), then builds an actual framework for the joint space. The substrate taxonomy — parametric model, reusable artifacts, execution harness — is the right axis, and the 3x3 source-destination transfer matrix plus the hybrid criterion (Eq 21) is a real new synthesis, not recycled work. The evaluation section earns its keep too: separating acquisition from utility from fidelity from deployment, the gap-closure metric, and the marginal-contribution and synergy formulas (Eqs 27-28) are concrete and usable.
The authors are careful in the ways that matter. Scope is drawn honestly — in-context learning and transient consultation are out, self-evolution is in. They correctly refuse to relabel AFlow and Automated Design of Agentic Systems as distillation when no teacher role is identifiable, and they're upfront that black-box harness recovery yields an equivalence class, not unique structure.
Now the soft spot, and it's the one worth refereeing. The classification rule in §3 says substrate membership is functional — "how the retained knowledge participates in future task solving" — and explicitly disavows physical format. But §6.2 assigns a learned router (RouteLLM) to MS because the knowledge lives in its parameters, reserving HS for the runtime that queries the router. The same decision function — which model to call — is parametric when learned and harness when written as rules. That's exactly the physical-format criterion §3 says doesn't decide. The paper sees the problem (§9.1 concedes different M/R/H combinations give the same behavior; §7.1 demands source-agnostic labels; §6.4 refuses a single label for joint changes), yet Sections 4-6 still hand out single-cell assignments for the corpus. The fix is cheap: an inter-coder reliability check on a sample of the corpus, or explicit boundary-case rulings per contested method. Without it, the taxonomy's organizing power rests on the authors' reading.
Minor: the substrate vocabulary isn't wholly new — AgentDistill's MCP modules, memory distillation, and harness surveys already use pieces of it. The unified framing is the contribution, and the paper hedges appropriately; citing published results for derived claims is normal survey practice, not a flaw.
Who it's for: anyone distilling agents, building agent memory/tool/workflow systems, or writing agent evaluation protocols. It deserves a serious referee; the classification issue is fixable in revision and doesn't undercut the core synthesis.
The strongest objection
The load-bearing engine of the survey is reproducibility of classifier assignment: the claim (contribution 2, §3, 3×3 matrix of §7.1, hybrid criterion of Eq 21) that every core method admits a primary substrate (M/R/H) under the §3 rule, 'substrate membership is determined by how the retained knowledge participates in future task solving, rather than by its physical format or the mechanism used to obtain it.' That rule is non-operational for a class central to the agentic literature: learned control components. §6.2 assigns a learned router (RouteLLM) to M_S ('if the transferable knowledge is retained in the parameters of a learned router, that router is a parametric decision model and the primary update belongs to M_S'), reserving H_S for 'the surrounding runtime that queries the router.' §5.4 does the same for learned retrievers; §7.2 for learned workflow generation. Yet under §3's own definition ('knowledge that governs the sequencing and control of execution belongs to the harness'), a router that selects models or artifacts is harness knowledge. So the same decision function is classified parametric when implemented as learned weights and harness when written as explicit rules or a lookup table — precisely the physical-format criterion §3 disavows. The paper acknowledges the underlying problem (§9.1: 'Different combinations of M, R, and H can produce the same final behavior'; §7.1 demands source-agnostic labels; §6.4 refuses a single label for joint changes), yet Sections 4–6 and Table 1 still issue single-substrate, single-cell assignments for the surveyed corpus (b254, b379, and the RouteLLM and Agent Memory Distillation placements). Those assignments are made by reading published method descriptions, which rarely document the functional-participation detail the rule requires. The taxonomy's organizing power therefore rests on the authors' privileged reading rather than on a criterion an independent coder can apply consistently.
The reader's weakest_assumption flagged that methods may resist single-primary-substrate assignment, weakening the taxonomy's organizing power. My concern is related but more specific: the failure is not only empirical overlap but an internal inconsistency in the classification rule itself. §3 declares substrate determined by how knowledge participates in future task solving, not by physical format, yet §6.2/§5.4/§7.2 assign learned control components (routers, retrievers, workflow generators) to M_S by virtue of living in neural weights, reserving H_S for surrounding runtime. A state-conditioned selection policy is harness knowledge by §3's own definition, so the same transferred knowledge receives M or H depending on the implementation the original paper chose — a physical-format judgment the survey disclaims. This is an internal-consistency concern (rule contradicts its own application), not merely disagreement with a broader consensus. I kept the verdict UNCHANGED for three reasons. First, the survey is heavily hedged: it explicitly stapled 'for organizational purposes' to the classification, admits in §9.1, §7.1, and §6.4 that substrate attribution is an identification problem, and demands source-agnostic labels when access is unavailable — the authors are not overclaiming. Second, the paper's most durable contributions, the scope definition (persistent teacher-derived knowledge incorporation) and the §8 evaluation framework (claim tuples, ablation, intervention, Eq 26–28), survive even if the single-substrate cells are heuristic; the evaluation section is explicitly substrate-agnostic in its caution. Third, the concern is testable and, if confirmed, would weaken the novelty of the substrate organiization but would not overturn acceptance of a careful survey that already flags the limits. My check is designed to measure the actual reproducibility cost: an inter-coder study applying the §3 rule blind, plus a direct consistency probe on a learned router. If the check showed kappa below roughly 0.5 with disagreements concentrated on learned-control methods, I would move to CONDITIONAL and require the authors to mark single-substrate assignments as interpretive or provide the coding protocol, since the taxonomy's organizing contribution — the claimed first unified definition plus substrate classification — would then rest on the authors' privileged reading. Short of that evidence, the current ACCEPT at MODERATE confidence is the right verdict, and my concern does not change it.
The referee's report
If the framework holds, the paper fills a genuine gap: distillation surveys are model-centered while agent surveys do not organize around persistent teacher-student transfer, and the substrate taxonomy gives the field a shared vocabulary and an evaluation protocol (claim tuple, substrate ablations, synergy measure) that is more rigorous than typical survey evaluation discussion. The formalization is definitional rather than empirical, but it is internally consistent, and the scope boundaries (2.3) are explicit enough to be falsifiable as classification rules. The evaluation framework in 8.1-8.2, particularly the claim tuple in Eq 22, the marginal-contribution and synergy measures in Eqs 27-28, and the benchmark caveats in Table 3, is a substantive contribution that goes beyond what most roadmaps offer. The main risk is the operationality of the substrate classification rule for learned control components, where the paper's own sections apply the rule inconsistently; this bears directly on contributions (2) and (3), so it needs to be resolved before the taxonomy can serve as the field's organizing dimension.
Pith's reading of the paper (a machine's summary, not the authors')
Modern agents are more than language models: they pair a parametric model with reusable artifacts (memories, tools, skills) and an execution harness that routes, verifies, and recovers. This paper argues that knowledge distillation, which traditionally asks a student model to imitate a teacher model, must be redefined at the level of the whole agent. It defines Agent Distillation as the persistent transfer of task-solving knowledge from a teacher agent into a student agent, and organizes the literature by where that knowledge is retained: inside the model, as external artifacts, or in the execution harness. On this substrate taxonomy, every agent-distillation method, and every cross-substrate or hybrid variant, can be classified and evaluated. The payoff is a common foundation for building, benchmarking, and safely maintaining agents that learn from experience and from other agents.
What carries the argument. The load-bearing object is the agent realization $A=(M,R,H)$ — the parametric model $M$, reusable artifacts $R$, and execution harness $H$ — together with the knowledge-substrate rule that assignment to a category follows how retained knowledge participates in future task solving, not its file format or how it was produced. Three substrates organize the survey: parametric knowledge in learned computation (fine-tuned adapters, policies), artifact knowledge in addressable objects (memories, demonstrations, plans, code, tools), and harness knowledge in execution logic (routing, verification, recovery, orchestration). A transfer path is expressed as $K^x_T \rightarrow E_T \rightarrow K^y_S$ over a $3\times 3$ source-destination matrix, and the hybrid criterion marks students where more than one substrate retains a teacher-derived change. A running example of a coding agent debugging an API regression carries the definition through teacher trace, filtered and abstracted knowledge, and persistent reuse on a fresh task.
Core claim. The paper's central move is to relocate distillation's unit of transfer from the model to the agent. It formalizes an agent as a triple $A=(M,R,H)$ of parametric model, reusable artifacts, and execution harness, and defines Agent Distillation as the persistent incorporation of teacher-derived knowledge $K_T$ into the student's components, yielding an updated realization $A^+_S$ that retains the knowledge after the teacher is gone. From this it derives a taxonomy: knowledge internalized in learned computation is parametric, knowledge externalized as addressable objects (memories, skills, tools) is artifact-based, and knowledge operationalized as execution and control logic is harness knowledge. Methods are classified by the primary substrate in which teacher knowledge persists, giving a $3\times 3$ source-destination matrix for cross-substrate transfer and a hybrid criterion ($|Z_S|>1$) for endpoints where multiple substrates retain change. It also contributes an evaluation framework that separates transfer evidence from outcome and asks whether claimed retention is causally responsible for deployed utility.
The premise it rests on. The load-bearing premise is that every agent realization decomposes exhaustively into model, artifacts, and harness, and that each distillation method can be assigned one primary substrate, so the taxonomy's categories do not overlap in practice.
If the paper is right (Pith's inferences, not the authors')
Conventional model-level distillation, in-context prompting, and inference-time consultation are grouped as mechanisms or neighbouring work, not Agent Distillation, unless they produce a persistent change in the student agent.
Every existing agent-distillation method can be filed under one of three branches (parametric, artifact, harness), and cross-substrate or hybrid methods are characterized by both their transfer path and their deployment endpoint.
Evaluation should tie transfer evidence to causal contribution: reporting which student substrates changed and testing by removal or swap whether they matter, rather than reporting an end-task score alone.
Self-evolution counts as Agent Distillation when prior experience supplies teacher-side evidence that the same agent persistently retains for future episodes.
Benchmarks are organized by the deepest claim they support, with substrate attribution and teacher provenance required before a gain is credited to a particular component.
A testable consequence of the taxonomy is that methods classed as parametric, artifact, or harness should show different failure and maintenance profiles — parameters are fastest but least auditable, artifacts are editable but impose retrieval cost, and harnesses guarantee ordering but break on interface change — so one could predict which substrate a given deployment constraint should favor.
The hybrid criterion suggests a design rule the paper only sketches: knowledge that is stable, frequent, and latency-sensitive should be internalized in the model, while explicit, exact, or rapidly changing knowledge belongs in artifacts, and conditional or safety-critical control belongs in the harness.
An external reader could turn the attribution framework into a predictive experiment: train the same student knowledge under two substrate allocations (parametric versus artifact-based) at equalized budgets, then measure not only utility but update cost, rollback ease, and portability across base models.
The definition implies a boundary case worth watching: black-box harness distillation recovers an observational equivalence class, not the teacher's unique organization, so porting third-party orchestration should be described as functional replication unless traced or causally identified.
Check this review
Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.
How often reviews like this one are right
Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.
How often objections from reviews like this one are right: not yet measured
Reviews by people
This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.
The paper
Abstract
Knowledge distillation provides a teacher-student framework for transferring capabilities at reduced cost and has been widely studied for large language models (LLMs). As LLM-based agents move from response generation to sustained, closed-loop task solving, distillation expands beyond model parameters to knowledge retained in reusable resources and execution workflows. This shift from model-level to agent-level knowledge transfer supports efficient deployment, broader capability access, and continual self-evolution. However, existing surveys largely examine model distillation and agent system design separately, limiting comparison of transfer strategies across agent components. To bridge this gap, we formalize Agent Distillation as persistent knowledge transfer between teacher and student agents and present its first unified survey. Distinguishing experience used for transfer from knowledge retained for reuse, we develop a knowledge-substrate taxonomy based on where knowledge persists: models, artifacts, or harnesses. Building on this taxonomy, we analyze cross-substrate transfer paths and hybrid strategies. To assess the resulting capabilities, we review benchmarks and evaluation protocols for retention, causal attribution, and deployed utility without further teacher support. This survey provides a unified basis for studying agent knowledge transfer and developing efficient, adaptable, and self-evolving agents.
The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.
abstract compiled · 2026-09-30
The authors' main claim, as the classifier picked it out 1.00
“Together, these contributions establish a foundation for the reliable, maintainable, and safe development of increasingly complex agentic systems.”
How the main claim is stated: asserted 1.00
What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.10 · human participants 0.05 · controlled comparison 0.04 · randomized assignment 0.01 · states quantitative result 0.03 · compares with prior results 0.10 · mentions code or data availability 0.03
The abstract, sentence by sentence
This table labels the abstract only (arXiv listing abstract v1). The number is the probability the classifier gave the label.
| Sentence | Role | The authors' own work | States the main conclusion | Strength |
|---|---|---|---|---|
| Modern agents increasingly rely on memories, tools, and execution logic, so their competence extends beyond model parameters. | background 1.00 | 0.07 | 0.04 | asserted 0.98 |
| This shift exposes a limitation of conventional knowledge distillation, which asks how a student model imitates a teacher model. | background 0.48 | 0.09 | 0.05 | asserted 1.00 |
| We define Agent Distillation as the persistent transfer of task-solving knowledge from a teacher agent to a student agent. | contribution 0.99 | 0.59 | 0.12 | asserted 1.00 |
| Our study organizes the field by where transferred knowledge is retained: within the model, as artifacts, through the execution harness, or across substrates. | contribution 0.94 | 0.78 | 0.08 | asserted 0.98 |
| This perspective separates transfer evidence from its outcome and clarifies how knowledge moves between agent components. | contribution 0.61 | 0.67 | 0.10 | asserted 1.00 |
| We develop an evaluation framework that relates retention to causal contribution and deployed utility. | contribution 0.78 | 0.93 | 0.09 | asserted 1.00 |
| Together, these contributions establish a foundation for the reliable, maintainable, and safe development of increasingly complex agentic systems. | interpretation 0.63 | 0.49 | 0.71 | asserted 1.00 |
What kind of paper the classifier reports this is: position or commentary 0.58 · evidence from argument 0.91
What was examined, and what was not
Sources read
- arXiv listing abstract v1
Not examined
- full text
- figures and tables
- references and citing sentences
- relations to other papers
Levels run
- P abstract
Not run
- S sections
- T body sentences
- C citations
- R cross-paper relations
How the labels were made
Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-30: 1 request, 5,434 input tokens.
Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON
Figures (5)
Figures are the authors', reproduced from the paper.
Cited by 1 Pith paper
Reviewed papers on Pith that cite this one. The full citation record
Challenges and discussion
Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.
Come back by email
A short list of the week's newly reviewed papers. Nothing is sent until you confirm from your inbox, and every email has a link that stops it. No account.
No comments yet.
Sign in to comment