Pith.

Artificial Intelligence · arXiv:2607.28048 v4 · submitted 30 Jul 2026

SKILL-KD: Contrastive Skill Distillation for LLM Agents

Qiming Shi, Yibo Dou, Jiawen Zhu, Yulong Tao, Linbo Jin, Zhaolu Kang, Yunfan Zhou, Di Weng

The paper says

“Skill-based prompting has become a practical mechanism for improving LLM agents, yet existing methods often treat skills as summaries of the agent's own experience or of successful demonstrations. This creates a mismatch for weaker student agents.”

more of the abstract

A failed trajectory may not reveal the missing knowledge or strategy, while a teacher trajectory may be too implicit to internalize. We propose SKILL-KD, a contrastive skill distillation framework that treats skills as an explicit distillation medium between agents of different capabilities. Given a student failure and the teacher trajectory on the same task, SKILL-KD distills their actionable discrepancy into a textual skill patch, evaluates it by re-running the student, and iteratively refines it when the student still fails. To prevent skill drift from repeated local updates, SKILL-KD maintains trace-linked edit histories and performs Drift-Aware Skill Consolidation to decide whether each patch is added, merged, or skipped. Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines.

from the abstract

Pith says

Machine review · deepseek-v4-flash · 5 Aug 2026 · of version 2 of the paper; version 4 is newer · what was read was not recorded · not checked by a person

It can be wrong. Challenge this review or reply to it.

Holds up, with conditions. Worth a serious look: a plausible, well-ablated distillation method whose 'consistent gains' claim is statistically undersupported by single-run, small-split evaluation. This line is wrong

3 major objections · 4 minor

How often objections from reviews like this one are right: not yet measured

Are you an author? What you can do.

Not checked: figures, code, what the references say.

The paper says, at Table 1 / Appendix A

The review did not quote this passage.

Pith says: Objection 1 of 3 · major

The central claim of 'consistent improvement' is an empirical statement, but Appendix A states that 'Each reported score was computed from a single complete evaluation run on the fixed test split.' Several decisive margins in Table 1 are within a few test instances (e.g., SpreadsheetBench Group 1 24.3 vs 23.9, n=280; DocVQA Group 1 89.3 vs 89.0, n=374; SearchQA Group 2 80.6 vs 80.3, n=1,400).

more of objection 1

With stochastic API models and small test splits, these differences are not significant at conventional levels under binomial standard errors (e.g., the ALFWorld Group 1 5.3-point gap on n=134 has SE about 4.5 points). The paper should report repeated-seed scores, confidence intervals, or a significance test, or temper the 'consistent' claim.

The paper says, at Section 3.2 / Table 5

The review did not quote this passage.

Pith says: Objection 2 of 3 · major

Training splits are very small for several benchmarks (ALFWorld 39, LiveMath 35, SpreadsheetBench 80, DocVQA 107 in Table 5). With a single run of the whole pipeline, the skill library itself is a single draw from stochastic models. The reported gains could therefore reflect sampling variation in which training instances were used. Please report variance over multiple training-sample draws or at least over repeated pipeline runs, and discuss coverage of task types given the tiny ALFWorld and LiveMath training sets.

The paper says, at Section 3.1-3.2

The review did not quote this passage.

Pith says: Objection 3 of 3 · major

The acceptance criterion is r(x, Apply(p,K))=1, so the evaluator r is the sole behavior gate. If r is noisy, partial, or gameable, patches can be committed on the basis of evaluator artifacts rather than reusable behavior. Although the benchmarks use official task metrics, this assumption is load-bearing for the generalization claim.

more of objection 3

I recommend a robustness check (e.g., evaluate accepted skills with a second independent scorer or hand-audit a sample of accepted patches) to confirm they encode transferable rules rather than exact-match tricks.

4 minor objections

The paper says, at Section 3.1

Pith says: minor objection 1 of 4

Typo in the subsection heading: 'Constrastive' should be 'Contrastive'.

The paper says, at Appendix A / Table 5

Pith says: minor objection 2 of 4

Please clarify how the ALFWorld and LiveMath splits were derived; if official splits were not used wholesale, state the sampling procedure and seed.

show 2 more

The paper says, at Experiments

Pith says: minor objection 3 of 4

No code or data release is mentioned. For reproducibility, please provide code, prompts, and the final skill libraries.

The paper says, at Experiments Setting

Pith says: minor objection 4 of 4

The consolidation agent is the teacher model in both groups. An ablation using a different curator model would control for teacher-specific bias, even though acceptance is gated by student reruns.

What would settle it

Re-run the full SKILL-KD and the strongest baseline (SkillOpt) pipeline 5 times with different split seeds (e.g., seeds 0,1,2,3,4 for the deterministic 2:1:7 partition; or if decoding is stochastic, re-seed student/teacher rollouts while fixing splits) and report mean ± std for every Table 1 cell and for the macro-average. Additionally, compute 95% bootstrap CIs over test instances for the SKILL-KD − SkillOpt difference. If more than two of the ten per-benchmark comparisons have CIs including zero, or the macro-average margin over SkillOpt falls below one within-condition std, the 'consistently improves' claim should be reduced to 'improves on a subset of benchmarks.'

from the skeptic's reading

How the review went

4 readings by deepseek-v4-flash · 5 Aug 2026

  1. Reader holds up, with conditions confidence moderate
  2. Skeptic partly agrees 1 min 33 s · 15,397 tokens read, 8,414 written
  3. Editor wrote the verdict above 30 s · 16,170 tokens read, 2,811 written
  4. Referee major revision: 3 major objections, 4 minor 1 min 18 s · 15,713 tokens read, 7,858 written

Not run on this paper: the simulated authors' defence.

Pith did not check this review's objections against the paper.

The review in full

The review in brief, machine-written

Here is my take. SKILL-KD is a real idea, not a repackaged baseline. Contrasting teacher and student trajectories on the same task, then iteratively validating a textual patch by actually rerunning the student, is a genuine departure from self-reflection or one-shot summarization. The trace-linked consolidation with edit history is also a thoughtful mechanism against skill drift. The ablations are the best part: they isolate the contribution of adaptive reruns and consolidation, and the teacher-outcome diagnostic is a nice check that the method doesn't assume an oracle teacher. The final skill library in the appendix looks like generalizable rules, not memorized episodes.

But the empirical case is weaker than the abstract admits. Every score is a single run on a fixed test split (stated in Appendix A). Several decisive margins in Table 1 are within a few test instances: SpreadsheetBench Group 1 is 24.3 vs 23.9 on n=280, DocVQA Group 1 is 89.3 vs 89.0 on n=374, SearchQA Group 2 is 80.6 vs 80.3 on n=1400. Those gaps are noise. Even the ALFWorld Group 1 gap (86.6 vs 81.3, n=134) has a binomial SE of about 4.5 points, so it's not significant. The big ALFWorld gains from the no-skill baseline (30.6 to 86.6) are harder to dismiss as noise, but with 39 training instances and a single draw, the skill library itself is one sample from a stochastic teacher-student pipeline. So 'consistently improves' is an overclaim; 'tends to improve' would be honest.

The other load-bearing premise is the evaluator. A patch is accepted only when a student rerun yields r(x, Apply(p,K)) = 1. If the evaluator is noisy or partial, accepted patches can fit evaluator artifacts rather than reusable behavior. The paper doesn't discuss evaluator reliability or report variance. That's a measurement concern, distinct from the method's validity.

The paper also has free parameters (max adaptive rounds, split ratio, reasoning effort) that are reasonable but only the rounds are analyzed. No code or data are released. These are fixable.

Net: the mechanism is plausible, the ablations are informative, and the claims should be treated as provisional. This deserves a serious referee and a revision that adds repeated seeds, error bars, and ideally a released skill library and harness. I'd cite it for the method, not for the quantitative numbers.

The strongest objection

The central claim is empirical: SKILL-KD 'consistently improves' frozen students over baselines on held-out splits. The most load-bearing premise is not the evaluator's construct validity but the statistical reliability of the reported scores. Appendix A states: 'Each reported score was computed from a single complete evaluation run on the fixed test split.' Several decisive Table 1 margins are within a few test instances: SpreadsheetBench Group 1 (24.3 vs 23.9, n=280), DocVQA Group 1 (89.3 vs 89.0, n=374), SearchQA Group 2 (80.6 vs 80.3, n=1400). A binomial SE for the ALFWorld Group 1 gap (86.6 vs 81.3, n=134) is about 4.5 points, so the 5.3-point gap is not significant. With stochastic API models ('medium reasoning effort') and small training splits (ALFWorld 39, LiveMath 35), both the skill library and test scores are single draws. If the evaluator is reliable, this variance problem remains; if the evaluator is imperfect, it compounds. The paper's ablations are informative but do not provide the error bars needed to support 'consistent' gains.

Reader's evaluator concern is legitimate but secondary: for these benchmarks the gate is a standard deterministic metric (exact match / hard success), so the dominant unaddressed threat to the headline claim is sampling variance. The paper's own appendix flags the single-run protocol, and small test/train sizes make several headline margins indistinguishable from noise. A conditional verdict is appropriate until the re-run check is provided. I agree partially with the reader: the evaluator is a plausible measurement risk, but the more immediate load-bearing gap is the missing repeated-run evidence.

The referee's report

If the reported gains are reliable, SKILL-KD is a practically useful, training-free adaptation mechanism for frozen agents: it uses a natural-language skill artifact as the distillation medium, so it transfers across model families, and its acceptance gate is grounded in student reruns rather than teacher opinion. The ablations in Tables 2 and 3 are well-designed: they separate signal sources (teacher-only, student-only, pairwise, batch) and show that adaptive validation and trace-linked consolidation improve both effectiveness and library compactness (38 rules, 3,010 words vs. 96 rules, 6,821 words for Student-only). The teacher-failure diagnostics in Table 4 and round-sensitivity analysis in Table 6 are useful. The central weakness is statistical: all scores are from single evaluation runs, several decisive margins are small, and training splits are tiny, so the 'consistently improves' claim is not yet established with confidence.

Pith's reading of the paper (a machine's summary, not the authors')

SKILL-KD claims that the right learning signal for an LLM agent skill library is not the student's own failure, nor the teacher's success, but the contrast between the two trajectories on the same task. The paper proposes to convert that contrast into a textual skill patch, validate it by re-running the frozen student, and only commit patches that demonstrably change the student's behavior. A second mechanism, drift-aware consolidation, keeps the growing library compact by letting a curator agent inspect trace-linked edit histories before each add, modify, delete, or skip decision. The central claim is that this recipe transfers procedural knowledge across models of different sizes and families without weight updates; the paper reports consistent held-out gains over no-skill and skill-learning baselines on five benchmarks.

What carries the argument. The central object is the skill patch p = (title, content, why, trace), a textual rule that is appended to a prompt-time skill library shared by student and teacher. The carrying mechanism is the contrastive distillation loop: on a student failure, a consolidation agent writes a candidate patch from the student-teacher trajectory pair, the frozen student is rerun with the patched library, and the loop repeats (up to n rounds) until the student succeeds; only behaviorally validated patches are committed. The second mechanism is drift-aware consolidation, an edit history where each entry records the operation, the patch, and a trace link to the trajectories that motivated it, queried through a

Core claim. SKILL-KD's central claim is that the actionable gap between a student's failed trajectory and a teacher's trajectory on the same task is the right learning signal for skill acquisition, and that this signal must be converted into a textual skill patch through iterative validation rather than one-shot summarization. The paper defines a skill as a four-field record (title, content, why, trace), where content is the runtime instruction and trace indexes the originating trajectories. A patch is accepted only if re-running the frozen student with the patched skill library yields a perfect evaluator score on that instance. To keep the library compact and stable, the consolidation agent maintains a

The premise it rests on. The pipeline assumes the task evaluator r(x, tau) correctly and completely decides whether a trajectory solves the task, because a patch is committed only when a student rerun scores 1.

If the paper is right (Pith's inferences, not the authors')

Frozen student agents can be improved purely through prompt-time skill artifacts, without fine-tuning or weight updates.

Contrastive information from failed teacher trajectories remains useful: 38.5% of accepted patches derive from cases where the teacher also failed.

The first adaptive validation round provides most of the gain, so bounded reruns (n=3) suffice.

Drift-aware consolidation produces compact libraries (38 rules, 3,010 words in Group 1) while improving accuracy.

The method transfers across model families (Qwen to ChatGPT-5.5), indicating the distillation carrier is model-agnostic.

A testable implication is that SKILL-KD's gains should be sensitive to the quality of the evaluator: with a noisy or partial scorer, accepted patches may fit scorer artifacts; one could measure this by corrupting a fraction of training evaluator scores and checking held-out clean-score performance.

The trace-linked consolidation could be adapted to other evolving artifact stores, such as codebases or prompt libraries, serving as a general anti-drift mechanism for iterative language-based editing.

The paper does not study retrieval routing; using the full skill file in the prompt caps the library at roughly a few thousand tokens. A learned retrieval layer could extend the method to larger skill inventories, but would add a selection bias the current experiments avoid.

Check this review

Pith stored no record of this review's passes and took no fingerprint of it. Reviews made by code from 13 Nov 2026 on carry both.

How often reviews like this one are right

Reviews like this one: written by deepseek-v4-flash, where what was read was not recorded.

How often objections from reviews like this one are right: not yet measured

Every value above, in the review's public record (JSON)

Reviews by people

This review was written by a machine. No person has reviewed this paper on Pith yet. A scientist listed on Pith can review it for a fee, and their review is published here whatever it concludes.

Request a human review

How Pith checks papers · The full audit of this paper · Review history (2 versions)

The paper

Abstract

Skill-based prompting has become a practical mechanism for improving LLM agents, yet existing methods often treat skills as summaries of the agent's own experience or of successful demonstrations. This creates a mismatch for weaker student agents. A failed trajectory may not reveal the missing knowledge or strategy, while a teacher trajectory may be too implicit to internalize. We propose SKILL-KD, a contrastive skill distillation framework that treats skills as an explicit distillation medium between agents of different capabilities. Given a student failure and the teacher trajectory on the same task, SKILL-KD distills their actionable discrepancy into a textual skill patch, evaluates it by re-running the student, and iteratively refines it when the student still fails. To prevent skill drift from repeated local updates, SKILL-KD maintains trace-linked edit histories and performs Drift-Aware Skill Consolidation to decide whether each patch is added, merged, or skipped. Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines.

The paper's own sentences, arranged, kept beside the review and separate from it. Each label is drawn from a closed vocabulary; the number beside it is the probability the classifier gave that label. No sentence in this record was written by a model.

abstract compiled · 2026-09-23

The authors' main claim, as the classifier picked it out 1.00

“Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines.”

How the main claim is stated: asserted 1.00

What the abstract itself states, as probabilities: claims first 0.06 · states limitation 0.11 · human participants 0.04 · controlled comparison 0.93 · randomized assignment 0.02 · states quantitative result 0.24 · compares with prior results 0.93 · mentions code or data availability 0.02

The abstract, sentence by sentence

This table labels the abstract only (arXiv listing abstract v2). The number is the probability the classifier gave the label.

SentenceRoleThe authors' own workStates the main conclusionStrength
Skill-based prompting has become a practical mechanism for improving large language model (LLM) agents, yet existing skill acquisition methods often treat skills as experience summaries, memory entries, or direct summaries of successful demonstrations. background 0.97 0.05 0.02 asserted 0.99
This creates a mismatch for weaker student agents: when a student fails because it lacks task knowledge or operational strategy, its failed trajectory may not contain enough evidence to infer the missing behavior, while the teacher trajectory may be too implicit to be internalized as reusable guidance. gap 0.81 0.08 0.03 hedged possibility 0.55
We propose SKILL-KD, a contrastive skill distillation framework that treats skills as an explicit distillation medium between agents of different capabilities. contribution 0.98 0.85 0.11 asserted 0.99
Given a student failure and the teacher trajectory on the same task, SKILL-KD distills their actionable discrepancy into a textual skill patch, evaluates the patch by re-running the student, and iteratively refines the patch when the student still fails. method 0.99 0.91 0.08 asserted 1.00
To prevent repeated local updates from causing skill drift, SKILL-KD further maintains trace-linked edit histories and performs Drift-Aware Skill Consolidation, deciding whether each patch should add a new rule, delete or modify an existing rule, or be skipped. method 0.97 0.91 0.04 asserted 1.00
Across five agent benchmarks and two student settings, SKILL-KD consistently improves frozen student agents over fixed-model adaptation baselines. result 1.00 0.96 0.82 asserted 1.00

What kind of paper the classifier reports this is: method or tool 1.00 · evidence from observation or experiment 0.96

What was examined, and what was not

Sources read

  • arXiv listing abstract v2

Not examined

  • full text
  • figures and tables
  • references and citing sentences
  • relations to other papers

Levels run

  • P abstract

Not run

  • S sections
  • T body sentences
  • C citations
  • R cross-paper relations

How the labels were made

Compiled by classifier-1.13.0 via TypeSafe API, question bank abstract-p1-20260923, 2026-09-23: 1 request, 5,146 input tokens.

Every quoted sentence on this record is the authors'. Every label is from Pith's closed vocabulary, with the probability shown beside it. This record as JSON

Figures (3)
Figure 1 of the paper
Figure 1. SKILL-KD overcomes insufficient self-experience
Figure 2 of the paper
Figure 2. Overview of SKILL-KD. Given a task instance, the student and teacher generate contrasting trajectories. The
Figure 3 of the paper
Figure 3. Marginal percentage-point gains from successive

Figures are the authors', reproduced from the paper.

Cited by 4 Pith papers
How the literature uses it

What this paper does with the arXiv papers it cites

Each citing sentence is taken from the citing paper's LaTeX source and typed by Pith's classifier. A label appears at probability 0.8 or more (0.95 for support), a contest at 0.7 or more; the quoted words are the citing authors'. How each label was checked · JSON

What it cites · 9 sentences without a confident label

Uses data or values from 2 papers
Cites as background 13 papers

Challenges and discussion

Disagree with the review, or know something it missed? Leave a comment, or challenge this review formally: a challenge becomes a permanent entry in this paper's record, and its decision is published here.

No comments yet.

Sign in to comment