{"id":"8a2c5eca-33b9-4518-a643-3a1934b8d428","arxiv_id":"2607.28048","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SKILL-KD converts teacher-student trajectory contrasts into validated textual skill rules, improving frozen LLM agents on five agent benchmarks over skill-learning baselines.","lead":"This paper presents SKILL-KD, a method that improves weaker AI agents by comparing the weaker agent's failed attempt with a stronger teacher's trajectory on the same task, then turning the difference into a written skill rule. The method tests each rule by rerunning the student, keeps only rules that work, and maintains an edit history to avoid skill drift, reporting gains on five agent benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Single-run evaluation with small test splits leaves the 'consistently improves' claim statistically unsupported; several benchmark margins are within a few instances.","rationale":"Reader's evaluator concern is legitimate but secondary: for these benchmarks the gate is a standard deterministic metric (exact match / hard success), so the dominant unaddressed threat to the headline claim is sampling variance. The paper's own appendix flags the single-run protocol, and small test/train sizes make several headline margins indistinguishable from noise. A conditional verdict is appropriate until the re-run check is provided. I agree partially with the reader: the evaluator is a plausible measurement risk, but the more immediate load-bearing gap is the missing repeated-run evidence.","tokens_in":15397,"tokens_out":8414,"duration_ms":93337,"concrete_test":"Re-run the full SKILL-KD and the strongest baseline (SkillOpt) pipeline 5 times with different split seeds (e.g., seeds 0,1,2,3,4 for the deterministic 2:1:7 partition; or if decoding is stochastic, re-seed student/teacher rollouts while fixing splits) and report mean ± std for every Table 1 cell and for the macro-average. Additionally, compute 95% bootstrap CIs over test instances for the SKILL-KD − SkillOpt difference. If more than two of the ten per-benchmark comparisons have CIs including zero, or the macro-average margin over SkillOpt falls below one within-condition std, the 'consistently improves' claim should be reduced to 'improves on a subset of benchmarks.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is empirical: SKILL-KD 'consistently improves' frozen students over baselines on held-out splits. The most load-bearing premise is not the evaluator's construct validity but the statistical reliability of the reported scores. Appendix A states: 'Each reported score was computed from a single complete evaluation run on the fixed test split.' Several decisive Table 1 margins are within a few test instances: SpreadsheetBench Group 1 (24.3 vs 23.9, n=280), DocVQA Group 1 (89.3 vs 89.0, n=374), SearchQA Group 2 (80.6 vs 80.3, n=1400). A binomial SE for the ALFWorld Group 1 gap (86.6 vs 81.3, n=134) is about 4.5 points, so the 5.3-point gap is not significant. With stochastic API models ('medium reasoning effort') and small training splits (ALFWorld 39, LiveMath 35), both the skill library and test scores are single draws. If the evaluator is reliable, this variance problem remains; if the evaluator is imperfect, it compounds. The paper's ablations are informative but do not provide the error bars needed to support 'consistent' gains.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces SKILL-KD, a framework for improving a frozen LLM agent by distilling teacher-student trajectory contrasts into textual skill patches. For each training instance, if the student fails, the teacher runs on the same task; a consolidation agent proposes a skill patch from the contrast, the student is rerun with the patched library, and the patch is refined up to n adaptive rounds until the student succeeds or the budget is exhausted. A drift-aware consolidation step maintains trace-linked edit histories and decides add/modify/delete/skip operations to keep the skill library compact. The method is evaluated on five benchmarks (SearchQA, SpreadsheetBench, DocVQA, LiveMath, ALFWorld) with two student-teacher pairs, comparing against No Skill, EvoSkill, Trace2Skill, SkillGen, and SkillOpt. The paper reports consistent numerical gains in Table 1, ablations isolating adaptive validation and consolidation, and sensitivity analyses of adaptive rounds and imperfect teacher signals.","tokens_in":15713,"tokens_out":7858,"duration_ms":78122,"significance":"If the reported gains are reliable, SKILL-KD is a practically useful, training-free adaptation mechanism for frozen agents: it uses a natural-language skill artifact as the distillation medium, so it transfers across model families, and its acceptance gate is grounded in student reruns rather than teacher opinion. The ablations in Tables 2 and 3 are well-designed: they separate signal sources (teacher-only, student-only, pairwise, batch) and show that adaptive validation and trace-linked consolidation improve both effectiveness and library compactness (38 rules, 3,010 words vs. 96 rules, 6,821 words for Student-only). The teacher-failure diagnostics in Table 4 and round-sensitivity analysis in Table 6 are useful. The central weakness is statistical: all scores are from single evaluation runs, several decisive margins are small, and training splits are tiny, so the 'consistently improves' claim is not yet established with confidence.","major_comments":[{"comment":"The central claim of 'consistent improvement' is an empirical statement, but Appendix A states that 'Each reported score was computed from a single complete evaluation run on the fixed test split.' Several decisive margins in Table 1 are within a few test instances (e.g., SpreadsheetBench Group 1 24.3 vs 23.9, n=280; DocVQA Group 1 89.3 vs 89.0, n=374; SearchQA Group 2 80.6 vs 80.3, n=1,400). With stochastic API models and small test splits, these differences are not significant at conventional levels under binomial standard errors (e.g., the ALFWorld Group 1 5.3-point gap on n=134 has SE about 4.5 points). The paper should report repeated-seed scores, confidence intervals, or a significance test, or temper the 'consistent' claim.","section":"Table 1 / Appendix A"},{"comment":"Training splits are very small for several benchmarks (ALFWorld 39, LiveMath 35, SpreadsheetBench 80, DocVQA 107 in Table 5). With a single run of the whole pipeline, the skill library itself is a single draw from stochastic models. The reported gains could therefore reflect sampling variation in which training instances were used. Please report variance over multiple training-sample draws or at least over repeated pipeline runs, and discuss coverage of task types given the tiny ALFWorld and LiveMath training sets.","section":"Section 3.2 / Table 5"},{"comment":"The acceptance criterion is r(x, Apply(p,K))=1, so the evaluator r is the sole behavior gate. If r is noisy, partial, or gameable, patches can be committed on the basis of evaluator artifacts rather than reusable behavior. Although the benchmarks use official task metrics, this assumption is load-bearing for the generalization claim. I recommend a robustness check (e.g., evaluate accepted skills with a second independent scorer or hand-audit a sample of accepted patches) to confirm they encode transferable rules rather than exact-match tricks.","section":"Section 3.1-3.2"}],"minor_comments":[{"comment":"Typo in the subsection heading: 'Constrastive' should be 'Contrastive'.","section":"Section 3.1"},{"comment":"Please clarify how the ALFWorld and LiveMath splits were derived; if official splits were not used wholesale, state the sampling procedure and seed.","section":"Appendix A / Table 5"},{"comment":"No code or data release is mentioned. For reproducibility, please provide code, prompts, and the final skill libraries.","section":"Experiments"},{"comment":"The consolidation agent is the teacher model in both groups. An ablation using a different curator model would control for teacher-specific bias, even though acceptance is gated by student reruns.","section":"Experiments Setting"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take. SKILL-KD is a real idea, not a repackaged baseline. Contrasting teacher and student trajectories on the same task, then iteratively validating a textual patch by actually rerunning the student, is a genuine departure from self-reflection or one-shot summarization. The trace-linked consolidation with edit history is also a thoughtful mechanism against skill drift. The ablations are the best part: they isolate the contribution of adaptive reruns and consolidation, and the teacher-outcome diagnostic is a nice check that the method doesn't assume an oracle teacher. The final skill library in the appendix looks like generalizable rules, not memorized episodes.\n\nBut the empirical case is weaker than the abstract admits. Every score is a single run on a fixed test split (stated in Appendix A). Several decisive margins in Table 1 are within a few test instances: SpreadsheetBench Group 1 is 24.3 vs 23.9 on n=280, DocVQA Group 1 is 89.3 vs 89.0 on n=374, SearchQA Group 2 is 80.6 vs 80.3 on n=1400. Those gaps are noise. Even the ALFWorld Group 1 gap (86.6 vs 81.3, n=134) has a binomial SE of about 4.5 points, so it's not significant. The big ALFWorld gains from the no-skill baseline (30.6 to 86.6) are harder to dismiss as noise, but with 39 training instances and a single draw, the skill library itself is one sample from a stochastic teacher-student pipeline. So 'consistently improves' is an overclaim; 'tends to improve' would be honest.\n\nThe other load-bearing premise is the evaluator. A patch is accepted only when a student rerun yields r(x, Apply(p,K)) = 1. If the evaluator is noisy or partial, accepted patches can fit evaluator artifacts rather than reusable behavior. The paper doesn't discuss evaluator reliability or report variance. That's a measurement concern, distinct from the method's validity.\n\nThe paper also has free parameters (max adaptive rounds, split ratio, reasoning effort) that are reasonable but only the rounds are analyzed. No code or data are released. These are fixable.\n\nNet: the mechanism is plausible, the ablations are informative, and the claims should be treated as provisional. This deserves a serious referee and a revision that adds repeated seeds, error bars, and ideally a released skill library and harness. I'd cite it for the method, not for the quantitative numbers.","headline":"Worth a serious look: a plausible, well-ablated distillation method whose 'consistent gains' claim is statistically undersupported by single-run, small-split evaluation.","tokens_in":16170,"tokens_out":2811,"duration_ms":30392},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SKILL-KD turns failed student episodes into reusable skills by contrasting them with teacher trajectories.","keywords":["skill distillation","LLM agents","skill library","knowledge distillation","prompt-time adaptation","drift-aware consolidation","contrastive learning","agent benchmarks"],"falsifier":"Run SKILL-KD on a benchmark while corrupting the training-set evaluator on a random subset of instances (e.g., flipping scores), then evaluate the resulting skill library on a held-out test set with the clean, original evaluator. If the held-out gains persist under evaluator noise, the claim that skills capture reusable behavior survives; if gains collapse, the measured improvements were at least partly artifacts of the scorer.","tokens_in":15332,"feed_emoji":"🧠","tokens_out":4595,"duration_ms":44532,"temperature":0.7,"pith_summary":"SKILL-KD claims that the right learning signal for an LLM agent skill library is not the student's own failure, nor the teacher's success, but the contrast between the two trajectories on the same task. The paper proposes to convert that contrast into a textual skill patch, validate it by re-running the frozen student, and only commit patches that demonstrably change the student's behavior. A second mechanism, drift-aware consolidation, keeps the growing library compact by letting a curator agent inspect trace-linked edit histories before each add, modify, delete, or skip decision. The central claim is that this recipe transfers procedural knowledge across models of different sizes and families without weight updates; the paper reports consistent held-out gains over no-skill and skill-learning baselines on five benchmarks.","feed_headline":"Failed attempts, distilled against teacher runs, lift frozen agents","feed_subtitle":"Five benchmarks, two student models: contrastive skill distillation beats summary-based skill learning without retraining.","key_machinery":"The central object is the skill patch p = (title, content, why, trace), a textual rule that is appended to a prompt-time skill library shared by student and teacher. The carrying mechanism is the contrastive distillation loop: on a student failure, a consolidation agent writes a candidate patch from the student-teacher trajectory pair, the frozen student is rerun with the patched library, and the loop repeats (up to n rounds) until the student succeeds; only behaviorally validated patches are committed. The second mechanism is drift-aware consolidation, an edit history where each entry records the operation, the patch, and a trace link to the trajectories that motivated it, queried through a","core_discovery":"SKILL-KD's central claim is that the actionable gap between a student's failed trajectory and a teacher's trajectory on the same task is the right learning signal for skill acquisition, and that this signal must be converted into a textual skill patch through iterative validation rather than one-shot summarization. The paper defines a skill as a four-field record (title, content, why, trace), where content is the runtime instruction and trace indexes the originating trajectories. A patch is accepted only if re-running the frozen student with the patched skill library yields a perfect evaluator score on that instance. To keep the library compact and stable, the consolidation agent maintains a","pith_inferences":["A testable implication is that SKILL-KD's gains should be sensitive to the quality of the evaluator: with a noisy or partial scorer, accepted patches may fit scorer artifacts; one could measure this by corrupting a fraction of training evaluator scores and checking held-out clean-score performance.","The trace-linked consolidation could be adapted to other evolving artifact stores, such as codebases or prompt libraries, serving as a general anti-drift mechanism for iterative language-based editing.","The paper does not study retrieval routing; using the full skill file in the prompt caps the library at roughly a few thousand tokens. A learned retrieval layer could extend the method to larger skill inventories, but would add a selection bias the current experiments avoid."],"forward_implications":["Frozen student agents can be improved purely through prompt-time skill artifacts, without fine-tuning or weight updates.","Contrastive information from failed teacher trajectories remains useful: 38.5% of accepted patches derive from cases where the teacher also failed.","The first adaptive validation round provides most of the gain, so bounded reruns (n=3) suffice.","Drift-aware consolidation produces compact libraries (38 rules, 3,010 words in Group 1) while improving accuracy.","The method transfers across model families (Qwen to ChatGPT-5.5), indicating the distillation carrier is model-agnostic."],"supporting_citations":[{"why":"Establishes persistent skill libraries for LLM agents, the foundation SKILL-KD builds on.","marker":"(Wang et al. 2023)"},{"why":"Provides the self-reflection baseline signal that SKILL-KD contrasts with teacher-student discrepancies.","marker":"(Shinn et al. 2023)"},{"why":"Experience memory baseline that SKILL-KD's contrastive and consolidation design is compared against.","marker":"(Zhao et al. 2024)"},{"why":"Trace2Skill baseline consolidates broad trajectories into skills; SKILL-KD must beat it.","marker":"(Ni et al. 2026)"},{"why":"SkillGen baseline uses intervention-style verification; SKILL-KD's adaptive validation is contrasted with it.","marker":"(Ma et al. 2026)"},{"why":"SkillOpt baseline treats the skill document as optimizable state; SKILL-KD's drift-aware consolidation is positioned against it.","marker":"(Yang et al. 2026c)"},{"why":"EvoSkill baseline proposes new skills from execution failures; a direct comparison in the main results.","marker":"(Alzubi et al. 2026)"}],"fun_headline_variants":["Contrastive skill patches from student-teacher gaps lift agents","Convert failed runs into skill patches via contrastive distillation","Learn agent skills by contrasting failed and expert trajectories","Turn student failures into reusable skills with contrastive KD","Skill patches distilled from failure gaps boost frozen LLM agents"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The pipeline assumes the task evaluator r(x, tau) correctly and completely decides whether a trajectory solves the task, because a patch is committed only when a student rerun scores 1.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive skill patches from student-teacher gaps lift agents","Convert failed runs into skill patches via contrastive distillation","Learn agent skills by contrasting failed and expert trajectories","Turn student failures into reusable skills with contrastive KD","Skill patches distilled from failure gaps boost frozen LLM agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000147,"raw_usage":{"total_tokens":1016,"prompt_tokens":732,"completion_tokens":284,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":476,"completion_tokens_details":{"reasoning_tokens":206}},"tokens_in":476,"tokens_out":284,"duration_ms":4239,"temperature":1.0,"reasoning_tokens":206,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T04:20:50.654156+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.3.0-alltime-deepseek","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run SKILL-KD on a benchmark while corrupting the training-set evaluator on a random subset of instances (e.g., flipping scores), then evaluate the resulting skill library on a held-out test set with the clean, original evaluator. If the held-out gains persist under evaluator noise, the claim that skills capture reusable behavior survives; if gains collapse, the measured improvements were at least partly artifacts of the scorer.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":2}