{"id":"2a4820c2-4fd9-4edb-891e-95db0c2a1224","arxiv_id":"2607.26643","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"SkillBoost reduces skill overfitting in self-evolving LLM agents by combining failure-localized editing, multi-candidate generation, and an anti-regression acceptance gate.","lead":"This paper proposes SkillBoost, a three-stage method for refining an LLM agent's reusable skills without retraining the model: it locates failures in specific skill components, generates several repair options, and only accepts edits that pass a regression check. The method reports large gains across five agent benchmarks and keeps the gap between training and held-out performance near zero, suggesting learned skills generalize instead of memorizing.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The near-zero overfitting claim rests on an unverifiable, potentially selected Δ=Test−Train: Train values and run-level variance are unreported, and the same D drives selection, acceptance, and hyperparameter choice.","rationale":"The reader's CONDITIONAL verdict identifies the same underlying evaluation weakness: D is used both for selection and for the regression gate, Train values are unreported, and no code/data are available. My stress-test sharpens this into the specific claim that Table III cannot distinguish a genuinely generalizing skill from a gate that caps regressions on a small or unrepresentative D, especially since the appendix shows train splits as small as 35 questions and hyperparameters such as N=4 and ε=0.05 are selected on the same benchmarks. I do not find an internal mathematical contradiction or a fatal flaw in the framework: the acceptance gate is a plausible regularization mechanism, the ablations are consistent, and the transfer experiments add supporting evidence. The concern is about evidence strength and verifiability, not about the method's soundness. Therefore the appropriate verdict remains CONDITIONAL — the central claim could be validated by code/data release and a held-out tuning protocol, but is not independently checkable as reported. My agreement is partial because the reader emphasizes D's representativeness as a deployment proxy, whereas I emphasize the unverifiability and selection of the Δ statistic itself; these are closely related but not identical concerns.","tokens_in":28652,"tokens_out":6144,"duration_ms":70277,"concrete_test":"Release code and per-configuration Train/Test tables. Then pre-register a protocol: evolution sees only Train; N, ε, and top-K are chosen on a held-out validation subset disjoint from Test; each configuration is run with at least 5 seeds (temperature is already 0.1); report bootstrap 95% CIs for Δ=Test−Train per configuration and a pooled estimate. If mean Δ remains within ±1 percentage point with CIs excluding large negative values across all 23 configurations, the concern is resolved. If Train values are small or CIs are wide, the near-zero-gap claim should be downgraded.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The core empirical assertion — SkillBoost 'reduces overfitting to a near-zero test–train gap' — rests on Table III, which reports only Δ = Test − Train. The Train column, per-configuration question counts, and run-level variance are not given, and no code/data are released. This is not a cosmetic omission: D is the rollout set, the candidate-selection set, and the regression-gate set, so the gate directly constrains the Train side of the gap. If a train split is small (the appendix's LiveMath example uses 35 questions), a Δ of ±1–3 points is within sampling noise and cannot substantiate 'near-zero overfitting.' In addition, Table VII/Figure 4 use the same benchmarks to select N=4, and ε=0.05 is fixed without sensitivity analysis; the headline numbers are therefore partly selected statistics. The method may be sound — the ablations and transfer results are consistent — but the central overfitting claim is not independently checkable as reported.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that skill self-evolution for LLM agents should be reframed as a constrained exploration–exploitation search rather than trajectory fitting. It proposes SkillBoost, a three-stage loop: structured exploitation attributes failures to specific skill components using trajectory evidence; prior-guided exploration generates N repair candidates through different repair strategies; verified acceptance commits a candidate only if it improves the full-set score and keeps regressions below a threshold. The authors report results on 23 model–benchmark configurations (20 text-based pairs plus three DocVQA pairs), component ablations, Best-of-N analyses, cross-model and cross-benchmark transfer experiments, and a theoretical lower bound on iteration complexity under a uniform-sampling model. The central claim is that SkillBoost achieves state-of-the-art performance while reducing the test–train generalization gap to near zero.","tokens_in":28879,"tokens_out":8275,"duration_ms":87113,"significance":"If the empirical claims hold, this is a useful contribution: it identifies skill overfitting as a distinct failure mode and shows that a generate-and-test loop with an explicit regression gate can produce large held-out gains across frontier models and diverse benchmarks. The paper's strengths include a broad evaluation across 20 model–benchmark pairs, component-level ablations, transfer tests, and a transparent acceptance criterion. The configuration-level statistical analysis is also a reasonable first step. However, the headline overfitting claim is under-supported as reported: the train accuracies behind the reported Δ values are not given, no per-run or per-question variance is reported, no code or data are released, and key hyperparameters (ε, N, T) are selected on the same benchmarks used for the main results. These gaps make the central 'near-zero overfitting' assertion not independently checkable from the manuscript.","major_comments":[{"comment":"The central claim that SkillBoost 'reduces overfitting to a near-zero test–train gap' rests entirely on Δ = Test − Train in Table III, but the Train values are not reported. Since D is the same split used for rollout, candidate selection, and the regression gate, the gate directly constrains the Train side of this gap, so a near-zero Δ could arise from a low Train accuracy rather than a genuinely high Test accuracy. Several SkillBoost entries are negative (e.g., SpreadsheetBench/Kimi-k2.6: −3.8; ALFWorld/Kimi-k2.6: −1.2), and the appendix's LiveMath example uses only 35 training questions, where ±1–3 points is within sampling noise. Please report Train and Test accuracies, per-split question counts, and confidence intervals (per-question bootstrap or per-seed variance) for every configuration.","section":"Overfitting Analysis (Table III)"},{"comment":"The headline results are partly selected statistics. N=4 is chosen from the saturation curves in Figure 4/Table VII on SpreadsheetBench and ALFWorld—the same benchmarks that appear in the main results of Table I—and the regression threshold ε=0.05 is fixed with no sensitivity analysis. Because ε directly controls which candidates are committed, the authors should show how accuracy and Δ vary with ε (e.g., ε=0.0, 0.02, 0.05, 0.10), and should justify N=4 on a validation split or through a protocol that does not reuse the headline test benchmarks for hyperparameter selection.","section":"Verified Acceptance (ε) and Effect of Candidate Pool Size N (Table VII, Figure 4)"},{"comment":"No code, data, prompts, or generated skill files are released, and Table XI does not list the number of evolution iterations T used in Algorithm 1 (Figure 3 suggests five skill versions, but this is not stated). Together with the missing train-side numbers, this means the paper's main empirical claims cannot be independently reproduced or checked. I would consider this acceptable only if the authors release the skill-evolution harness, the exact prompts for the analyzer/generator, and the per-configuration train/test scores; otherwise the empirical contribution remains unverifiable.","section":"Reproducibility (Experiments/Appendix)"}],"minor_comments":[{"comment":"The sentence 'per-model gains ranging from +10.6 to +28.4' is ambiguous: Table I's gains relative to No-Skill range from +2.3 to +47.4. Please specify the baseline and whether these are averages over benchmarks.","section":"Performance Comparison"},{"comment":"The symbol ε is used both for the regression threshold in Verified Acceptance (ε=0.05) and for the radius of the candidate ball in the theoretical analysis. These are different quantities and should use different symbols to avoid confusion.","section":"Theoretical Analysis"},{"comment":"Table XI omits T, the number of evolution iterations. Please state the value used (Figure 3 implies five versions). Also clarify the relationship between the stated 'roughly 2.5× a single full evaluation' cost and the formula C(N)≈N|F|τ_fail + 2|D|τ_full.","section":"Table XI"},{"comment":"The text refers to 'Table 1' instead of 'Table I.' The paired t-test and Wilcoxon test treat each model–benchmark pair as one observation; this is acknowledged, but the reader should be reminded that this does not capture per-question sampling variance, which is precisely the variance relevant to the Δ=Test−Train claim.","section":"Appendix: Statistical Analysis"},{"comment":"Figure 1 has no labeled axes, and the caption says 'six skill versions' while Figure 3 shows five. Please align these details and label the axes.","section":"Figures and captions"}],"recommendation":"major_revision","confidential_remarks":"The paper is promising and the empirical scope is strong, but the central overfitting claim is not currently verifiable from the reported numbers. The missing train accuracies, lack of per-run variance, and unreleased code/data are the main risks. I would encourage a revision that adds the train-side numbers, confidence intervals, and sensitivity analyses for ε and N, and ideally releases the code and prompts. The theoretical analysis is a nice addition but is not the load-bearing part of the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Bottom line: this is a competent and genuinely useful take on skill overfitting in self-evolving agents, but the marquee claim — near-zero test–train gap — is not verifiable from the paper as written. I'd send it to review, but I'd insist on code/data and full Train numbers.\n\nThe actual contribution is the three-stage loop: structured exploitation localizes failures to skill components, prior-guided exploration generates N candidates sharing the same diagnosis, and verified acceptance commits only when gain is positive and regressions are below ε. That Fix/Regress decomposition is simple and usable. The evaluation is broad — 20 model–benchmark pairs plus three DocVQA settings — and the ablations show each stage earns its keep. The gate ablation (w/o Gate drops accuracy everywhere) is persuasive. Cross-model and cross-benchmark transfer are a nice bonus, and the appendix's concrete skill diffs make the mechanism tangible. The MDP framing is mostly repackaging; the real substance is the acceptance gate.\n\nThe soft spots are the ones you flagged, and they land. Table III reports only Δ = Test − Train, never Train, so the near-zero overfitting claim is a number without a denominator. The same D is used for rollout, candidate selection, and the regression gate, so the Train side of the gap is directly optimized by construction; r>0 means Train accuracy strictly rises on every accepted edit. A small train split (the LiveMath example uses 35 questions) makes a ±1–3 point Δ sampling noise. No code or data, no per-run or per-question variance, and the statistical test is at the model–benchmark level, which doesn't address per-question uncertainty. Hyperparameters N=4 and ε=0.05 are chosen on the same benchmarks used for headline results, with no held-out tuning. None of this makes the method wrong; it makes the central claim uncheckable.\n\nTheorem 1 is explicitly based on uniform independent sampling, and the authors concede real candidates are correlated, so treat it as intuition, not a guarantee.\n\nWho's this for? Anyone building skill libraries or test-time learning loops for LLM agents. It deserves a serious referee; the conditional should be resolvable with code, full Table III, and a few sensitivity runs.","headline":"SkillBoost is a well-engineered answer to skill overfitting in self-evolving agents, but the headline near-zero overfitting claim is not independently checkable as reported.","tokens_in":29400,"tokens_out":2949,"would_cite":true,"duration_ms":31026,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillBoost recasts LLM skill self-evolution as a constrained exploration–exploitation search, not a data-fitting problem, and reports that this keeps the test–train gap near zero.","keywords":["skill self-evolution","LLM agents","skill overfitting","exploration-exploitation","Best-of-N","regression gate","agentic skills","skill transfer"],"falsifier":"Take a skill version accepted by the gate (r>0, Regress<0.05 on the training split) and evaluate it on a new disjoint sample from the same task family, or a lightly perturbed version of the benchmark. If the test-minus-train gap turns strongly negative, or if a held-out set's solved-case regressions exceed its fixes, then the training split is not a sufficient proxy and the claimed overfitting control does not generalize. A cheaper check: rerun the loop with ε=0 and with ε=0.05; if performance and generalization are identical, the regression bound is not doing the work claimed.","tokens_in":28505,"feed_emoji":"🔧","tokens_out":7342,"duration_ms":70995,"temperature":0.7,"pith_summary":"This paper argues that when an LLM agent improves itself by refining reusable skills, the process should be treated as a constrained exploration–exploitation search rather than fitting the skill to observed trajectories. The proposed SkillBoost loop attributes each failure to a specific skill component, generates several repair candidates that share that diagnosis but differ in repair strategy, and commits an edit only if it improves the full split while keeping regressions below a fixed bound. The authors report consistent gains across 23 model–benchmark configurations, a near-zero test–train gap, and skills that transfer to other agents and related benchmarks. The contribution, if it holds, is a lightweight path for frozen-parameter agents to keep improving in deployment without memorizing the current batch.","feed_headline":"SkillBoost keeps LLM skills from overfitting the training batch","feed_subtitle":"Localize the failure, explore candidate fixes, accept only verified gains — test and train scores stay aligned.","key_machinery":"The load-bearing mechanism is the verified-acceptance gate, Accept(s′) ⇔ r(s′)>0 ∧ Regress(s′)<ε, applied to a Best-of-N pool of candidate skills. Because r(s′) decomposes as Fix(s′)−Regress(s′), the first condition is exactly \"fix more than it breaks,\" and the second caps how many previously correct cases may regress. This gate, together with a diagnosis step that restricts edits to failure-attributed skill components, turns skill mutation into constrained search. The theoretical complement is a Best-of-N sampling bound: with N independent candidate directions, expected per-step progress scales like ε·sqrt(2 ln N / d), so gains from enlarging the pool saturate quickly.","core_discovery":"The central claim is that skill overfitting — gains on the current trajectory batch that fail to transfer — is not mainly a data-quantity problem but a consequence of optimizing the wrong objective. SkillBoost therefore casts self-evolution as a Markov decision process over skill states and runs three stages each round: structured exploitation analyzes failed trajectories to produce an evidence-grounded diagnosis naming which skill components to edit; prior-guided exploration keeps that diagnosis fixed and generates N candidate skills through different repair strategies; verified acceptance evaluates candidates on the training split and accepts only updates with r(s′)>0 and Regress(s′)<ε. Th","pith_inferences":["Editorial: because the acceptance gate and the selection metric use the same training split, the method's overfitting control is only as strong as that split's representativeness; adding a small held-out validation set to the acceptance rule would test this directly.","Editorial: the learned rules read as behavioral disciplines the model already possesses but applies unreliably zero-shot; a testable extension is whether compact versions of these rules give similar gains for smaller models or non-frontier executors.","Editorial: the Best-of-N theoretical model assumes independent uniform sampling, which real LLM-generated candidates violate; measuring per-round progress across N=2,4,8,16 on several tasks would show how early saturation occurs in practice."],"forward_implications":["Accepted edits are guaranteed by construction to fix more cases than they break on the training split while keeping regressions under the bound, which is what keeps the reported test–train gap near zero.","Skills evolve into compact procedural rules — exact function-name matching, static-value spreadsheet discipline, amnesia-robust exploration — rather than growing monotonically.","Best-of-N exploration shows diminishing returns as N grows; the paper finds N=4 captures most of the gain at moderate cost.","Optimized skills can be reused by other agents and on related benchmarks, outperforming human-written skills in the paper's transfer experiments.","The framework inherits the base model's ability to attribute failures, generate edits, and evaluate candidates; the paper states that weaker models may limit self-evolution quality."],"fun_headline_variants":["SkillBoost curbs LLM skill overfitting with constrained search","Constrained exploration stops LLM skills from overfitting","SkillBoost: localize, explore, verify—no skill overfitting","LLM skills evolve without overfitting via SkillBoost"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The training split used both to select candidate edits and to enforce the regression bound is a sufficient stand-in for the distribution the agent will face after deployment, and the fixed ε=0.05 cap is appropriate for every task; if either fails, the near-zero test–train gap may not survive contact with new data.","fun_headline_variants_meta":{"raw":{"variants":["SkillBoost curbs LLM skill overfitting with constrained search","Constrained exploration stops LLM skills from overfitting","SkillBoost: localize, explore, verify—no skill overfitting","LLM skills evolve without overfitting via SkillBoost"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000239,"raw_usage":{"total_tokens":1339,"prompt_tokens":720,"completion_tokens":619,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":549}},"tokens_in":464,"tokens_out":619,"duration_ms":5100,"temperature":1.0,"reasoning_tokens":549,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T11:40:26.348928+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a skill version accepted by the gate (r>0, Regress<0.05 on the training split) and evaluate it on a new disjoint sample from the same task family, or a lightly perturbed version of the benchmark. If the test-minus-train gap turns strongly negative, or if a held-out set's solved-case regressions exceed its fixes, then the training split is not a sufficient proxy and the claimed overfitting control does not generalize. A cheaper check: rerun the loop with ε=0 and with ε=0.05; if performance and generalization are identical, the regression bound is not doing the work claimed.","supporting_citations":[],"review_version":1}