{"id":"8c390a8b-0d6e-4d12-8645-be72928a4344","arxiv_id":"2607.27360","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A small RL-trained Mentor can discover a frozen LLM executor's blind spots and curate skills that improve AppWorld/BFCLv3 performance by 44.2% on average.","lead":"SkillMentor trains an RL-powered Mentor policy to generate diagnostic tasks, find a frozen agent's failure modes, and write corrective skills, improving that agent's benchmark scores by 44.2% on average. The paper argues that blind-spot diagnosis is a learnable capability separate from execution, enabling self-evolution without executor weight updates or human labels.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 44.2% gain is attributed to learned diagnosis, but the only training signal (Eq. 3–4) is an uncalibrated LLM-judge score gap; unless judge scores align with objective task success, the improvement could be reward hacking rather than diagnosis.","rationale":"The reader's verdict is CONDITIONAL, with the weakest assumption being the validity of the diagnostic gap as an oracle-quality measure. My stress-test identifies the same load-bearing point and sharpens it: the LLM judge in Eq. 3 is not merely a component of the pipeline but the sole reward source, and the curation reward in Eq. 4 reuses the same judge. Therefore, if the judge is biased or miscalibrated, the entire learning signal can be optimized without improving objective task success, and the abstract's causal claim that all gains come from learned diagnosis is unsupported. I additionally note that the formal definition of a blind spot requires consistent underperformance, but the method curates a skill after a single high-gap task, so even a perfect judge would not guarantee that the discovered failures are recurrent. This reinforces the need for the proposed calibration/recurrence check. The final benchmark numbers are real and the method is plausible, so the concern does not warrant rejection; it warrants an explicit condition: the authors should release code and either calibrate the judge against programmatic success or show that the same results hold with a programmatic reward. Thus the reader's CONDITIONAL verdict remains appropriate, and no change to the verdict is needed.","tokens_in":22394,"tokens_out":7768,"duration_ms":91903,"concrete_test":"Sample 200 diagnostic tasks from a trained SkillMentor checkpoint, retaining the stored reference and executor trajectories. Re-score both trajectories for each task using the benchmark's own deterministic success evaluator (AppWorld's success conditions, BFCLv3's function-call correctness), yielding a ground-truth gap δ_GT. Compute the rank correlation (e.g., Kendall's τ) between judge-based δ_i and δ_GT. If τ < 0.5, the reward is misaligned; then retrain SkillMentor using the programmatic evaluator as the scoring function for the same 200 tasks and compare final benchmark accuracy. If accuracy stays within ~2 points, judge bias is not load-bearing; if it drops materially, the reported 44.2% gain is at least partly an artifact of the LLM judge rather than learned diagnosis.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that SkillMentor learns blind-spot diagnosis and that all improvement originates from that capability—rests on the diagnostic gap δ_i = max(0, s^M_i − s^E_i) defined in Eq. 3. This gap is computed from scores produced by a single LLM judge (DeepSeek-V4-Flash) applied to trajectories of a strong reference model and the frozen executor. The same judge supplies the executor-grounded utility Δ in the curation reward (Eq. 4) and the Δ > 0.5 admission gate. No calibration of this judge against the benchmarks' own programmatic success signals is reported, and no check is made that the self-generated tasks are solvable, representative, or that the failures they expose are recurrent rather than one-off. Because the executor is frozen and no human labels are used, the judge is the sole source of learning signal. If the judge is biased toward reference-model style or is simply noisy, the RL policy can improve judge scores without improving real capability; the final benchmark numbers are objective and impressive, but they do not by themselves establish the mechanism—the same outcome could arise from a Mentor that learns to game its own reward. The paper also defines a blind spot as a task on which the executor 'consistently underperforms' (§3.1), yet the training objective never verifies recurrence: a single high-gap task can trigger skill curation. This gap between the conceptual claim and the operationalized reward makes the attribution to learned diagnosis insecure without external validation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SkillMentor proposes a learned diagnosis policy (Mentor) that, with a frozen executor and no human labels, generates diagnostic tasks, measures a diagnostic gap between a strong reference model and the executor using an LLM judge, and curates corrective skills into an external repository. The policy is trained with GRPO on a discovery reward (gap size) and a curation reward (executor score improvement). On AppWorld and BFCLv3, across three executors, SkillMentor achieves on average +44.2% relative improvement over a no-skill baseline and outperforms prompt-based mentors. The paper argues that diagnosis is a learnable capability and that all improvement stems from it, since the executor is frozen and no supervision is used.","tokens_in":22734,"tokens_out":6548,"duration_ms":66976,"significance":"If the central claim is sustained, the work establishes a new axis for agent self-evolution: learning to discover and repair blind spots, with human-interpretable skills that transfer across models. The empirical package is strong: main tables, ablations, an 8x8 transfer matrix, sensitivity sweeps, and five rescue case studies. The 44.2% headline is arithmetically consistent with Table 1. However, the mechanism attribution depends on the validity of the LLM-judge-based diagnostic gap, which is not calibrated against objective success signals. The paper would be significantly stronger with such a calibration and with a recurrence check for blind spots; as written, the empirical results are credible but the central interpretational claim is not yet fully established.","major_comments":[{"comment":"The diagnostic gap δ_i = max(0, s_i^M − s_i^E) is computed from LLM-judge scores, and the same judge supplies s_c in the curation reward (Eq. 4) and the Δ > 0.5 gate. No calibration of the judge against the benchmarks' programmatic success signals is reported. If the judge systematically prefers the reference model's trajectory style or is merely noisy, the RL policy could improve judge scores without improving real capability. The final benchmark numbers are objective and impressive, but they do not by themselves establish the mechanism. Please report a calibration study on a sample: compare judge scores with AppWorld's programmatic pass/fail and BFCLv3's correctness metrics, with agreement statistics; also report judge accuracy for the executor's own trajectories. Without this, the abstract's claim that 'all improvements originate from the learned diagnostic capability' is not justifie","section":"§3.3, Eq. (3)"},{"comment":"The blind-spot definition requires a 'task on which the Executor consistently underperforms' (recurrent), but the operationalization selects the single task q_max with the largest gap δ_max at a given step; no recurrence check is performed. A single high-gap task can trigger skill curation and reward, so the policy may be rewarded for one-off anomalies rather than recurrent failure modes. This inconsistency between the conceptual and operational definitions is load-bearing for the claim that the system learns to identify blind spots. Add a recurrence requirement (e.g., the same failure pattern must appear in multiple independently sampled tasks, or the curated skill must rescue a held-out task of the same type) or revise the definition to match the implementation.","section":"§3.1 vs §3.3–3.4"},{"comment":"The text states that the strong reference model M generates and scores its own trajectory (τ_i^M, s_i^M), while experiments identify the judge as DeepSeek-V4-Flash. Clarify whether M and the judge are the same model or different. If the judge is the same as the reference, self-preference bias is a concrete risk; if they are different, specify M and its relationship to the judge. Also, the paper does not verify that the generated diagnostic tasks are solvable by a correct procedure, which is necessary for the gap to represent a genuine capability gap rather than an artifact of task ill-posedness. A small human audit (e.g., 50 generated tasks, rating solvability and whether the executor's failure is a real blind spot) would substantially strengthen the diagnosis claims.","section":"§3.3 / Appendix C"}],"minor_comments":[{"comment":"There are numerous typographical/formatting artifacts, e.g., 'Acc Step' column with merged decimals like '.010 21.0.009' and the reproduced prompt figures (Figures 6–10) are too small to read. Please proofread the tables and provide high-resolution text versions of all prompts.","section":"Table 1 and Appendix C figures"},{"comment":"The sentence 'while BFCLv3 exhibits comparable increase' is confusing: Table 1 shows no step metric for BFCLv3. This clause appears to be a leftover or typo and should be clarified or removed.","section":"§4.1"},{"comment":"The x-axis labels in the AppWorld plot appear to jump from 80 to 100, skipping 90 (and the 'step' ticks after 80 are irregular). Please verify the tick labels and align them with the actual evaluation steps.","section":"Figure 4"},{"comment":"The phrase 'average of 44.2%' should specify that it is the average relative improvement across the six executor-benchmark combinations in Table 1 (three executors × two benchmarks), not a single benchmark average. This will avoid reader confusion.","section":"Abstract"}],"recommendation":"major_revision","confidential_remarks":"The paper's framing overclaims the isolation of diagnosis; the main missing piece is judge calibration. If the authors add a calibration study and a recurrence check, the paper could become acceptable. The empirical work appears solid, well-resourced, and the transfer matrix is a valuable contribution. No concerns about novelty; the related work is appropriately cited."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis one is worth a look, but read the fine print. The novel part is real: instead of assuming failures are given, they train a separate Mentor policy to generate diagnostic tasks and curate corrective skills, with the executor frozen and no human labels. That's a clean way to isolate diagnosis as a capability, and the empirical package is much better than the arXiv average. The 44.2% average improvement is internally consistent with Table 1, the 8x8 transfer matrix in Table 3 shows skills generalize across executors, and the appendix rescue cases show concrete failures fixed by concrete procedural skills. The ablations also make a decent case that joint discovery-curation training matters.\n\nThe soft spot is the training signal. The entire learning signal is the diagnostic gap delta = max(0, s^M - s^E), where both scores come from a single LLM judge (DeepSeek-V4-Flash). The curation gate and reward use the same judge. The paper never calibrates that judge against AppWorld's or BFCLv3's own programmatic success signals. So the method could in principle be learning to please the judge rather than to diagnose real failures. The final benchmark numbers are objective and good, which suggests the judge is approximately aligned, but the isolation claim—\"all improvements originate from learned diagnosis\"—is not fully pinned down. The stress-test puts this as a load-bearing flaw; I'd call it a moderate gap rather than a fatal one, because the final metrics and the qualitative skill examples give indirect evidence that the learned skills are real.\n\nTwo smaller issues. The paper defines a blind spot as a recurrent failure but never verifies recurrence; a single high-gap task can trigger curation. That's a mismatch between the conceptual definition and the operationalization. And no code or artifacts are released, so independent verification is hard right now. The threshold schedule and curation gate were selected on the same benchmarks used for the headline number; the sensitivity analysis helps, but some selection is likely.\n\nBottom line: this deserves a serious referee. The core idea is new, the experiments are extensive, and the weaknesses are addressable in revision—calibrate the judge, enforce recurrence, release code. I'd want to see those before fully believing the \"learned diagnosis\" story, but the paper is well above the bar for peer review.","headline":"Solid empirical paper with a genuinely new framing, but the headline claim about 'learned diagnosis' is only as strong as the uncalibrated judge that supplies the training signal.","tokens_in":23265,"tokens_out":4226,"would_cite":true,"duration_ms":43064,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SkillMentor claims that blind-spot diagnosis — discovering what an executor agent does not know — is a learnable capability, and that a Mentor trained only on its own diagnostic tasks improves frozen executors by 44.2% on average without hu","keywords":["blind-spot diagnosis","self-evolving agents","LLM agents","reinforcement learning","skill curation","frozen executor","zero human supervision","transfer learning"],"falsifier":"On a subset of tasks with programmatically checkable outcomes (e.g., unit-testable function calls), replace the LLM judge's scores with binary pass/fail from ground truth and retrain. If the performance gain largely evaporates, the learned signal was judge bias, not genuine blind-spot discovery. Alternatively, sample a batch of Mentor-generated diagnostic tasks and check by hand whether the reference model's trajectory is actually correct; if a substantial fraction are unsolvable, the diagnostic gap is an artifact.","tokens_in":22246,"feed_emoji":"🎯","tokens_out":5892,"duration_ms":53629,"temperature":0.7,"pith_summary":"The paper tries to establish that an AI agent can learn to diagnose another agent's blind spots, separate from learning to act. The authors freeze the executor and withhold all human labels, so the only possible source of improvement is the learned diagnostic capability. They train a Mentor policy with reinforcement learning to generate diagnostic tasks, measure the gap between a strong reference model and the frozen executor, and turn the biggest gap into a corrective skill. Across two benchmarks and three executors, this improves average accuracy by 44.2% over the no-skill baseline, and the learned skills transfer to unseen executors. If the claim holds, self-evolution no longer requires updating weights or human-curated data.","feed_headline":"Trained diagnosis boosts frozen AI agents 44% on average","feed_subtitle":"A small RL-trained Mentor finds and fixes a larger model's blind spots with no weight updates or human labels.","key_machinery":"The load-bearing object is a blind spot, defined as a pair (T, c) where T is a task on which the executor consistently underperforms and c is a corrective skill. The learning signal is the diagnostic gap δ_i = max(0, s_i^M − s_i^E), the difference in scores between a strong reference model and the frozen executor on a Mentor-generated diagnostic task. This gap serves as the discovery reward and selects which failure to repair; the curation reward is the executor-grounded improvement Δ = s_c − s_E_max after a candidate skill is added. A skill repository stores the repairs, a decaying threshold tracks the shifting capability boundary, and joint optimization of discovery and curation via RL cre","core_discovery":"The central claim is that blind-spot diagnosis is itself a learnable capability. A Mentor policy, trained via reinforcement learning on the diagnostic gap δ = max(0, s_reference − s_executor) and on executor-grounded skill utility, can discover recurrent failure modes of a frozen executor, curate them into reusable procedural skills, and thereby improve the executor's performance by an average of 44.2% relative to no skills. The paper also reports that a small trained Mentor outperforms a much larger prompted model, that weaker executors benefit more, and that skills transfer across models without retraining, especially from weaker to stronger executors.","pith_inferences":["Because most of the gain comes from 'mechanical' fixes (wrong parameter names, missing auth tokens, ghost APIs), the same discovery-and-curate loop could plausibly transfer to other API-driven tasks such as code repair or browser automation, where failures are similarly procedural.","One testable extension: replace the LLM judge with a programmatic verifier on a subset of tasks with known ground truth. If the 44.2% gain mostly disappears, the learned diagnosis was chasing judge bias rather than genuine blind spots — a direct check on the paper's core assumption.","The asymmetry that weaker-to-stronger transfer works better than stronger-to-weaker suggests the Mentor learns more from visible failure patterns than from near-perfect demonstrations; a concrete prediction is that deliberately degrading the training executor (e.g., adding more noise) would yield even broader repositories.","Since the Mentor is trained only against a reference model of the same domain, the approach should be sensitive to reference-model quality; a cheap resilience test is to retrain with reference models of different sizes and measure how the diagnostic gap and final accuracy co-move."],"forward_implications":["A frozen executor can be improved purely through an external skill repository curated by a learned diagnosis policy, with no weight updates and no human-labeled data.","Learned diagnostic skills transfer across models: a repository trained on a weaker executor often helps a stronger one, implying the skills encode portable procedural knowledge rather than model-specific quirks.","Blind spots are non-stationary: successful repair shifts the failure distribution, which is why the curation threshold must decay as training progresses.","A small RL-trained Mentor can outperform a much larger prompted model, suggesting that diagnostic skill is learned competence rather than a by-product of scale.","The resulting skill repository is interpretable: skills are stored as human-readable Markdown with trigger conditions and step-by-step rules, making the executor's deficiencies inspectable."],"fun_headline_variants":["LLMs learn to spot their blind spots, boosting performance 44%","Frozen AI agents improve 44% via learnable blind-spot diagnosis","RL-trained mentor finds blind spots, giving frozen agents 44% boost","Diagnosis is a skill: RL mentor lifts frozen LLMs by 44%"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire learning signal rests on the diagnostic gap δ_i = max(0, s_reference − s_executor), which assumes the strong reference model and the LLM judge provide an unbiased, oracle-quality score of task success; if the judge systematically favors the reference model, or the generated diagnostic tasks are not actually solvable, 'diagnosis' degrades into stylistic imitation.","fun_headline_variants_meta":{"raw":{"variants":["LLMs learn to spot their blind spots, boosting performance 44%","Frozen AI agents improve 44% via learnable blind-spot diagnosis","RL-trained mentor finds blind spots, giving frozen agents 44% boost","Diagnosis is a skill: RL mentor lifts frozen LLMs by 44%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000547,"raw_usage":{"total_tokens":2427,"prompt_tokens":695,"completion_tokens":1732,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":439,"completion_tokens_details":{"reasoning_tokens":1659}},"tokens_in":439,"tokens_out":1732,"duration_ms":12034,"temperature":1.0,"reasoning_tokens":1659,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T08:48:02.761426+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a subset of tasks with programmatically checkable outcomes (e.g., unit-testable function calls), replace the LLM judge's scores with binary pass/fail from ground truth and retrain. If the performance gain largely evaporates, the learned signal was judge bias, not genuine blind-spot discovery. Alternatively, sample a batch of Mentor-generated diagnostic tasks and check by hand whether the reference model's trajectory is actually correct; if a substantial fraction are unsolvable, the diagnostic gap is an artifact.","supporting_citations":[],"review_version":1}