{"id":"9d1e9a50-2683-45a4-89a1-9cec0538c5fa","arxiv_id":"2607.28777","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A comparative loop that extracts evidence for and against observed behaviors from judge preferences over unlabeled executions can optimize agent skill documents without any ground-truth labels or rewards.","lead":"SSO improves a frozen LLM agent's reusable \"skill\" document using only unlabeled task examples, comparing executions with an LLM judge instead of ground-truth labels. It beats other label-free prompt optimizers on the tested benchmarks and sometimes matches a label-based optimizer, so it could cut the cost of adapting agents in data-scarce settings.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Unvalidated LLM judge drives both training and acceptance signals; without judge–ground-truth calibration, reported GT gains and self-judged open-ended win rates may reflect judge self-consistency rather than task quality.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the untested reliability of the LLM judge. I agree with that assessment. After reading the full text, I find no additional fatal flaw; the evidence aggregation, behavior extraction, and rendering are heuristics that could be noisy, but their effect is bounded by the validation gate. However, the validation gate itself is corrupted if the judge is biased, because it uses the same judge for acceptance. This makes the judge validity the single point of failure. The proposed test directly measures whether pairwise judge decisions align with ground truth on the actual benchmarks used for optimization. It is inexpensive and decisive. The reader's CONDITIONAL verdict is appropriate: SSO is promising but unproven until this calibration is established. Therefore no verdict change is needed.","tokens_in":13913,"tokens_out":5061,"duration_ms":55382,"concrete_test":"Rerun SSO on SpreadsheetBench and LiveMathematicianBench with GPT-5.4-mini as executor and GPT-5.5 as judge, following the paper's setup. At each step, record every pairwise decision y_i^k (training) and v_m (validation). For comparisons where the two executions differ in GT score (e.g., spreadsheet task success, math exact match), compute the agreement between the judge's preferred execution and the GT-preferred execution. Report accuracy and Cohen's kappa over all such pairs. If accuracy <60% or kappa <0.2, the judge is not a reliable proxy and the optimization signal is not sound; if accuracy >90% and kappa >0.6, the concern is resolved and the method's mechanism is validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that SSO optimizes reusable skills for a frozen agent rests entirely on the comparative signal of a single LLM judge (f_judge). This judge is used in three load-bearing places: (i) to generate training evidence via y_i^k in Eq. 5 and Algorithm 1 lines 9/19–27; (ii) to decide acceptance via v_m in Eq. 17 and lines 39–42; and (iii) as the evaluation metric for all open-ended results in Table 2 (the same GPT-5.5 family as the optimization judge). No calibration of the judge's pairwise preferences against ground truth is reported anywhere. If the judge is biased—e.g., toward longer or more fluent outputs, or toward its own stylistic preferences—SSO will encourage and validate behaviors that satisfy the judge, and the open-ended win rates become self-consistent artifacts rather than evidence of task quality. The closed-ended metrics are GT-based, so they are not self-referential, but the optimization signal is still judge-based; the observed GT gains would then be an incidental byproduct of whatever the judge happens to reward. Because the paper provides no measure of judge–GT agreement on the exact tasks, the most essential condition for the method's validity is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SSO is a framework for optimizing a reusable natural-language skill document for a frozen LLM agent using only unlabeled task instances. At each step it generates K complete skill probes from a subset of the current executions, runs them on the same training batch, and uses an LLM judge to compare each probe execution against the anchor (Eq. 5). A behavior extractor that does not see judge decisions identifies behavioral differences (Eq. 6), which are clustered (Eq. 8); evidence is aggregated per changed cluster with normalization (Eqs. 9–13), and the top-ranked behaviors are rendered into a new skill (Eq. 15). The candidate is accepted only if it wins a judge-based validation comparison (Eqs. 17–18). Experiments across three target models, six closed-ended benchmarks, and three MT-Bench-101 open-ended tasks report that SSO outperforms GT-free prompt optimizers SPO and PDO, and approaches or exceeds the GT-based SkillOpt on some model/benchmark combinations.","tokens_in":14294,"tokens_out":6879,"duration_ms":73575,"significance":"SSO introduces a novel optimization unit—behavior-level evidence aggregated across probes and tasks—rather than candidate-level selection, and addresses a practically important setting where GT feedback is unavailable. The design is principled: the behavior extractor is blinded to judge decisions, and evidence normalization controls for variable numbers of changed behaviors. The paper provides component ablations (Table 3), hyperparameter sensitivity sweeps (Figure 3), and an analysis distinguishing rendered candidates from the best probes (Figure 4), all of which strengthen the internal validity of the closed-ended results. However, the central validity question—whether the LLM judge's pairwise preferences are a reliable proxy for task quality—is not empirically addressed. This is especially problematic for the open-ended results (Table 2), where the evaluation judge is from the same family as the optimization judge. The paper is potentially significant, but the missing calibration and variance reporting are needed before the contribution is fully established.","major_comments":[{"comment":"The open-ended win rates are computed by an LLM judge (GPT-5.5) that is also the optimizer-side judge. The optimization signal, acceptance gate, and evaluation metric are therefore drawn from the same judge family, creating a self-consistency confound. No human evaluation or independent judge is reported. To support the claim that SSO is preferred on open-ended tasks, please report agreement with human judgments on the MT-Bench-101 tasks, or at least evaluate with a different-model judge and show that the two judges correlate.","section":"§Experimental Setup (Metrics), Table 2"},{"comment":"The method's core assumption is that pairwise LLM preferences are a reliable proxy for task quality. No calibration of the judge against ground truth is given anywhere in the paper. On the closed-ended benchmarks, where GT is available, compute judge–GT agreement (e.g., accuracy of judge choices versus correct answers, or rank correlation with task scores) on a held-out set. On open-ended tasks, report human agreement. Without this calibration, the closed-ended gains could be an incidental effect of judge bias that happens to align with GT on these tasks, and the open-ended results are uninterpretable as evidence of task quality.","section":"§Methodology, Eqs. (5), (17), Algorithm 1 lines 9, 39–42"},{"comment":"All reported numbers are point estimates with no error bars or repeated seeds. Given the stochastic components (batch sampling, probe generation, clustering, rendering), single-run results cannot establish that SSO reliably outperforms SPO/PDO, particularly for small differences such as 78.8 vs 76.3 on SearchQA with GPT-5.4-mini or 89.6 vs 88.0 on DocVQA with Qwen3.5-4B. Please report means and standard deviations across at least three seeds, or include significance tests.","section":"Table 1, Experimental Setup"},{"comment":"The adaptation of SPO and PDO to optimize complete skill documents is not described; the paper states only \"We adapt both prompt optimizers to optimize complete skills.\" Because these baselines are the primary GT-free competitors and their numbers are claimed to be reproduced, the fairness of the comparison is not verifiable. Provide the exact adaptation details (prompt templates, number of candidates, validation procedure, and compute budget) or release the code.","section":"§Experimental Setup (Baselines)"}],"minor_comments":[{"comment":"The paper refers repeatedly to the supplementary material for schedules, prompts, and additional results, but the arXiv version does not include the supplementary. Ensure the supplementary file is attached in the submission.","section":"General"},{"comment":"The note says SPO and PDO are reproduced, while other baselines are reported from (Yang et al. 2026a). This cross-paper comparison is acceptable for context but should be stated earlier in the text so readers do not infer all baselines were run under identical harnesses.","section":"Table 1 footnote"},{"comment":"The win-rate table reports no sample size or confidence interval. Please state the number of comparisons per cell (e.g., number of tasks × number of repeats) so the reader can assess the precision of the 50–85% win rates.","section":"Table 2"},{"comment":"The text claims the default settings achieve the best performance in all sweeps, but the figure is small and not all points are annotated with values. Add numeric labels or a table of the swept values and corresponding scores.","section":"Figure 3"},{"comment":"Minor typographical issue: \"Skil-lOpt\" appears split across lines in the Introduction; it should be \"SkillOpt\" throughout.","section":"Introduction"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a CL venue and addresses a real gap in the literature. The central idea is novel and the closed-ended results are encouraging, but the missing judge–ground-truth calibration is a core validity concern, and the open-ended evaluation is self-referential as currently designed. The absence of variance estimates also makes the comparative claims hard to evaluate. I believe these issues are addressable in major revision; I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nPunchline: SSO is worth your attention. The step from candidate-level preference selection to behavior-level evidence aggregation over complete skill probes is real—I don't see it in SPO, PDO, UPA, or SkillOpt. And the closed-ended results are surprisingly strong: on GPT-5.5, SSO averages 79.2 vs 69.4/68.7 for SPO/PDO, and on GPT-5.4-mini it edges past GT-based SkillOpt (65.5 vs 64.3). If those numbers hold up, this is a practical win for anyone adapting frozen agents without labels.\n\nWhat the paper does well: the method is clearly specified. The behavior extractor is deliberately blind to the judge's decisions, the normalization in Eq. 11 gives every decisive comparison equal influence, and the validation gate (Eq. 18) rejects regressions. The ablations are sensible—dropping clustering, normalization, the validation gate, or switching to direct probe selection all hurt. The hyperparameter sweeps look clean. The writing is unusually concrete about what the judge does and does not see.\n\nNow the soft spots, in proportion. The biggest one is the LLM judge. It is used in three places: training evidence (Eq. 5), the acceptance gate (Eq. 17), and the open-ended evaluation in Table 2. The paper reports no calibration of the judge against ground truth on these tasks. For the closed-ended benchmarks, the final metrics are hard accuracy, so the judge's bias cannot inflate those numbers directly—but it can distort the optimization trajectory, and the fact that hard accuracy improves is indirect evidence that the judge aligns with GT on those tasks. That's decent but not ideal. The open-ended results are weaker: the evaluation judge is the same GPT-5.5 family as the optimization judge, so the win rates in Table 2 could partly reflect judge self-consistency rather than task quality. Some human checks or a different judge family would have helped a lot.\n\nOther issues: no error bars or repeated seeds anywhere, so we can't tell if the large average gains are stable; the SPO/PDO adaptations are described but the prompts and datasets aren't included; SkillOpt baseline numbers are imported from the prior paper rather than re-run. Code, data, and prompts are missing. These are all fixable, but they are real limitations for a paper whose value is empirical.\n\nBottom line: the method is new, the closed-ended evidence is substantial, and the paper deserves a serious referee. I'd want the authors to add judge calibration and repeated seeds before accepting the open-ended claims. I'd bring it to reading group.","headline":"SSO is a genuinely new behavior-level GT-free skill optimizer with strong closed-ended results; the open-ended win rates and missing judge calibration are the main questions.","tokens_in":14770,"tokens_out":4699,"would_cite":true,"duration_ms":44089,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A reusable agent skill can be optimized from unlabeled task instances alone, using an LLM judge's pairwise comparisons of executions, and on closed-ended benchmarks this approaches or exceeds ground-truth-based skill optimization.","keywords":["self-supervised skill optimization","ground-truth-free optimization","reusable agent skills","skill documents","pairwise LLM judge","behavior-level evidence aggregation","prompt optimization","frozen LLM agents"],"falsifier":"Hold out ground-truth labels from the optimizer but keep them available to the experimenter, run SSO on a closed-ended benchmark, and measure the agreement between the judge's pairwise decisions and the true outcomes; if agreement is near chance yet the same judge reports large pairwise win-rate gains for the accepted skill, the reported improvement is likely judge self-consistency rather than genuine task quality. A direct variant would flip the judge's preferences and check whether accepted updates still appear to improve.","tokens_in":13842,"feed_emoji":"⚙️","tokens_out":8879,"duration_ms":78378,"temperature":0.7,"pith_summary":"The paper's method, Self-Supervised Skill Optimization (SSO), asks whether a reusable skill document for a frozen LLM agent can be improved using only unlabeled task instances, with no ground-truth labels, scores, rewards, or task-specific evaluators. The paper argues yes, provided optimization is driven by pairwise judge comparisons of executions: complete skill probes run alongside the current skill, an LLM judge says which execution is better, and a behavior extractor blind to the judge identifies what changed. Equivalent behaviors are clustered across tasks, evidence for and against each behavior is aggregated and normalized, and a new skill is rendered from the top-ranked behaviors, accepted only if it wins a validation comparison. Across six closed-ended benchmarks and three open-ended dialogue tasks, SSO beats existing ground-truth-free prompt optimizers and approaches, sometimes exceeding, the strongest ground-truth-based skill optimizer. If correct, this makes skill adaptation practical wherever unlabeled instances and a comparative judge exist but labeled feedback does not.","feed_headline":"No labels needed: pairwise judge comparisons sharpen agent skills","feed_subtitle":"Evidence-ranked behaviors from unlabeled executions rival score-based skill tuning on closed-ended benchmarks.","key_machinery":"The central unit is the behavior record: a natural-language description of an observable behavioral difference paired with an activation vector marking that behavior as present, absent, or unclear in each execution. The argument is carried by the evidence-aggregation formula e = y * d / |A|, where y is the judge's preference, d is +1 or -1 depending on whether the behavior appears or disappears in the preferred execution, and |A| is the number of changed behavior clusters in that comparison. This splits one normalized unit of evidence per decisive comparison, preventing any one execution from dominating because it differs in many behaviors; clustering lets the same behavior accumulate eviden","core_discovery":"The paper's central claim is that a single natural-language skill document can be optimized for a frozen agent using only unlabeled task instances and an LLM judge's pairwise preferences. SSO runs the current skill and several complete skill probes on the same tasks, gets an anchored win/loss/tie decision for each probe-versus-current comparison, then uses a behavior extractor that never sees the judge's decisions to record observable behavioral differences and their activation states. Equivalent behavior descriptions are clustered, and each decisive comparison contributes one normalized unit of evidence split among the behaviors that changed, with sign set by whether the preferred execution","pith_inferences":["Because the paper's open-ended evaluation also relies on an LLM judge of the same general kind as the optimization judge, some gains may reflect the agent conforming to the judge's stylistic preferences rather than improving on an external notion of quality; a small human or ground-truth calibration set would settle how much is externally valid.","The behavior-extraction, clustering, and normalized-evidence steps are not specific to a single skill document; they could plausibly update a whole skill library or combine evidence across heterogeneous execution formats such as answers, trajectories, artifacts, and terminal states.","A biased but self-consistent judge could encode its bias into the validation gate; a testable variant would gate updates with a different judge or occasional low-cost human checks.","SSO could be hybridized with sparse supervision: a few labeled examples used only to calibrate the judge might make the loop robust without requiring dense ground truth, though the paper does not explore this."],"forward_implications":["Reusable skills can be improved in settings where ground-truth labels, task scores, rewards, or task-specific evaluators are unavailable, as long as unlabeled task instances and a comparative judge exist.","Behavior-level evidence aggregation beats candidate-level selection: the rendered skill outperforms the best individual probe, so optimization should not be reduced to choosing among generated candidates.","A judge-based validation gate prevents regressions, allowing iterative refinement to continue across optimization steps without external supervision.","The approach transfers across several frozen target models and both closed-ended and open-ended multi-turn dialogue tasks, so it is not tied to a single model or answer format.","On closed-ended benchmarks, ground-truth-free skill optimization can approach and in one tested configuration exceed the best ground-truth-based skill optimizer, narrowing the practical gap between supervised and unsupervised skill tuning."],"fun_headline_variants":["No labels needed: pairwise judgements refine skills","Self-supervised skill tuning rivals score-based methods","Unlabeled tasks alone polish frozen-agent skills","LLM judge preferences optimize skills without ground truth","Skill optimization from unlabeled instances via comparisons"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The entire optimization signal rests on the untested premise that an LLM judge's pairwise preferences over unlabeled executions are a reliable proxy for task quality, and the paper does not report calibration of the judge's agreement with ground truth.","fun_headline_variants_meta":{"raw":{"variants":["No labels needed: pairwise judgements refine skills","Self-supervised skill tuning rivals score-based methods","Unlabeled tasks alone polish frozen-agent skills","LLM judge preferences optimize skills without ground truth","Skill optimization from unlabeled instances via comparisons"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000171,"raw_usage":{"total_tokens":1095,"prompt_tokens":719,"completion_tokens":376,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":308}},"tokens_in":463,"tokens_out":376,"duration_ms":4704,"temperature":1.0,"reasoning_tokens":308,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-03T00:23:16.894960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out ground-truth labels from the optimizer but keep them available to the experimenter, run SSO on a closed-ended benchmark, and measure the agreement between the judge's pairwise decisions and the true outcomes; if agreement is near chance yet the same judge reports large pairwise win-rate gains for the accepted skill, the reported improvement is likely judge self-consistency rather than genuine task quality. A direct variant would flip the judge's preferences and check whether accepted updates still appear to improve.","supporting_citations":[],"review_version":1}