{"id":"7af0cb95-1ed5-4d65-b8c4-b799cd0c4c09","arxiv_id":"2608.22610","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Skill reliability in self-evolving agents is a joint property of the skill, its skill bank, and the deployment domain; selecting and masking skills based on their coalition effect improves performance.","lead":"This paper shows that skills an AI agent saves from past experience can look useful on average while actually hurting performance when used together, and that skills that work in one setting can backfire after transfer. It introduces two automatic fixes, one that checks skills in groups before saving them and one that masks harmful skills after moving to a new domain, improving results on four agent benchmarks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Coalition pollution as demonstrated may be measurement noise: gate margin +0.006 vs audit Δ −0.005 differ by less than the stated ±0.04 resolution, and Theorem 1 does not cover the admitted skill's own full-bank marginal.","rationale":"The reader's weakest assumption—that V_D is a faithful measure of downstream contribution—is the right locus, and my analysis sharpens it: the problem is not merely that the reversal CI includes zero; the coalition pollution example is internally inconsistent with the paper's own definition of the gate once V is faithful. Given G_e = V(B∪{s_new})−V(B), a noiseless gate would observe the candidate's positive full-bank marginal and would not accept a candidate whose audit Δ is negative. The observed mismatch can only arise from using different noisy estimators, and the reported difference is well within the noise floor. This means the paper's most concrete evidence for its central failure mode is underpowered, and the structural theorem does not repair the gap. The proposed enlarged paired-sample test would settle whether the phenomenon survives. Because the conceptual claim remains plausible and the theorem is correct as far as it goes, the verdict should stay CONDITIONAL rather than moving to ACCEPT or REJECT, so I recommend UNCHANGED relative to the reader's conditional verdict.","tokens_in":14288,"tokens_out":10952,"duration_ms":106024,"concrete_test":"Re-run the B3→B4 SMRA audit on a single enlarged paired sample, roughly 4,000 queries (extrapolating the stated 314-query ±0.04 resolution to ±0.01), using the identical query set and judge for both the gate reward r_out and the audit value V. Test whether G = V(B4)−V(B3) is positive while Δ_AP(B4) = V(B4)−V(B4\\{AP}) is negative on that same sample at p<0.01. If the two estimates converge to the same sign or to zero, the observed coalition pollution is measurement noise; if they diverge as reported with p<0.01, the phenomenon is a genuine structural gate failure. The same enlarged sample should also be used to compute the Shapley value of AP, since a positive full-bank marginal with negative Shapley would support the paper's structural claim but has not yet been measured.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing weakness is that the signature 'coalition pollution' example is not a consequence of the paper's structural theorem, and its magnitude is inside the stated measurement noise. If V is taken as the true value, the outcome gate's gain G_e = V(B∪{s_new}) − V(B) is definitionally the candidate's own full-bank marginal Δ_snew(B∪{s_new}). A noiseless outcome-only gate therefore cannot accept a skill whose post-admission full-bank marginal is negative. Section 3.2 nevertheless presents gate margin +0.006 and audit Δ_AP(B4) = −0.005 as the same event; the difference (0.011) is far below the paper's own single-skill audit resolution of ±0.04 and within the ±1 SE margins of Figure 3a. The apparent pollution is thus statistically indistinguishable from noise in r_out or V_D, not a demonstrated structural failure. Theorem 1 proves only that endpoints leave incumbents' signs unconstrained; it does not cover the candidate's own full-bank marginal, and it cannot make the Table 1 estimates consistent. The cross-domain reversal is similarly a point estimate (−0.035) whose 95% CI includes zero, as the paper concedes in §3.3. Both headline failure modes therefore currently rest on effects whose noise is comparable to their size.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper studies the reliability of skills accumulated in LLM-agent skill banks. It introduces a Skill Mechanistic Reliability Audit (SMRA) that measures each skill's marginal contribution by paired knockout evaluations and reports two failure modes: coalition pollution (a newly admitted skill contributes little while an incumbent carries the bank) and cross-domain utility reversal (a source-beneficial skill harms the target). It then proposes CASS, which augments the acceptance criterion with sampled Shapley marginals, and u-SMCO, which masks transferred skills using an unlabeled retrieval-quality signal. Experiments on LoCoMo, LongMemEval, HotpotQA, and ALFWorld report consistent improvements over skill-based baselines, and a theorem argues that outcome-only gates cannot identify negative contributors. The central conceptual claim is that skill reliability is a joint property of the skill, the bank, and the domain.","tokens_in":14601,"tokens_out":8165,"duration_ms":73517,"significance":"If the empirical failure modes were cleanly established, the paper would make an important contribution: it would shift skill evaluation from isolation-based scores to coalition- and domain-conditioned reliability, and it proposes two concrete, label-light interventions. The paired knockout audit design is a principled way to measure mechanistic contributions, and the paper is transparent about the uncertainty in its reversal example. CASS and u-SMCO are well-motivated and evaluated across four benchmarks, and the u-SMCO result that a label-free retrieval signal can anti-correlate with the training reward channel is suggestive. However, the headline audit findings currently rest on effects whose magnitudes are comparable to the stated measurement noise, so the empirical basis for the central claim needs strengthening.","major_comments":[{"comment":"The claimed 'coalition pollution' example is not statistically supported, and Theorem 1 does not establish the claimed blindness. Equations (1) and (2) together imply G_e = V(B_{e+1}) - V(B_e) = Δ_{s_new}(B_{e+1}), which the paper itself states when it says a noise-free gate reads exactly the candidate's own marginal. A noiseless outcome-only gate therefore cannot accept a skill whose full-bank marginal is negative. The reported gate margin +0.006 and the audit value Δ_AP(B_4) = -0.005 for the admitted skill differ by 0.011, below the stated single-skill audit resolution of ±0.04 and within the ±1 SE error bars of Figure 3(a). The example is thus indistinguishable from measurement noise in r_out or V_D. Theorem 1 proves only that a positive gain does not identify whether some bank member has negative contribution; it does not cover the candidate's own full-bank marginal, so it cannot make the Table 1 estimates consistent. To support the coalition-pollution failure mode, the authors should either demonstrate a case where the admitted skill's audit value is significantly negative while the exact-value gain is positive, or re-frame the failure as one of attribution to the wrong skill and support that re-framing with a suitable statistic.","section":"§3.2, Eqs. (1)–(2), Theorem 1"},{"comment":"The cross-domain utility reversal is only a directional effect. The target-domain point estimate for Skill_IN is Δ = -0.035 with a 95% confidence interval that includes zero, as the paper concedes. Given the stated audit resolution of ±0.04, this estimate cannot distinguish reversal from no effect. The u-SMCO results in Table 4 provide circumstantial support, but the direct audit evidence for the reversal failure mode should be strengthened, for example by increasing the number of paired queries or reporting a significant negative target-domain marginal for at least one skill.","section":"§3.3, Fig. 3(c)"},{"comment":"The main results omit the standard deviations promised in Section 5.1. For example, the F1 gain over MemSkill on LongMemEval is 0.01 percentage points (30.03 vs. 30.02), which is well within any plausible seed variance, yet no variance or significance test is reported. The claim of 'consistently improve' across benchmarks cannot be reliably assessed without per-seed numbers or confidence intervals for the headline comparisons.","section":"§5.2, Table 2"},{"comment":"There is no end-to-end ablation that isolates CASS and u-SMCO separately on the full set of benchmarks. Table 2 compares the combined system against baselines; Table 3 evaluates CASS with a K-controlled setup on one protocol, and Table 4 evaluates u-SMCO on six trained banks on HP-50. The relative contribution of each intervention to the Table 2 gains, and any interaction between them, is therefore not identified. An ablation matrix (MemSkill, MemSkill+u-SMCO, MemSkill+CASS, MemSkill+CASS+u-SMCO) on at least the main benchmarks would be needed to support the stated attributions.","section":"§5, Tables 3–4"}],"minor_comments":[{"comment":"The choices λ=0.2, the number of sampled coalitions N (apparently 8 from the cost statement in Section 5.5), the masking threshold τ, and the 20-query probe size used in Table 4 are introduced without justification or sensitivity analysis. Please add ablations for these choices or state explicitly how they were selected.","section":"§4.1, Eq. (6), Algorithm 1"},{"comment":"The Shapley value definition is standard, but the paper should state explicitly whether ϕ is ever computed exactly in the experiments or used only as motivation for CASS's coalition sampling; the current text leaves this ambiguous.","section":"§3.1, Eq. (3)"},{"comment":"CASS accepts 1 of 12 candidates, but no analysis is provided of whether any of the 11 rejected candidates would have been beneficial; the low acceptance rate may be over-conservatism. Reporting the audit marginals of the rejected candidates would strengthen the claim.","section":"§5.3, Fig. 5"},{"comment":"The stop threshold τ is never given a numerical value or a sensitivity analysis; without it the masking procedure is not fully specified.","section":"Algorithm 1, Table 4"},{"comment":"The statement that the audit toolkit, method code, and trained checkpoints 'will be released' appears without a release link or appendix; please provide an availability statement with a URL or state that code is included in the supplementary material.","section":"Abstract and §1"}],"recommendation":"major_revision"},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the paper's central reframing—that skill reliability is a joint property of the skill, the surrounding bank, and the deployment domain—is genuinely useful and not something I've seen in the cited prior work. Second, the two headline failure modes (coalition pollution and cross-domain utility reversal) are demonstrated with effect sizes that are within the paper's own stated measurement resolution, so the empirical proof is weaker than the abstract implies.\n\nWhat's new: SMRA as a skill-bank audit using paired knockouts, the two failure-mode concepts, and two cheap interventions (CASS with sampled Shapley marginals, u-SMCO with unlabeled retrieval quality). The interventions are simple and clearly described. The matched-protocol comparison against MemSkill is the right way to evaluate them, and the consistent gains across four domains are a real plus.\n\nWhere it's soft: The coalition pollution example in §3.2 hinges on a gate margin of +0.006 and an audited contribution of −0.005, a gap of 0.011, which is below the paper's own ±0.04 single-skill audit resolution. That specific case is statistically indistinguishable from noise. The cross-domain reversal for Skill_IN on HotpotQA has a 95% CI that includes zero; the paper calls it \"directional,\" which is honest but means the reversal is not established. Theorem 1 is true but weaker than the paper's framing: it shows endpoint ambiguity, not that the candidate's own marginal is unknowable. The Table 2 main results omit the standard deviations the text says are reported. There is also no full ablation separating CASS and u-SMCO on all benchmarks. None of these are fatal to the conceptual story, but they mean the evidence fits the theory rather than proves it.\n\nThe citation pattern looks fine; the related work is appropriate and the distinction from SkillLens is fair. No code or data yet, though the release announcement is good.\n\nBottom line: this deserves a serious referee. The ideas are worth developing and the interventions are cheap enough that others will try them. I'd want the authors to release the audit code and either tighten the audit resolution or weaken the causal claims before acceptance. But this is not a desk reject.","headline":"Genuinely useful reframing of skill reliability as coalition- and domain-dependent, but the headline failure modes rest on effect sizes within the paper's own stated noise floor.","tokens_in":15136,"tokens_out":2202,"duration_ms":19821},"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that a skill's reliability is determined jointly by the skill, its skill bank, and its deployment domain, and that outcome-only admission gates cannot see the two failure modes this creates.","keywords":["skill reliability","coalition pollution","cross-domain utility reversal","coalition-aware skill selection","unsupervised skill masking","Shapley value","self-evolving agents","skill banks"],"falsifier":"Run the SMRA knockout audit on HotpotQA with several thousand paired queries instead of 314; if the skill `insert` fails to show a significantly negative contribution (95% CI excluding zero), the paper's central example of cross-domain utility reversal loses its support.","tokens_in":14117,"feed_emoji":"🧩","tokens_out":15004,"duration_ms":115754,"temperature":0.7,"pith_summary":"Self-evolving agents accumulate reusable skills by accepting candidates whose addition raises aggregate task reward. This paper argues that such outcome-only gates are structurally blind to which skill actually caused the gain, and that a skill's reliability is not an intrinsic attribute: it shifts with the surrounding skill bank and with the deployment domain. Through paired knockout audits, the authors identify two recurring failures—coalition pollution, where an admitted skill contributes near zero while an incumbent carries the bank, and cross-domain utility reversal, where a source-beneficial skill harms after transfer. They then show that replacing the scalar gate with coalition-aware selection (CASS), and masking transferred skills with a label-free retrieval-quality signal (u-SMCO), improves task performance and cross-domain generalization on conversational and embodied benchmarks. If correct, the paper establishes skill coalitions, not individual skills, as the unit of reliable skill evolution.","feed_headline":"Skill reliability is a coalition property, not a skill attribute","feed_subtitle":"Aggregate reward gates admit inert or harmful skills; coalition-aware selection and label-free masking fix both.","key_machinery":"The central object is the coalition-conditioned reliability profile $R(s;B,D)$: a skill's contribution measured in a specific bank $B$ and domain $D$. The audit computes full-bank knockout marginals $\\Delta_s^D(B)=V_D(B)-V_D(B\\setminus\\{s\\})$ and, for composition dependence, the Shapley value $\\phi_s^D(B)$—the expected marginal contribution over all subsets of the other skills. Theorem 1 (outcome-only non-identifiability) carries the argument: a positive bank-level gain $V(\\tilde B_{e+1})-V(B_e)>0$ does not identify whether the new bank contains a negatively contributing skill. Two mechanisms implement the coalition view online. CASS samples $N$ coalitions, scores each by the normalized knockout margin $\\Delta_S(B)=[V(B)-V(B\\setminus S)]/|S|$, and ranks candidate banks by $\\sigma(B)=\\bar\\Delta_N(B)+\\lambda r_{\\rm out}(B)$. u-SMCO scores each skill by its retrieval-quality knockout $\\psi(s)=RQ(B)-RQ(B\\setminus\\{s\\})$ on unlabeled target queries and greedily masks the worst scorer. Together they turn the audit's counterfactual into a gate and a mask.","core_discovery":"The central claim, stated in Section 3.4, is that skill reliability is a joint property of the skill, the surrounding skill bank, and the deployment domain, and not an intrinsic attribute of the individual skill. The Skill Mechanistic Reliability Audit (SMRA) measures each skill's contribution by paired knockout: $\\Delta_s^D(B) = V_D(B) - V_D(B\\setminus\\{s\\})$, where $V_D(C)$ is the downstream value of a coalition $C$ on domain $D$. Auditing a baseline self-evolving agent's bank across compositions exposes coalition pollution—the admitted skill contributes $\\approx 0$ while an incumbent's marginal carries the bank—and auditing the same bank across source and target domains exposes utility reversal, e.g. `insert` is significantly positive on LoCoMo ($\\Delta=+0.084$) but its point estimate reverses on HotpotQA to $\\Delta=-0.035$ with a 95% confidence interval that still includes zero. The paper proves (Theorem 1) that a gate observing only the endpoints $V(B_e)$ and $V(\\tilde B_{e+1})$ cannot identify whether the enlarged bank contains a negatively contributing skill. CASS augments the acceptance criterion with Monte Carlo sampled Shapley marginals, and u-SMCO greedily masks the transferred skills whose removal improves retrieval quality on unlabeled target queries; both improve task performance and cross-domain generalization over the baseline.","pith_inferences":["If the reversal phenomenon replicates across other banks and target domains, then cross-domain transfer of skill banks should include an unsupervised masking step as a standard deployment phase, not an optional repair.","The non-identifiability theorem suggests that any skill-evolution method relying on scalar reward deltas—even with a less noisy reward estimator—cannot attribute gains to specific skills without changing what it measures; adding coalition probes is an architectural change, not a statistical refinement.","A natural stress test is to run CASS with the same number of coalition evaluations but an aggregate-reward-only ranking rule; if performance matches, the gain comes from extra compute, and if it drops, the coalition signal is the active ingredient.","Because the audit resolves a single skill only to about $\\pm0.04$ on 314 paired queries, practical deployments should spend additional paired queries on borderline skills whose measured marginal falls inside that band."],"forward_implications":["Skill-evolution gates that rely on aggregate reward alone should be replaced or augmented by coalition-conditioned signals; otherwise they will keep retaining non-contributing or harmful skills.","After a skill bank is transferred to a new domain, it should be re-audited under target-domain queries before deployment; source-domain usefulness is not evidence of target-domain usefulness.","Coalition-level reliability modeling reduces sensitivity to noisy outcome-reward fluctuations during reinforcement learning, so training decisions can trust the bank's composition rather than a single noisy scalar.","On the four tested benchmarks (LoCoMo, LongMemEval, HotpotQA, ALFWorld), both interventions improve task performance and cross-domain generalization over strong skill-based baselines.","The two failure modes are structural, not statistical: no amount of additional outcome sampling lets an endpoint-only gate separate a bank that improved from one that only appeared to."],"supporting_citations":[{"why":"It supplies the coalition marginal-contribution view that the audit uses to attribute bank value to individual skills.","marker":"Shapley et al. 1953"},{"why":"It brings Shapley values into model explanation and underlies the averaging over subsets in the reliability profile.","marker":"Lundberg and Lee 2017"},{"why":"It provides the set-function and game-theoretic basis for higher-order skill interactions and for sampled-coalition scoring in CASS.","marker":"Grabisch et al. 2016"},{"why":"It is the base self-evolving agent whose skill-bank evolution gate CASS modifies and against which the main comparisons run.","marker":"Zhang et al. 2026b"},{"why":"It documents the failure mode of entropy-minimization test-time adaptation, which motivates u-SMCO's decoupled retrieval-quality signal.","marker":"Iwasawa and Matsuo 2021"},{"why":"It establishes that source-domain competence is a poor predictor of target-domain behavior, underpinning cross-domain utility reversal.","marker":"Wang et al. 2020"},{"why":"It provides LoCoMo, the source domain where the audit finds an incumbent skill carrying the bank and where main conversational results are measured.","marker":"Maharana et al. 2024"},{"why":"It provides HotpotQA, the target domain where the same skill's contribution reverses to a negative point estimate.","marker":"Yang et al. 2018"},{"why":"It provides ALFWorld, the embodied benchmark used for the seen/unseen success-rate results.","marker":"Shridhar et al. 2020"},{"why":"It provides LongMemEval, one of the conversational benchmarks used to evaluate CASS and u-SMCO.","marker":"Wu et al. 2025"}],"fun_headline_variants":["Coalition-aware skill selection beats aggregate reward gates","Skill gates blind to negative coalition contributions","Skill reliability: a property of the bank, not just the skill","CASS and u-SMCO fix hidden failures in skill banks","Coalition pollution and utility reversal plague self-evolving agents"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The diagnosis rests on treating the measured change in task performance when a skill is removed as an accurate signal of that skill's contribution; on the key reversal example, the 95% confidence interval still includes zero, so the signal is about as large as the noise.","fun_headline_variants_meta":{"raw":{"variants":["Coalition-aware skill selection beats aggregate reward gates","Skill gates blind to negative coalition contributions","Skill reliability: a property of the bank, not just the skill","CASS and u-SMCO fix hidden failures in skill banks","Coalition pollution and utility reversal plague self-evolving agents"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001911,"raw_usage":{"total_tokens":7571,"prompt_tokens":1113,"completion_tokens":6458,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":729,"completion_tokens_details":{"reasoning_tokens":6378}},"tokens_in":729,"tokens_out":6458,"duration_ms":42275,"temperature":1.0,"reasoning_tokens":6378,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-27T18:22:58.684765+00:00","model_set":{"reader":"deepseek-v4-flash"},"pipeline_version":"pith-pipeline@v1.4.0-alltime-deepseek-medium","input_kind":null,"input_chars":null,"input_total_chars":null,"passes_missing":null,"published_standing":null,"published_confidence":null,"objection_audit":null,"run_record":null,"basis":{"kind":"not_recorded","source":null,"chars_given":null,"chars_total":null},"receipt":null,"falsifier":"Run the SMRA knockout audit on HotpotQA with several thousand paired queries instead of 314; if the skill `insert` fails to show a significantly negative contribution (95% CI excluding zero), the paper's central example of cross-domain utility reversal loses its support.","supporting_citations":[],"lane_accuracy":{"status":"not yet measured","lane":{"model":"deepseek-v4-flash","input_kind":null},"url":"https://pith.science/accuracy"},"review_version":1}