{"id":"490e411d-a78f-49da-abe4-c2d61b3a6512","arxiv_id":"2608.04482","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Cluster-contrastive Capability Pages that add positive triggers, negative boundaries, and discriminative bodies to skill documents improve first-stage Recall@10 by 2.94 points on average and end-to-end task success by 3.62 points.","lead":"An offline method rewrites skill descriptions into Capability Pages, each containing a positive trigger, a negative boundary, and a discriminative body. It improves recall and end-to-end skill routing in LLM agents across five retrievers and four executors, without changing the online models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"T− boundaries are never checked against execution; the +3.62 end-to-end gain may not generalize if the LLM-written negative boundary over-excludes solvable queries.","rationale":"The reader's weakest assumption identifies exactly the same load-bearing concern: the compiler's T− is generated from documents alone, with no verification against actual execution outcomes. This is central because the entire method's novelty is the cluster-contrastive negative boundary, and the paper's formal definition of capability (Eq. 1) is execution-based. The manual QC pass in Section 4.3 checks only whether the text is clear, not whether it correctly locates the executable-region boundary. Thus the empirical end-to-end gain, while positive on average, does not establish that the mechanism works as claimed. The text-length confound in the router ablation is also a legitimate concern, but it is secondary to the boundary-validity issue. My recommendation matches the reader's conditional acceptance: the paper is promising and the experiments are well structured, but it needs an execution-based validation of T− (and ideally an artifact release) before the central claim can be taken as robust.","tokens_in":17096,"tokens_out":5362,"duration_ms":46302,"concrete_test":"Sample 200 skills from SRA-Bench. For each skill, use its compiled T− statements to generate (or select from the benchmark) 5 queries that the T− says are not executable by the focal skill. Execute the focal skill on those queries with the same executors and evaluators used in Table 5. Measure the false-exclusion rate: the fraction of queries the focal skill actually solves. If this rate is non-trivial (e.g., >5%), T− boundaries are inaccurate and the router is being taught to reject valid queries; the +3.62 gain may not generalize. As a control, also run the router with a length-matched neutral text instead of T−; if the gain persists, the effect is not due to the boundary semantics.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that adding T− improves end-to-end routing by 3.62 points rests on an unverified premise: that the T− strings generated by DeepSeek-V4-Pro in Section 4.3 accurately describe the boundary between a skill's executable region R(si) and its neighbors' regions, as defined in Eq. (6). The compiler reads only skill documents; it has no access to execution outcomes. The paper's only safeguard is a manual QC pass, which judges clarity rather than correctness against execution. If a T− entry over-excludes — naming a query form that the focal skill can actually solve — the router will reject the correct skill and end-to-end success will drop. The benchmark average of +3.62 shows that such errors do not dominate on SRA-Bench, but it does not validate the boundary. On a new library or query distribution, the same compiler may produce boundaries that are wrong more often, and the measured gain would not transfer. Because T− is the key novel component, the paper should provide execution-based evidence that generated T− statements match executable regions, rather than relying on subjective QC.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses skill retrieval for LLM agents, arguing that retrieval errors arise not only from the scorer but from the text being scored. It formalizes a skill's capability as its executable region R(s_i), the set of queries the skill can solve, and treats the skill document as a lossy observation of that region. The authors propose Capability Pages: offline-compiled, cluster-contrastive representations containing a positive trigger T+, a negative boundary T−, and a discriminative body B, appended to the original document. At inference time the retrieval index uses T+, B, and the original document, while the router receives T− as an explicit exclusion condition. On SRA-Bench (26,262 skills, 5,400 questions) the index view improves Recall@10 for all five tested retrievers, with a mean gain of 2.94 points; exposing T− on routing cards improves end-to-end task success by 3.62 points on average across four executors and six datasets. A transfer experiment on Chinese SSL-SkillDiscovery reaches 73.07% MRR@50 with a fixed encoder. The method requires no modification to online models, only offline rewriting of skill documents.","tokens_in":17464,"tokens_out":5008,"duration_ms":45395,"significance":"If the empirical claims hold, the work is practically significant: it shows that offline document rewriting can improve both first-stage retrieval and downstream routing without retraining or changing online models. The paper's strengths are its careful experimental design: leave-one-out field contributions, an index-with-T− ablation, a doc2query-style baseline, a top-1 preservation analysis, fixed-encoder transfer evaluation, and repeated-run uncertainty estimates with confidence intervals. The authors also honestly report cases where the representation does not help on every metric, such as the small nDCG@10 and MRR@10 decreases for Qwen3-Embedding-0.6B. The cross-executor consistency of the end-to-end gain is reassuring because the compiler and one executor are both DeepSeek-V4-Pro, yet the gains also appear for Qwen3.6 and Gemma executors. The main open risk is that the novel component, T−, is generated without execution-based validation of the boundary it describes, and the router ablation does not control for the added text length.","major_comments":[{"comment":"The central end-to-end claim rests on an unverified premise: that the T− strings written by DeepSeek-V4-Pro correctly describe the boundary between the focal skill's executable region and its neighbors' regions. Eq. (6) defines T− as (∪_{j∈N_i} R(s_j)) \\ R(s_i), but the compiler never observes execution outcomes and the manual quality-control pass judges clarity and distinctness, not whether the focal skill would actually fail on the queries described by T−. If a T− statement over-excludes, the router will reject the correct skill and task success will drop; the measured +3.62 average shows such errors do not dominate on SRA-Bench, but it does not validate boundary accuracy for transfer to new libraries or query distributions. I request an execution-based audit: sample generated T− statements, construct or extract queries of the forms they describe, run the focal skill and the named neighbor on those queries, and report how often the focal skill fails while the neighbor succeeds, separately for each executor or at least for the strongest and weakest executors.","section":"§5.3, Table 5"},{"comment":"The router ablation compares a base card containing the skill name and original description with a +T− card that additionally contains the T− field. Because the +T− arm always has strictly more text, the +3.62-point improvement cannot be attributed specifically to negative-boundary semantics; any additional discriminative text, or even a neutral filler of the same length, might help the router compare candidates. A control arm that appends an equal-length positive or neutral field (for example, the candidate's own T+ or B, or a benign paraphrase) is needed to isolate the contribution of the negative boundary. Without this control, the attribution of the end-to-end gain to T− as exclusion evidence is not cleanly established.","section":"§5.3, Table 5"}],"minor_comments":[{"comment":"Appendix D refers to \"main-paper Table 3\" for the 95% confidence intervals, but the intervals appear in Table 5; the cross-reference should be corrected.","section":"Appendix D"},{"comment":"The sentence in §5.2 that T− \"still cannot be produced by single-document expansion\" is too strong because §4.3 describes a singleton prompt that infers likely confusable tasks from a single document alone. The statement should be qualified to cluster-contrastive T− or acknowledge the singleton case.","section":"§5.2, §4.3"},{"comment":"The leave-one-out margins d(T+) and d(B) in Table 8 are defined in the table caption, but the main text refers to Figure 4 for the same quantities; it would help to state the definition consistently near the first mention of leave-one-out contributions.","section":"Table 8"},{"comment":"The limitations section addresses the recall-dependence of T− and the structural nature of the clustering sweep, but it does not mention the absence of execution-based validation of generated T− boundaries; adding that caveat would make the scope clearer.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"The paper is well-executed and the central empirical claim is plausible, but the two load-bearing issues above should be addressed: an execution-based audit of T− boundaries and a control arm for the router ablation. The first is feasible with the existing SRA-Bench executors and would substantially reduce the main generalization risk. I do not see grounds for rejection; the work is within scope for an IR audience and the contribution is sufficiently novel. I would also encourage the authors to release at least the compiled Capability Pages or a sample of generated T− statements at revision time, since the paper currently states that code and data will be released upon publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth your time. The core idea is real: skill documents are lossy observations of executable regions, and the missing information is the boundary between neighboring capabilities. The executable-region formalism and the lower bound in Eq. (5) are clean, and the bound is correctly identified as document-imposed rather than retriever-imposed. The practical contribution is the cluster-contrastive Capability Page, with a positive trigger, a discriminative body, and a negative boundary T- that is compiled offline by comparing neighbors. The two-view deployment - keeping T- out of the similarity index but exposing it on the routing card - is a smart, transferable design, and the index-with-T- ablation backs it up.\n\nWhat the paper does well: the evaluation is unusually thorough for this area. Five retrievers, four executors, six datasets, plus a cross-distribution Chinese check. The ablations are thoughtful: leave-one-out field contributions, top-1 hit preservation, a doc2query baseline. The per-dataset numbers in the appendix let you see where the method helps and where it doesn't. The gains are consistent and not huge - mean Recall@10 +2.94, end-to-end +3.62 - which is the right scale for a representation rewrite that leaves online models untouched.\n\nWhere the soft spots are, in proportion:\n\nFirst, and most important: T- is never checked against actual execution. The compiler reads only skill documents and a human QC pass checks clarity, not correctness. The end-to-end gain shows that T- errors do not dominate on SRA-Bench, but it does not validate the boundary as a true executable-region separator. On a new library or query distribution, an LLM-written boundary could over-exclude solvable queries and the benefit could vanish. This is a genuine limitation, not a fatal one, but the authors should provide execution-based evidence that T- statements match R(si).\n\nSecond, the router ablation (Table 5) lacks a length-matched positive control. Both arms have name and description; the +T- arm adds a field. Part of the gain could simply be additional text, not the negative content. A control field with equal character count but positive content would isolate the effect. This is a moderate omission.\n\nThird, the compiler is DeepSeek-V4-Pro and one of the four executors is the same model family. That is a minor concern because gains persist across three other executors, but it would be nice to see at least one open-weight compiler.\n\nNo code or data has been released yet; the authors say it will come upon publication. That is normal for a preprint.\n\nOverall: the central claim holds up about as far as it is tested. The missing validation of T- against execution is the thing I would push on in review. Send it to a serious referee. Ask for (a) execution-based T- validation, (b) a positive-field control in the router arm, and (c) ideally an open-weight compiler as one condition. I would cite it.","headline":"A genuinely useful representation-side fix for skill retrieval, carefully evaluated, with one real gap (T- never validated against execution) and one missing control (a text-length-matched positive field in the router ablation).","tokens_in":17833,"tokens_out":1585,"would_cite":true,"duration_ms":15904,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a large share of skill-routing error is baked into skill documents, and that offline-compiled Capability Pages with positive and negative evidence improve first-stage recall and end-to-end task success without…","keywords":["skill retrieval","LLM agents","Capability Pages","executable region","negative boundary","cluster-contrastive representation","skill routing","document-side retrieval error"],"falsifier":"Take a random sample of compiled $T^-$ statements, execute the focal skill on the queries its $T^-$ says belong to a neighbor, and measure how often the focal skill actually succeeds; a nontrivial failure rate would mean the boundary is unreliable and the reported +3.62 end-to-end gain would not generalize.","tokens_in":16889,"feed_emoji":"🧭","tokens_out":9284,"duration_ms":69608,"temperature":0.7,"pith_summary":"As skill libraries grow, an LLM agent must pick one skill from candidates that often look alike: two calculators for the QT interval can differ only in the formula name, and two math skills can share vocabulary yet solve different query sets. The paper's central claim is that a portion of this routing error is not a retriever problem at all—it is missing from the indexed text, so no better scorer can remove it. To fix that, it proposes Capability Pages, offline-compiled additions to each skill document that state a positive trigger (when to use this skill), a negative boundary (which similar requests belong to a neighbor), and a discriminative rule. On a benchmark of 26,262 skills, the rewritten index improves Recall@10 for all five tested retrievers by 2.94 points on average, and exposing the negative boundary to the router improves end-to-end task success by 3.62 points across four executors. The reason to care is that routing improves without touching online models, by editing the offline library instead.","feed_headline":"Rewriting skill docs lifts recall on every retriever tested","feed_subtitle":"Offline Capability Pages add 2.94 recall points and 3.62 task-success points on average.","key_machinery":"The central object is the Capability Page, a per-skill text card compiled offline by comparing each skill with its nearest neighbors in embedding space. Each page carries three fields: a positive trigger $T^+$ (query forms that should select the skill), a negative boundary $T^-$ (confusable queries that belong to a named neighbor), and a discriminative body $B$ (the core formula or decision rule). The theoretical carrier is the document-imposed lower bound: when two documents are identical but executable regions differ, any scorer reading only $(q,x)$ must err on at least half the symmetric difference $\\mathcal{R}(s_i) \\triangle \\mathcal{R}(s_j)$, so the only cure is changing the observation. The mechanism works by giving the retrieval index the positive view $(T^+, B, x)$ and giving the router the exclusion view $T^-$, which keeps negative descriptions out of similarity embeddings while still using them for comparative candidate selection.","core_discovery":"The paper models a skill's capability as its executable region $\\mathcal{R}(s_i)$—the set of queries the executor can actually solve—and treats the skill's document $x_i$ as a lossy observation of that region. From this it derives a diagnostic decomposition splitting retrieval error into document error (information absent from the text) and retriever error (failure to read what is there), plus a lower bound: if two skills have identical documents but different executable regions, every scoring function over the raw documents must make an error proportional to the functional difference between the skills. The proposed remedy is the Capability Page: a cluster-contrastive representation whose positive trigger $T^+$ and discriminative body $B$ expand what the retrieval index can match, while the negative boundary $T^-$ names neighboring skills' query regions and is reserved for the routing card, where it rejects confusable candidates. Empirically, the index view improves Recall@10 for BM25, TF-IDF, BGE-M3, and two Qwen3 embeddings, and the $T^-$ routing card improves end-to-end success over all four tested executors; the same fixed encoder reaches 73.07% MRR@50 on a Chinese skill library.","pith_inferences":["A natural next step is to audit generated negative boundaries by execution rather than manual review, feeding misclassified $T^-$ entries back into the compiler to close the boundary loop.","The document-imposed lower bound suggests a diagnostic tool for any skill library: estimate document error versus retriever error by comparing an optimal reader on raw documents with the observed retriever, then decide whether to invest in better text or better models.","The Capability Page schema could generalize beyond agent skills to API documentation, tool catalogs, or RAG corpus entries where near-duplicate text hides different behaviors."],"forward_implications":["Skill libraries can be improved text-side: rewriting the indexed skill documents yields recall gains for sparse and dense retrievers alike, with BM25 gaining 7.63 Recall@10 points.","Keeping the negative boundary out of the similarity index and exposing it only on the routing card is the right division of labor; appending $T^-$ to the indexed tail hurts all three dense retrievers.","The positive trigger and discriminative body contribute in different regimes: $T^+$ supplies query-form coverage, largest on LogicBench and ToolQA, while $B$ provides a compact capability fingerprint.","The gains transfer across language and distribution: with a fixed encoder, $B$ adds 3.60 MRR@50 points and $T^+$ another 2.97 on a predominantly Chinese skill library.","Capability Pages complement rather than replace model-side improvements, targeting the document term of the error decomposition while stronger encoders and rerankers target the retriever term."],"supporting_citations":[{"why":"Supplies SRA-Bench, the 26,262-skill benchmark and its six dataset evaluators used for the main retrieval and routing experiments.","marker":"Su et al., 2026"},{"why":"Provides the SSL-SkillDiscovery structured-representation baseline, 72.95% MRR@50, that the transfer evaluation compares against.","marker":"Liang et al., 2026"},{"why":"Provides BGE-M3, one of the five first-stage retrievers whose Recall@10 is measured.","marker":"Chen et al., 2024"},{"why":"Provides the Qwen3-Embedding models used both as dense retrievers and as the encoder for clustering skills into neighborhoods.","marker":"Zhang et al., 2025"},{"why":"Supplies the DeepSeek-V4-Pro model that executes the cluster-contrastive compiler writing $T^+$, $T^-$, and $B$.","marker":"DeepSeek-AI, 2026"},{"why":"Provides the doc2query document-expansion method used as the ablation baseline to show single-document expansion does not explain the index-view gains.","marker":"Nogueira et al., 2019"},{"why":"Supplies the BM25 formulation used as one of the sparse first-stage retriever baselines.","marker":"Robertson & Zaragoza, 2009"},{"why":"Supplies the ReAct execution engine used for ToolQA in the end-to-end routing evaluation.","marker":"Yao et al., 2023"}],"fun_headline_variants":["Offline skill pages lift Recall@10 by 2.94 on average","Capability Pages boost task success by 3.62 points","Cluster-contrastive docs improve all 5 retriever scores","Skills know their neighbors: New pages fix retrieval error","Document error is fixable: off-line rewrite gains recall"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes the offline compiler's negative-boundary statements correctly mark queries the focal skill cannot solve, yet it verifies them only with a manual quality-control pass rather than by executing the skills.","fun_headline_variants_meta":{"raw":{"variants":["Offline skill pages lift Recall@10 by 2.94 on average","Capability Pages boost task success by 3.62 points","Cluster-contrastive docs improve all 5 retriever scores","Skills know their neighbors: New pages fix retrieval error","Document error is fixable: off-line rewrite gains recall"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000225,"raw_usage":{"total_tokens":1546,"prompt_tokens":1112,"completion_tokens":434,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":346}},"tokens_in":728,"tokens_out":434,"duration_ms":4241,"temperature":1.0,"reasoning_tokens":346,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T14:38:38.563225+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of compiled $T^-$ statements, execute the focal skill on the queries its $T^-$ says belong to a neighbor, and measure how often the focal skill actually succeeds; a nontrivial failure rate would mean the boundary is unreliable and the reported +3.62 end-to-end gain would not generalize.","supporting_citations":[{"cited_title":"M3-Embedding: Multi-linguality, multi-functionality, multi-granularity text embeddings through self-knowledge distillation","cited_arxiv_id":null,"evidence_quote":"Provides BGE-M3, one of the five first-stage retrievers whose Recall@10 is measured."}],"review_version":2}