Pith. sign in

REVIEW 3 major objections 5 minor 1 cited by

Skill retrieval is not document retrieval: top-K success needs query-conditioned skill compatibility, recovered from LLM rejections usually thrown away.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-12 15:14 UTC pith:V6YRODC2

load-bearing objection Solid agent-IR systems paper: query-conditional skill compatibility is real, reject-as-resource is useful, and the bi-encoder vs cross-encoder split is cleanly argued—with the LLM oracle as the main provisional piece. the 3 major comments →

arxiv 2606.03565 v4 pith:V6YRODC2 submitted 2026-06-02 cs.IR

Skill Is Not Document: A Query-Conditional Benchmark and Two-Stage Retriever for LLM Agent Skill Routing

Zifei Wang , Wei Wen , Qiang Ji , Ruizhi Qiao , Xing Sun This is my paper
classification cs.IR
keywords skill retrievalLLM agentsquery-conditioned compatibilityreject-as-resourcebi-encodercross-encoder rerankerbilingual benchmarkagent skill routing
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

LLM agents compose reusable skills, so the front-end problem is retrieving the right set of skills for a user request. Document retrieval can treat relevant items as independent; skill retrieval cannot, because two skills that each look plausible may still conflict under a particular query—overlapping roles, mismatched ecosystems, or forced stacking. That query-conditioned compatibility factor is not recoverable from independent relevance scores alone. The paper shows that standard skill–query synthesis pipelines already produce a dense signal for it in the form of LLM WRITE/SKIP decisions, which are usually discarded. R3-Skill keeps those rejections as compatibility supervision, builds a bilingual Chinese–English benchmark with expert-checked test ground truth, and trains a two-stage retriever. Gradient analysis and ablations explain why the same signal is weak under a bi-encoder InfoNCE objective with bilateral balancing, yet useful as graded listwise labels in a cross-encoder reranker. The resulting pipeline improves joint set retrieval on the new benchmark.

Core claim

Top-K joint correctness in skill retrieval factors into per-skill relevance times a query-conditioned compatibility term C(q, S*_q) that independent relevance cannot recover. LLM WRITE/SKIP rejections supply usable binary supervision for C; injecting them as graded labels in a cross-encoder improves joint set metrics, whereas injecting them into the tested bi-encoder InfoNCE objective under bilateral balancing does not help.

What carries the argument

Reject-as-Resource (R3): retain LLM SKIP annotations as compatibility negatives, train R3-Embedding with multi-positive InfoNCE plus a sibling-reward term, then train R3-Reranker with graded ListNet labels (GT=3, SKIP partner=1, other=0) so the cross-encoder can enforce C-conditioned ranking.

Load-bearing premise

The load-bearing premise is that LLM WRITE/SKIP joint-plausibility judgments are a faithful enough stand-in for true skill compatibility under real user requests, even though training labels are not exhaustively human-verified.

What would settle it

A multi-expert or real-user study that systematically re-labels a large sample of multi-skill WRITE/SKIP decisions and shows that models trained on corrected labels reverse the reported Set-Compat gains, or that SKIP-as-graded-label no longer beats the no-SKIP reranker.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Agent skill routers should treat joint set correctness (Set-Compat) as a first-class metric alongside Hit@K and NDCG, not as a side effect of independent relevance.
  • Synthesis pipelines that currently discard LLM rejections should keep them as compatibility supervision rather than pure low-quality waste.
  • Bi-encoders are a poor place to inject query-conditional compatibility under shared-vector bilateral balancing; cross-encoders can use the same signal as graded ranking labels.
  • Bilingual and cross-lingual skill routing (especially Chinese queries over English skill pools) needs explicit compatibility training, not only multilingual embedding alignment.
  • Sandbox and cold-start deployments that install only top-K skills per session become safer if the retriever respects query-conditioned non-co-occurrence.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Any retrieval setting where the returned set is jointly executed (tools, workflows, agents, code packs) may inherit the same relevance-times-compatibility decomposition, not only agent skills.
  • The 8-class rejection taxonomy suggests training separate heads or curricula for hard domain mismatch versus query-conditional forced chaining, rather than one global negative class.
  • Expanding the sparse Chinese skill pool and human-verifying training SKIPs would likely shift absolute numbers more than the bi-encoder vs cross-encoder split, which is structural.
  • End-to-end agent completion metrics may still diverge from Set-Compat if downstream planners re-order or drop retrieved skills; the paper’s offline gains need a handoff study.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that skill retrieval for LLM agents differs from document retrieval because top-K joint correctness depends on a query-conditioned compatibility factor C(q,S*_q) that independent per-skill relevance cannot recover. It constructs R3-Skill, a bilingual (Chinese–English) skill-routing benchmark with 10,246 skills, 41,592 WRITE queries, and 32,828 retained LLM SKIP annotations organized into an 8-class rejection taxonomy, with multi-expert verification of the test-set ground truth and a skill-pool-disjoint train/test split. On this data the authors train a two-stage system: R3-Embedding (multi-positive InfoNCE with a sibling-reward term on Qwen3-Embedding-0.6B) and R3-Reranker (graded ListNet with labels {3,1,0} for GT / SKIP partners / others on Qwen3-Reranker-0.6B). Appendix B derives bilateral balancing for bi-encoder SKIP injection (Theorem 1); ablations show SKIP hurts or fails to help the bi-encoder while helping the cross-encoder. The full pipeline reports Hit@1=0.7521, NDCG@10=0.8173, Set-Compat=0.3188 on R3-Skill and also leads on SkillRet transfer metrics.

Significance. If the results hold, the paper makes a concrete contribution to agent skill routing: it formalizes why skill retrieval is not document retrieval, releases a sizable bilingual resource that retains the usually discarded LLM rejection signal, and shows a practical two-stage training recipe with a clean dual-encoder gradient argument and stage-wise ablations. Strengths include the held-out skill-pool split, multi-expert test verification, shared-upstream reranker comparisons, SkillRet transfer, the Set-Compat metric, and the planned open release of data, weights, and scripts. These are useful for the growing ecosystem of agent skills (Anthropic Agent Skills, SkillRouter, SkillRet, etc.) where full-library prompting is infeasible and multi-skill compatibility matters.

major comments (3)
  1. §1.3 and §2.1–2.4 operationalize C(q,S*_q) via DeepSeek-V4-Pro / Qwen3 WRITE–SKIP joint-plausibility judgments; training SKIP labels are not exhaustively human-verified (only multi-skill test GT intent is multi-expert reviewed; Table 3). If those judgments systematically diverge from real multi-skill executability (sandbox install failures, ecosystem conflicts, or forced chaining an LLM never sees), then graded ListNet supervision and Set-Compat gains largely recover the LLM’s own rejection surface. The Limitations section already flags this; a load-bearing revision would add a human agreement study on a stratified SKIP sample (by direction and reject class) or a small end-to-end executability check, and report inter-annotator / LLM–human agreement so readers can bound oracle noise.
  2. Table 9 and §4.4: zh2en (105) and zh2zh (92) are very small diagnostic slices, and Chinese skills are sparse (246 raw; §2.2, Limitations). Cross-lingual claims and the en2zh forced_chaining contrast are interesting, but the paper should either demote zh2* numbers to qualitative trends only (as partially done) or enlarge those slices / the Chinese skill pool before treating bilingual coverage as a primary contribution. Without that, the four-direction framing overstates the strength of the Chinese-side evidence relative to en2en/en2zh.
  3. §4.1 and Appendix A define Set-Compat only on |GT|≥2 as 1 iff S*_q ⊆ top-m with m=|GT|. This is a useful set-level metric, but the manuscript should clarify how sensitive the reported 0.3188 is to the fixed reranker pool size (top-20) and to the graded labels {3,1,0}. A short sensitivity table (pool 10/20/50; label schemes binary vs graded) would show whether the Set-Compat gain is robust or an artifact of the engineering budget noted in Limitations (vi).
minor comments (5)
  1. Figure 1 caption reports p95=5,526 / p99=9,983 while the body text says p95=5,524 / p99=9,976; align the numbers.
  2. §3: release checklist promises optimizer, LR, batch size, epochs, seed, hardware, and checkpoint rule for both models; ensure these appear in the camera-ready or appendix so the two-stage recipe is fully reproducible.
  3. Table 5 note that R@K equals Comp@K on R3-Skill because |GT|≤3≤K is helpful; consider stating this once in Appendix A and dropping the redundant Comp columns on R3-Skill tables to reduce clutter.
  4. Appendix D prompts are valuable; a short note on temperature / decoding settings used for DeepSeek-V4-Pro and Qwen3-235B would help others regenerate or audit the synthesis pipeline.
  5. Typos / formatting: “Y outu Lab”, “difficult”, “official”, and occasional full-width punctuation in the PDF should be cleaned for the camera-ready.

Circularity Check

0 steps flagged

Empirical IR pipeline with held-out skill pools and expert-checked test GT; no derivation reduces by construction to its inputs.

full rationale

This paper is a benchmark-and-retriever systems paper, not a first-principles derivation. The formalization Pr[S*_q ⊆ top-K|q] ≈ (product of per-skill relevance) · C(q,S*_q) is a modeling decomposition that motivates retaining LLM WRITE/SKIP as supervision; C is operationalized as binary SKIP labels for training, not algebraically identified with the reported metrics. R3-Embedding uses multi-positive InfoNCE with a sibling-reward term; R3-Reranker uses graded ListNet with SKIP partners as intermediate labels (label=1) that are explicitly not counted as GT at evaluation. Theorem 1 (Appendix B) is a standard gradient analysis of InfoNCE bilateral balancing on a shared bi-encoder vector—it explains an ablation outcome, it does not force the headline numbers. Train and test skill pools are disjoint; test multi-skill GT is multi-expert verified; the pipeline is also evaluated on the external SkillRet official test set, where it leads on all reported columns. Ablations (Table 10) show SKIP hurts the bi-encoder and helps the cross-encoder on Hit@1/NDCG/Set-Compat—standard empirical evidence, not a fit renamed as prediction. Concerns about LLM oracle faithfulness for true executability are assumption/correctness risks, not circularity under the stated patterns. No self-definitional loop, no fitted-input-as-prediction, no load-bearing uniqueness theorem from overlapping authors, and no renaming of a known result as a derivation.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 3 invented entities

The central empirical claim rests on standard contrastive/listwise IR machinery plus domain assumptions that LLM joint-plausibility judgments and near-neighbor hard sampling define useful compatibility supervision. Free parameters are ordinary training knobs (temperature, sibling weight, graded labels, cluster K, candidate pool size). No new physical entities are postulated; R3-Skill and Set-Compat are constructed artifacts, not independent natural kinds.

free parameters (6)
  • InfoNCE temperature τ
    Set to 1/30 in the multi-positive embedding loss; scales logits and thus gradient magnitudes for positives, hard negatives, and siblings.
  • sibling-reward weight λ
    Set to 0.25 on Lsib; controls how strongly multi-GT siblings are pulled without being treated as in-batch negatives.
  • graded ListNet labels {3,1,0}
    Heuristic compatibility grades for GT / SKIP partners / other candidates in the reranker; intermediate grade 1 is a design choice, not derived.
  • KMeans K and hierarchy merge
    K=40 chosen from {10,20,30,40} via cluster quality and human spot checks, then merged into 8 super-domains that constrain hard multi-skill sampling.
  • reranker candidate pool size (top-20)
    Fixed by engineering budget and prior skill-routing settings; limits what Set-Compat can recover after embedding recall.
  • hard-negative mining band
    Offline negatives from Qwen3-Embedding-8B with rank in [20,50) and cosine < 0.85; defines the hardness of embedding contrast.
axioms (5)
  • domain assumption Top-K joint skill correctness approximately factors as product of per-skill relevance times a query-conditioned compatibility correction C(q,S*_q).
    Stated in §1.3 as the modeling decomposition that motivates separate compatibility supervision; not derived from first principles.
  • domain assumption LLM WRITE/SKIP joint-plausibility judgments are valid binary supervision for whether a skill set should be jointly retrieved under a query.
    Operational definition of C in §1.3 and the Reject-as-Resource pipeline in §2.1–2.4; training labels rely on this oracle.
  • domain assumption Near-neighbor hard-constraint sampling (same sub-cluster for k=2, same super-domain for k=3) yields informative compatibility negatives rather than trivial unrelated SKIPs.
    §2.3 sampling design; acceptance rates 56.3% (k=2) and 36.5% (k=3) are used as empirical proxies for C.
  • standard math Standard multi-positive InfoNCE and listwise CE / ListNet are appropriate objectives for bi-encoder recall and cross-encoder reranking respectively.
    §3 method; classical contrastive and ranking losses with the paper’s sibling-mask and graded-label modifications.
  • ad hoc to paper A shared bi-encoder embedding of a skill cannot freely encode query-specific push/pull forces without bilateral balancing dilution (Theorem 1).
    Appendix B gradient analysis specific to injecting SKIP partners into InfoNCE on a dual encoder; justifies placing SKIP only at the reranker.
invented entities (3)
  • R3-Skill benchmark (skills, WRITE queries, SKIP annotations, 8-class reject taxonomy) no independent evidence
    purpose: Provide bilingual training/eval data and compatibility supervision that prior skill datasets discard.
    Constructed resource, not an external natural object; value depends on synthesis quality and expert test checks.
  • Set-Compat metric no independent evidence
    purpose: Score whether the entire multi-skill GT set appears in the top-m positions simultaneously.
    Paper-defined joint-set metric for C→1 at the top-m boundary; useful but not independently standardized outside this work.
  • skill compatibility factor C(q,S*_q) no independent evidence
    purpose: Name the query-conditioned correction beyond independent relevance in the top-K decomposition.
    Modeling construct operationalized via LLM WRITE/SKIP; no external measurement protocol independent of that oracle.

pith-pipeline@v1.1.0-grok45 · 24586 in / 4083 out tokens · 41894 ms · 2026-07-12T15:14:27.914164+00:00 · methodology

0 comments
read the original abstract

LLM agents often solve complex tasks by composing skills, making skill retrieval a front-end component of agent systems. Unlike document retrieval, top-K correctness in skill retrieval depends not only on the relevance of each query-skill pair, but also on whether the retrieved skills can work together under the query. This query-conditioned "skill compatibility" cannot be recovered from independent relevance alone. However, LLM-based synthesis pipelines already produce a useful signal for it: the LLM's own rejection decisions, which specify which skills should not be retrieved together for a given query, but are usually discarded as low-quality data. We propose Reject-as-Resource Retriever (R3) and construct R3-Skill, a bilingual (Chinese-English) benchmark for agent skill routing. R3-Skill covers four language directions and uses LLM-rewritten queries that better approximate user requests; its test-set ground truth is verified by multiple experts. It contains 10,246 skills grouped into 8 thematic super-domains, 41,592 accepted queries, and 32,828 LLM-rejected annotations, further organized into an 8-class rejection-reason taxonomy. R3-Skill keeps this normally discarded rejection signal and uses it as compatibility supervision. On R3-Skill, we train a two-stage retriever consisting of R3-Embedding and R3-Reranker. Gradient analysis explains why this query-conditional signal is weak when injected into the tested bi-encoder objective under bilateral balancing, while a cross-encoder can use it as graded ranking supervision; R3-Skill ablations support this split. The R3-Embedding + R3-Reranker pipeline reaches Hit@1 = 0.7521, NDCG@10 = 0.8173 and Set-Compat = 0.3188 on R3-Skill. The dataset, model weights, and evaluation scripts will be open-sourced.

Figures

Figures reproduced from arXiv: 2606.03565 by Qiang Ji, Ruizhi Qiao, Wei Wen, Xing Sun, Zifei Wang.

Figure 1
Figure 1. Figure 1: Qwen3 tokenizer length distribution over the 10,246 skills in R3-Skill, computed on the concatenation of name + description + body. Mean = 2,073 tokens, p95 = 5,526, p99 = 9,983, max = 31,571; loading the entire library at once requires roughly 21.2M input tokens—and that is for skill text alone, before counting system prompts, user queries and LLM outputs, already far beyond the single-call context window… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of skill compatibility. Whether the same skill pair should be jointly retrieved can flip across queries—the core distinction between skill retrieval and document retrieval. means that even though every member is independently a plausible candidate, the set conflicts under query q—e.g. overlapping functionality, inconsistent style, or incompatible ecosystems—and should not be jointly retrieved;… view at source ↗
Figure 2
Figure 2. Figure 2: Illustration of skill compatibility. Whether the same skill pair should be jointly retrieved can flip across queries—the core distinction between skill retrieval and document retrieval. Formalization (modeling decomposition). Given a query q and the target skill set S ∗ q ⊆ S, we decompose the probability of jointly retrieving S ∗ q in the top-K into two factors: 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: overviews the construction pipeline, from raw collection to the final splits [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Bilateral balancing on the shared e(s ′ ): a SKIP push under query q is balanced by a positive pull under another query q˜; their equilibrium fixes the geometric position of e(s ′ ). B.2 DERIVATION OF THEOREM 1 The InfoNCE loss on the candidate pool C is LNCE = − log ps+ . Among the logits {zc}c∈C, only zs ′ depends on e(s ′ ), with ∂zs ′/∂e(s ′ ) = e(q)/τ . Differentiating yields ∇e(s ′) L (q) NCE = − ∂ l… view at source ↗
Figure 5
Figure 5. Figure 5: Cluster quality vs. K (K ∈ {10, 20, 30, 40}). The three indicators are inertia (within-cluster sum of squared distances, lower is better), intra-cluster sim (within-cluster topical similarity, higher is better), and inter-centroid sim (similarity between adjacent centroids, lower is better); for joint comparison, all three curves are normalized to [0,1] and re-oriented so that “higher is better”. K = 40 is… view at source ↗
Figure 6
Figure 6. Figure 6: Hierarchical taxonomy view of the R3-Skill skill pool under K = 40 clustering. The inner ring shows the 8 super-domains (A–H), obtained by topic-merging the 40 sub-clusters; the outer ring shows the 40 sub-clusters, with sub-cluster names and arc lengths proportional to skill counts. Inner and outer rings sharing the same color family belong to the same super-domain. The figure exposes the hierarchical str… view at source ↗
Figure 7
Figure 7. Figure 7: LLM annotation flow: skill set → DECISION → WRITE branch generates queries in 6 styles × 4 directions / SKIP branch is archived → enters taxonomy classification and multi-expert verification [PITH_FULL_IMAGE:figures/full_fig_p015_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Stacked bar chart of the 8 reject reason classes across the 4 language directions. en2en is dominated by domain_mismatch; in en2zh, forced_chaining jumps to 56.0%—reject morphology differs structurally between Chinese and English [PITH_FULL_IMAGE:figures/full_fig_p016_8.png] view at source ↗

Review history (2 revisions) →

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. SkillSight: Seeing Through Shared Descriptions for Accurate Skill Retrieval

    cs.AI 2026-07 conditional novelty 6.0

    SkillSight filters corpus-shared descriptive patterns from dense embeddings and re-weights rare query terms, improving skill retrieval Recall@10 by up to 20 percentage points with no additional training.

Reference graph

Works this paper leans on

18 extracted references · cited by 1 Pith paper

  1. [1]

    ҂ିԛགྷ skill֥nameĠ҂ିԛགྷૼཁՖႇ໓ nameಪ෪ཿ čPDF / API / SQL / GitHub ᆃোĎᄍྸb

  2. [2]

    ҂ေᅶӒ description / bodyb

  3. [3]

    ۱skill টϺ෰ປӮ

    сྶିಞ AI؎"۱skill টϺ෰ປӮ"b 5.൤b Skill: {skill_rendered} en2zh — English skill → Chinese query. ۱AI ᇹ൭Ϻફቓ൙b ٺ**ႇ໓ skill**ି৯čname / description / bodyऌ෱ ۱**໙ี**đѩ౏ **҂ି** ԛགྷ skillႇ໓ nameᇏ໓ᆰၲđ ္҂ିᅶӒ description / bodyჰओb ቎࿖Ď ğ{style_desc} ཿቔ႗ჿඏğ

  4. [4]

    Ⴈ**ᇏ໓**؇50~300ࡆן1~2۬ڄ Ќӻഈ૫ପ่b

  5. [5]

    ປಆႨᇏ໓đ**҂ေԛགྷႇ໓ֆՍ**ಪ෪ཿೂ PDF / API / SQL / GitHub ᆃোᄍྸĎ b

  6. [6]

    ҂ିԛགྷ skillႇ໓ nameି/ಀ૭ඍb

  7. [7]

    ҂ေᅶӒ description / bodyओሰb

  8. [8]

    You play a real user asking an AI assistant for help

    сྶିಞ AI؎"۱skill"b 6.൤b Skill: {skill_rendered} en2en — English skill → English query. You play a real user asking an AI assistant for help. You can only see the skill below (name / description / body excerpt). The question you write **must not** mention the skill's name (or close synonyms), and **must not** copy sentences from description / body verbatim....

  9. [9]

    You may layer in 1~2 secondary styles to keep it natural, but stay anchored to the main style above

    Write in English, 50~300 words. You may layer in 1~2 secondary styles to keep it natural, but stay anchored to the main style above

  10. [10]

    Do not mention the skill's name or any near-synonym translation of it

  11. [11]

    Do not copy whole sentences from description / body; rephrase

  12. [13]

    Skill: {skill_rendered} 17 zh2en — Chinese skill → English query

    Output only the question itself; no prefixes, tags, or explanations. Skill: {skill_rendered} 17 zh2en — Chinese skill → English query. You play a real user asking an AI assistant for help in English. The skill below is in **Chinese** (name / description / body excerpt). Write an **English user question** based on it; you **must not** mention the skill's n...

  13. [14]

    You may layer 1~2 secondary styles for naturalness, but the main style above must show

    Write in **English** only, 50~300 words. You may layer 1~2 secondary styles for naturalness, but the main style above must show

  14. [15]

    No Chinese characters in the output

  15. [16]

    Do not surface the skill's name or any direct translation of it; describe by function/scenario instead

  16. [17]

    Do not copy sentences from description / body

  17. [18]

    I should use this skill

    The AI must be able to read your question and infer "I should use this skill"

  18. [19]

    ۱X"ᆃᇕ႗ೖĠ -ᆃહ໙b Ď ğ -৘ğᄝ <DECISION>WRITE</DECISION>໙ีĠ -৘ğᄝ <DECISION>SKIP</DECISION>ඪૼჰၹđ҂ေᄜཿ໙ีb ֒ࣇWRITE ൈĎ ğ -ေ౰ğ{style_desc} -؇50~300ࡆן1~2Ġ - сྶ**๝ൈ**ླေᆃ {k}۱skillൔ

    Output only the question itself; no prefixes, tags, or explanations. Skill: {skill_rendered} D.3 Multi-skill prompts ( k = 2 / k = 3, two-stage CoT WRITE/SKIP) zh2zh — Chinese skill set → Chinese query. b ୆ {k}۱skillି৯čname / description / bodyൻԛğ ᆃ{k}۱skill ** ๝ൈ**ᄝ **߅/ླ౰** ѓሙğ -ປӮ҂ਔ / ૼཁҗಌĠ - ҂఼ྛ௓՘ğ҂ି൞"۱X"ᆃᇕ႗ೖĠ -ᆃહ໙b Ď ğ -৘ğᄝ <DECISION>WRITE</DECISION>...