Pith. sign in

REVIEW 3 major objections 5 minor 2 cited by

Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing

T0 review · 3 major / 5 minor · reviewed 2026-07-12 · grok-4.5

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read Skill retrieval is not document retrieval: top-K success needs query-conditioned skill compatibility, recovered from LLM rejections usually thrown away.

desk verdict Solid agent-IR systems paper: query-conditional skill compatibility is real, reject-as-resource is useful, and the bi-encoder vs cross-encoder split is cleanly argued—with the LLM oracle as the main provisional piece. read the letter →

arxiv 2606.03565 v5 pith:V6YRODC2 submitted 2026-06-02 cs.IR

Zifei Wang , Wei Wen , Qiang Ji , Keyu Chen , Ruizhi Qiao , Xing Sun This is my paper · ORCID
classification cs.IR
keywords skillretrievalLLMagentsquery-conditionedcompatibilityreject-as-resourcebi-encodercross-encoderrerankerbilingualbenchmarkagentrouting
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

LLM agents compose reusable skills, so the front-end problem is retrieving the right set of skills for a user request. Document retrieval can treat relevant items as independent; skill retrieval cannot, because two skills that each look plausible may still conflict under a particular query—overlapping roles, mismatched ecosystems, or forced stacking. That query-conditioned compatibility factor is not recoverable from independent relevance scores alone. The paper shows that standard skill–query synthesis pipelines already produce a dense signal for it in the form of LLM WRITE/SKIP decisions, which are usually discarded. R3-Skill keeps those rejections as compatibility supervision, builds a bilingual Chinese–English benchmark with expert-checked test ground truth, and trains a two-stage retriever. Gradient analysis and ablations explain why the same signal is weak under a bi-encoder InfoNCE objective with bilateral balancing, yet useful as graded listwise labels in a cross-encoder reranker. The resulting pipeline improves joint set retrieval on the new benchmark.

What carries the argument

Reject-as-Resource (R3): retain LLM SKIP annotations as compatibility negatives, train R3-Embedding with multi-positive InfoNCE plus a sibling-reward term, then train R3-Reranker with graded ListNet labels (GT=3, SKIP partner=1, other=0) so the cross-encoder can enforce C-conditioned ranking.

What would settle it

A multi-expert or real-user study that systematically re-labels a large sample of multi-skill WRITE/SKIP decisions and shows that models trained on corrected labels reverse the reported Set-Compat gains, or that SKIP-as-graded-label no longer beats the no-SKIP reranker.

Watch

Extended reading notes

Core claim

Top-K joint correctness in skill retrieval factors into per-skill relevance times a query-conditioned compatibility term C(q, S*_q) that independent relevance cannot recover. LLM WRITE/SKIP rejections supply usable binary supervision for C; injecting them as graded labels in a cross-encoder improves joint set metrics, whereas injecting them into the tested bi-encoder InfoNCE objective under bilateral balancing does not help.

Load-bearing premise

The load-bearing premise is that LLM WRITE/SKIP joint-plausibility judgments are a faithful enough stand-in for true skill compatibility under real user requests, even though training labels are not exhaustively human-verified.

Editorial extensions

If this is right

  • Agent skill routers should treat joint set correctness (Set-Compat) as a first-class metric alongside Hit@K and NDCG, not as a side effect of independent relevance.
  • Synthesis pipelines that currently discard LLM rejections should keep them as compatibility supervision rather than pure low-quality waste.
  • Bi-encoders are a poor place to inject query-conditional compatibility under shared-vector bilateral balancing; cross-encoders can use the same signal as graded ranking labels.
  • Bilingual and cross-lingual skill routing (especially Chinese queries over English skill pools) needs explicit compatibility training, not only multilingual embedding alignment.
  • Sandbox and cold-start deployments that install only top-K skills per session become safer if the retriever respects query-conditioned non-co-occurrence.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Any retrieval setting where the returned set is jointly executed (tools, workflows, agents, code packs) may inherit the same relevance-times-compatibility decomposition, not only agent skills.
  • The 8-class rejection taxonomy suggests training separate heads or curricula for hard domain mismatch versus query-conditional forced chaining, rather than one global negative class.
  • Expanding the sparse Chinese skill pool and human-verifying training SKIPs would likely shift absolute numbers more than the bi-encoder vs cross-encoder split, which is structural.
  • End-to-end agent completion metrics may still diverge from Set-Compat if downstream planners re-order or drop retrieved skills; the paper’s offline gains need a handoff study.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper argues that skill retrieval for LLM agents differs from document retrieval because top-K joint correctness depends on a query-conditioned compatibility factor C(q,S*_q) that independent per-skill relevance cannot recover. It constructs R3-Skill, a bilingual (Chinese–English) skill-routing benchmark with 10,246 skills, 41,592 WRITE queries, and 32,828 retained LLM SKIP annotations organized into an 8-class rejection taxonomy, with multi-expert verification of the test-set ground truth and a skill-pool-disjoint train/test split. On this data the authors train a two-stage system: R3-Embedding (multi-positive InfoNCE with a sibling-reward term on Qwen3-Embedding-0.6B) and R3-Reranker (graded ListNet with labels {3,1,0} for GT / SKIP partners / others on Qwen3-Reranker-0.6B). Appendix B derives bilateral balancing for bi-encoder SKIP injection (Theorem 1); ablations show SKIP hurts or fails to help the bi-encoder while helping the cross-encoder. The full pipeline reports Hit@1=0.7521, NDCG@10=0.8173, Set-Compat=0.3188 on R3-Skill and also leads on SkillRet transfer metrics.

Significance. If the results hold, the paper makes a concrete contribution to agent skill routing: it formalizes why skill retrieval is not document retrieval, releases a sizable bilingual resource that retains the usually discarded LLM rejection signal, and shows a practical two-stage training recipe with a clean dual-encoder gradient argument and stage-wise ablations. Strengths include the held-out skill-pool split, multi-expert test verification, shared-upstream reranker comparisons, SkillRet transfer, the Set-Compat metric, and the planned open release of data, weights, and scripts. These are useful for the growing ecosystem of agent skills (Anthropic Agent Skills, SkillRouter, SkillRet, etc.) where full-library prompting is infeasible and multi-skill compatibility matters.

major comments (3)
  1. §1.3 and §2.1–2.4 operationalize C(q,S*_q) via DeepSeek-V4-Pro / Qwen3 WRITE–SKIP joint-plausibility judgments; training SKIP labels are not exhaustively human-verified (only multi-skill test GT intent is multi-expert reviewed; Table 3). If those judgments systematically diverge from real multi-skill executability (sandbox install failures, ecosystem conflicts, or forced chaining an LLM never sees), then graded ListNet supervision and Set-Compat gains largely recover the LLM’s own rejection surface. The Limitations section already flags this; a load-bearing revision would add a human agreement study on a stratified SKIP sample (by direction and reject class) or a small end-to-end executability check, and report inter-annotator / LLM–human agreement so readers can bound oracle noise.
  2. Table 9 and §4.4: zh2en (105) and zh2zh (92) are very small diagnostic slices, and Chinese skills are sparse (246 raw; §2.2, Limitations). Cross-lingual claims and the en2zh forced_chaining contrast are interesting, but the paper should either demote zh2* numbers to qualitative trends only (as partially done) or enlarge those slices / the Chinese skill pool before treating bilingual coverage as a primary contribution. Without that, the four-direction framing overstates the strength of the Chinese-side evidence relative to en2en/en2zh.
  3. §4.1 and Appendix A define Set-Compat only on |GT|≥2 as 1 iff S*_q ⊆ top-m with m=|GT|. This is a useful set-level metric, but the manuscript should clarify how sensitive the reported 0.3188 is to the fixed reranker pool size (top-20) and to the graded labels {3,1,0}. A short sensitivity table (pool 10/20/50; label schemes binary vs graded) would show whether the Set-Compat gain is robust or an artifact of the engineering budget noted in Limitations (vi).
minor comments (5)
  1. Figure 1 caption reports p95=5,526 / p99=9,983 while the body text says p95=5,524 / p99=9,976; align the numbers.
  2. §3: release checklist promises optimizer, LR, batch size, epochs, seed, hardware, and checkpoint rule for both models; ensure these appear in the camera-ready or appendix so the two-stage recipe is fully reproducible.
  3. Table 5 note that R@K equals Comp@K on R3-Skill because |GT|≤3≤K is helpful; consider stating this once in Appendix A and dropping the redundant Comp columns on R3-Skill tables to reduce clutter.
  4. Appendix D prompts are valuable; a short note on temperature / decoding settings used for DeepSeek-V4-Pro and Qwen3-235B would help others regenerate or audit the synthesis pipeline.
  5. Typos / formatting: “Y outu Lab”, “difficult”, “official”, and occasional full-width punctuation in the PDF should be cleaned for the camera-ready.

Circularity Check

0 steps flagged · score 0.0 of 10

Empirical IR pipeline with held-out skill pools and expert-checked test GT; no derivation reduces by construction to its inputs.

full rationale

This paper is a benchmark-and-retriever systems paper, not a first-principles derivation. The formalization Pr[S*_q ⊆ top-K|q] ≈ (product of per-skill relevance) · C(q,S*_q) is a modeling decomposition that motivates retaining LLM WRITE/SKIP as supervision; C is operationalized as binary SKIP labels for training, not algebraically identified with the reported metrics. R3-Embedding uses multi-positive InfoNCE with a sibling-reward term; R3-Reranker uses graded ListNet with SKIP partners as intermediate labels (label=1) that are explicitly not counted as GT at evaluation. Theorem 1 (Appendix B) is a standard gradient analysis of InfoNCE bilateral balancing on a shared bi-encoder vector—it explains an ablation outcome, it does not force the headline numbers. Train and test skill pools are disjoint; test multi-skill GT is multi-expert verified; the pipeline is also evaluated on the external SkillRet official test set, where it leads on all reported columns. Ablations (Table 10) show SKIP hurts the bi-encoder and helps the cross-encoder on Hit@1/NDCG/Set-Compat—standard empirical evidence, not a fit renamed as prediction. Concerns about LLM oracle faithfulness for true executability are assumption/correctness risks, not circularity under the stated patterns. No self-definitional loop, no fitted-input-as-prediction, no load-bearing uniqueness theorem from overlapping authors, and no renaming of a known result as a derivation.

Assumptions & free parameters 6 free parameters · 5 assumptions · 3 invented entities

The central empirical claim rests on standard contrastive/listwise IR machinery plus domain assumptions that LLM joint-plausibility judgments and near-neighbor hard sampling define useful compatibility supervision. Free parameters are ordinary training knobs (temperature, sibling weight, graded labels, cluster K, candidate pool size). No new physical entities are postulated; R3-Skill and Set-Compat are constructed artifacts, not independent natural kinds.

free parameters (6)
  • InfoNCE temperature τ
    Set to 1/30 in the multi-positive embedding loss; scales logits and thus gradient magnitudes for positives, hard negatives, and siblings.
  • sibling-reward weight λ
    Set to 0.25 on Lsib; controls how strongly multi-GT siblings are pulled without being treated as in-batch negatives.
  • graded ListNet labels {3,1,0}
    Heuristic compatibility grades for GT / SKIP partners / other candidates in the reranker; intermediate grade 1 is a design choice, not derived.
  • KMeans K and hierarchy merge
    K=40 chosen from {10,20,30,40} via cluster quality and human spot checks, then merged into 8 super-domains that constrain hard multi-skill sampling.
  • reranker candidate pool size (top-20)
    Fixed by engineering budget and prior skill-routing settings; limits what Set-Compat can recover after embedding recall.
  • hard-negative mining band
    Offline negatives from Qwen3-Embedding-8B with rank in [20,50) and cosine < 0.85; defines the hardness of embedding contrast.
assumptions (5)
  • domain assumption Top-K joint skill correctness approximately factors as product of per-skill relevance times a query-conditioned compatibility correction C(q,S*_q).
    Stated in §1.3 as the modeling decomposition that motivates separate compatibility supervision; not derived from first principles.
  • domain assumption LLM WRITE/SKIP joint-plausibility judgments are valid binary supervision for whether a skill set should be jointly retrieved under a query.
    Operational definition of C in §1.3 and the Reject-as-Resource pipeline in §2.1–2.4; training labels rely on this oracle.
  • domain assumption Near-neighbor hard-constraint sampling (same sub-cluster for k=2, same super-domain for k=3) yields informative compatibility negatives rather than trivial unrelated SKIPs.
    §2.3 sampling design; acceptance rates 56.3% (k=2) and 36.5% (k=3) are used as empirical proxies for C.
  • standard math Standard multi-positive InfoNCE and listwise CE / ListNet are appropriate objectives for bi-encoder recall and cross-encoder reranking respectively.
    §3 method; classical contrastive and ranking losses with the paper’s sibling-mask and graded-label modifications.
  • ad hoc to paper A shared bi-encoder embedding of a skill cannot freely encode query-specific push/pull forces without bilateral balancing dilution (Theorem 1).
    Appendix B gradient analysis specific to injecting SKIP partners into InfoNCE on a dual encoder; justifies placing SKIP only at the reranker.
invented entities (3)
  • R3-Skill benchmark (skills, WRITE queries, SKIP annotations, 8-class reject taxonomy)
    purpose: Provide bilingual training/eval data and compatibility supervision that prior skill datasets discard.
    Constructed resource, not an external natural object; value depends on synthesis quality and expert test checks.
  • Set-Compat metric
    purpose: Score whether the entire multi-skill GT set appears in the top-m positions simultaneously.
    Paper-defined joint-set metric for C→1 at the top-m boundary; useful but not independently standardized outside this work.
  • skill compatibility factor C(q,S*_q)
    purpose: Name the query-conditioned correction beyond independent relevance in the top-K decomposition.
    Modeling construct operationalized via LLM WRITE/SKIP; no external measurement protocol independent of that oracle.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing." pith.science (2026). https://pith.science/paper/V6YRODC2

@misc{pith2026260603565,
  author       = {Pith},
  title        = {Pith review of: Skill Is Not Document: Query-Conditioned Compatibility for LLM Agent Skill Routing},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/V6YRODC2}},
  note         = {Machine review of arXiv:2606.03565}
}
abstract

Large language model agents increasingly rely on reusable skills, making skill retrieval a critical front-end component of agent systems. Skill retrieval, however, is not ordinary document retrieval: a useful top-$K$ result must contain individually relevant skills that also form an executable set for the current query. Existing benchmarks and training pipelines largely supervise pairwise relevance and discard the rejection decisions produced when a language model judges a sampled skill combination to be implausible. We introduce R3-Skill, a Chinese--English benchmark that retains these rejections as query-conditioned compatibility supervision. R3-Skill contains 10,246 deduplicated skills, 41,592 accepted queries, and 32,828 rejected annotations across four language directions; all multi-skill test labels were independently reviewed by multiple experts, and 15,962 parseable rejections are organized into an eight-class taxonomy. We further propose a two-stage system composed of R3-Embedding, a multi-positive bi-encoder for large-pool recall, and R3-Reranker, a cross-encoder trained with graded ListNet supervision. Our analysis shows that this signal is stage-dependent, helping cross-encoder reranking while providing no benefit for the tested bi-encoder objective. On R3-Skill, the complete pipeline achieves $75.39\%$ Hit@1, $81.97\%$ NDCG@10, and $33.27\%$ Set-Compat, a $36.6\%$ relative gain over the strongest reranking baseline. It also obtains $83.87\%$ NDCG@10 on SkillRet, demonstrating transfer beyond R3-Skill.

Figures

Figures reproduced from arXiv: 2606.03565 by the authors.

Figure 1
Figure 1. Qwen3 tokenizer length distribution over the 10,246 skills in R3-Skill, computed on the concatenation of name + description + body. Mean = 2,073 tokens, p95 = 5,526, p99 = 9,983, max = 31,571; loading the entire library at once requires roughly 21.2M input tokens—and that is for skill text alone, before counting system prompts, user queries and LLM outputs, already far beyond the single-call context windows of mains… view at source ↗
Figure 2
Figure 2. Illustration of skill compatibility. Whether the same skill pair should be jointly retrieved can flip across queries—the core distinction between skill retrieval and document retrieval. means that even though every member is independently a plausible candidate, the set conflicts under query q—e.g. overlapping functionality, inconsistent style, or incompatible ecosystems—and should not be jointly retrieved; C > 1 mea… view at source ↗
Figure 2
Figure 2. Illustration of skill compatibility. Whether the same skill pair should be jointly retrieved can flip across queries—the core distinction between skill retrieval and document retrieval. Formalization (modeling decomposition). Given a query q and the target skill set S ∗ q ⊆ S, we decompose the probability of jointly retrieving S ∗ q in the top-K into two factors: 3 [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figures from the paper (6 more)
Figure 3
Figure 3. Figure 3: overviews the construction pipeline, from raw collection to the final splits [PITH_FULL_IMAGE:figures/full_fig_p005_3.png]
Figure 4
Figure 4. Figure 4: Bilateral balancing on the shared e(s ′ ): a SKIP push under query q is balanced by a positive pull under another query q˜; their equilibrium fixes the geometric position of e(s ′ ). B.2 DERIVATION OF THEOREM 1 The InfoNCE loss on the candidate pool C is LNCE = − log p…
Figure 5
Figure 5. Figure 5: Cluster quality vs. K (K ∈ {10, 20, 30, 40}). The three indicators are inertia (within-cluster sum of squared distances, lower is better), intra-cluster sim (within-cluster topical similarity, higher is better), and inter-centroid sim (similarity between adjacent centr…
Figure 6
Figure 6. Figure 6: Hierarchical taxonomy view of the R3-Skill skill pool under K = 40 clustering. The inner ring shows the 8 super-domains (A–H), obtained by topic-merging the 40 sub-clusters; the outer ring shows the 40 sub-clusters, with sub-cluster names and arc lengths proportional t…
Figure 7
Figure 7. Figure 7: LLM annotation flow: skill set → DECISION → WRITE branch generates queries in 6 styles × 4 directions / SKIP branch is archived → enters taxonomy classification and multi-expert verification [PITH_FULL_IMAGE:figures/full_fig_p015_7.png]
Figure 8
Figure 8. Figure 8: Stacked bar chart of the 8 reject reason classes across the 4 language directions. en2en is dominated by domain_mismatch; in en2zh, forced_chaining jumps to 56.0%—reject morphology differs structurally between Chinese and English [PITH_FULL_IMAGE:figures/full_fig_p016…

Review history (2 revisions) →

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. SkillSight: Seeing Through Shared Descriptions for Accurate Skill Retrieval

    cs.AI 2026-07 conditional novelty 6.0 of 10

    SkillSight filters corpus-shared descriptive patterns from dense embeddings and re-weights rare query terms, improving skill retrieval Recall@10 by up to 20 percentage points with no additional training.

  2. From Relevance to Execution Utility: Reward-Aware Dynamic Execution Gating for Skill-Based LLM Agents

    cs.AI 2026-08 conditional novelty 5.0 of 10

    A lightweight post-retrieval gate trained on verifier feedback can skip most low-value LLM agent executions while retaining most of the achievable reward.

Pith tools

Reviewed July 12, 2026 · model on record in the stance chip above.