{"id":"6e6ac20c-b502-4dcd-8343-7476be24a512","arxiv_id":"2608.04719","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Canary tools turn a binary wrong-tool error into a typed diagnosis of the reasoning weakness, and a six-type taxonomy separates models by capability.","lead":"A new diagnostic method, canary tools, plants six types of subtly wrong tools into an LLM agent's tool set so that the wrong tool a model picks reveals which reasoning step failed. Across eight models, the authors find that more capable models are generally less susceptible, but capability tier alone does not predict safety.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Canary attribution is underdetermined: for semantic/capability types the LLM rewrites both name and description, so a call may reflect surface attraction rather than the targeted weakness; the de-telling ablation cannot rule this out.","rationale":"The reader's weakest assumption and my concern coincide: the canary call must be caused by the targeted weakness and nothing else. The strongest evidence FOR attribution is the full-condition dilution (§6.6) and the de-telling ablation (§6.7), which rules out phrase-spotting as the sole mechanism. The strongest evidence AGAINST full unconfoundedness is that semantic and capability canaries are not single-axis perturbations: the LLM rewrite changes name and description jointly, and the paper itself admits tell and lure are entangled. The proposed control experiment directly tests the counterfactual. If controls trip at comparable rates, the central diagnostic claim fails, and the paper would need re-analysis of which types are actually measured. If controls are inert, the conditional verdict can be upgraded. No machine-checked proof or released artifact is available to substitute for this test (the release link is absent), so the empirical control is the decisive step. The reader's CONDITIONAL verdict is therefore the right call, unchanged by this pass.","tokens_in":11920,"tokens_out":6681,"duration_ms":81436,"concrete_test":"Build, for each of the six types, a matched-surface control canary: use the same LLM-generated name/description rewrite and the same schema as the real canary, but restore the target property to the correct value (e.g., a capability mirage with identical 'research-grade' wording but where the claimed capability is real and needed for the task; a semantic decoy with the same paraphrased name/description but fresh data). Inject each control alone alongside the real tools, using the declared condition and the same 120 tasks, models, and seeds. If control trap rates are comparable to real-canary trap rates (same order of magnitude), a canary call is not attributable to the targeted weakness and the six-way diagnosis collapses; if control trap rates are near zero while real canary rates stay high, the §3 attribution is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central inference in §3 — 'observing a(q,T∪{c})=c shows the agent's selection was insensitive to the perturbed property' — requires that the canary c differs from the real tool t⋆ only along the targeted axis. This holds for the deterministic schema transforms (parameter, prerequisite, temporal, granularity), but the two types that matter most empirically, semantic and capability canaries, are generated by an LLM that rewrites both the name and the description (Appendix C). A capability mirage therefore differs from its real counterpart in wording, name, and the inflated claim simultaneously; a semantic decoy differs in name, description phrasing, and the staleness flag. A call to c could be driven by any of these differences, or by a generic novelty or positional attraction to the added tool, rather than by the specific reasoning weakness. The full-condition result (CSR drops when more canaries are added, §6.6) argues against a simple 'more tools → more traps' effect, and baseline CSR=0 rules out spontaneous canary calls, so the concern is not that the phenomenon is absent. But the typed diagnosis is not uniquely determined: the same wrong pick could localize to the wrong axis. The subtlety ablation (§6.7) only softens the give-away phrase; the paper concedes it 'does not fully separate a canary's tell from its lure.' Because capability mirage is the only strongly active probe on hosted models (trap rate 0.216 vs. ≤0.094 for the rest, Table 2), the unconfoundedness of this single type is load-bearing for the entire taxonomy's claim to be capability-stratified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces \"canary tools,\" diagnostic probe tools planted in an MCP tool set, each engineered to test one of six tool-selection weaknesses: semantic decoys, parameter traps, capability mirages, prerequisite blindness, temporal decoys, and granularity traps. The authors argue that observing a canary call yields a typed diagnosis rather than a one-bit \"wrong tool\" outcome. They evaluate eight models (six hosted, two 8B open-weight) on 120 tasks under baseline, declared, and full canary-density conditions with three seeds each, plus a 2,880-run subtlety ablation, using a provider-independent judge validated against a second judge and a human sample. The main findings are that canary susceptibility spans roughly 36x across models, that capability tier does not order safety, and that the taxonomy is capability-stratified, with capability mirages being the only probe active on strong hosted models. The framework, schemas, tasks, and logs are released.","tokens_in":12249,"tokens_out":4524,"duration_ms":54119,"significance":"If the central attribution claim holds, this is a genuinely useful diagnostic methodology: it converts a one-bit failure signal into a multi-dimensional profile, and it is unusually carefully evaluated. The empirical apparatus is a strength: a provider-independent judge with a second-judge check (Cohen's kappa = 0.75) and a human check (kappa = 0.90), cluster-robust statistics, deterministic canary generation, and explicit limitations. However, the diagnostic inference for the two empirically most important canary types (semantic and capability) is not directly validated: these canaries differ from the real tool in both name and description, so a call may reflect surface attraction rather than the targeted reasoning weakness. In addition, the CSR-TSR correlation is partly mechanical. The contribution is therefore stronger as a carefully executed measurement framework and a descriptive cross-model finding than as a fully established causal diagnosis of specific reasoning failures.","major_comments":[{"comment":"The central inference in §3 — \"observing a(q,T∪{c})=c shows the agent's selection was insensitive to the perturbed property\" — is valid only if c differs from the real tool t* along the targeted axis alone. This holds for the deterministic schema transforms (parameter, prerequisite, temporal, granularity). For semantic and capability canaries, Appendix C states that an LLM rewrites both the name and the description, so a call to c could be driven by the reworded surface form, the name change, or generic novelty/positional attraction to an added tool, rather than by the specific weakness. This is load-bearing because capability mirage is the only strongly active probe on hosted models (Table 2, hosted average 0.216 versus at most 0.094 for the other types), so the headline empirical claim rests on an unconfoundedness assumption that is not established. The §6.7 de-telling ablation changes only the description phrasing and keeps names and ids fixed, so it does not address name-based or surface-form confounds; the paper itself concedes in the Limitations that the ablation \"does not fully separate a canary's tell from its lure.\" I would suggest adding controlled variants (e.g., name-only rewording, description-only rewording, and a canary with an inflated capability claim but a neutral name) and/or independently validating the diagnosis, for example by coding the trace immediately before a canary call or by comparing canary calls against matched non-canary decoys.","section":"§3, §4, Appendix C, §6.7"},{"comment":"The validation that \"canaries predict failure\" relies on the Spearman rho = -0.34 correlation between per-task CSR and TSR. This correlation is partly circular: a canary is constructed to be wrong for the task (stale data, auth errors, unsatisfiable parameters, or inflated but unreliable output), so a run that calls a canary is, by design, more likely to fail. The correlation is therefore a consistency check that the canaries have the intended effect rather than independent evidence that susceptibility reveals a distinct reasoning weakness. The baseline-versus-declared degradation analysis (Figure 7) is more informative but still conflates trap rate with outcome because a trap counts even when the agent recovers. I recommend framing the correlation explicitly as a sanity check and, if possible, reporting a version that controls for the raw number or timing of canary calls, or that uses recovery-adjusted outcomes.","section":"§6.4"}],"minor_comments":[{"comment":"The definition of per-task CSR as \"mean over tasks of (canary calls / tool calls)\" does not specify how tasks with zero tool calls are handled; Llama 3.1 8B in the full condition appears to make very few calls, and this should be stated explicitly in the metric definition.","section":"§4"},{"comment":"The human validation of the judge uses a sample of 40 task-runs graded by an author; please state whether the human was blind to model identity and canary condition, and report whether the two disagreements shown are the only cases of judge strictness or merely representative examples.","section":"Appendix D"},{"comment":"The radial diagnostic profiles would benefit from a shared legend and explicit axis labels; as printed, it is difficult to compare the six spokes across the small multiples.","section":"Figures 5 and 6"}],"recommendation":"major_revision","confidential_remarks":"The paper is well within scope for cs.AI and the empirical design is unusually careful. The load-bearing issue is the unvalidated attribution for the semantic and capability canary types; if the authors add the controlled experiments suggested in the major comments, I would support publication. The release of the framework and logs is a clear strength."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read it. The core idea is good: plant a perturbed tool that fails along one known axis, and a wrong pick becomes a reason code instead of a one-bit error. Applying misconception-probe design to tool selection is new as far as I know, and the six-type taxonomy is simple enough to be usable. Four of the six types are deterministic schema transforms, which gives clean failure semantics. The empirical work is careful: provider-independent judge with a second-judge check and a small human hand-grading, cluster-robust statistics, a controlled subtlety ablation, and 8,640 runs. The limits section is honest. The capability-mirage finding—that inflated capability claims catch even frontier models while other probes stay inert until small open models—is practically relevant and credible.\n\nThat said, the central attribution claim is oversold. For semantic and capability canaries, the generator rewrites both name and description, so a call could reflect surface attraction or novelty rather than the targeted weakness. The de-telling ablation softens the tell but cannot separate tell from lure, and the paper concedes exactly that. Because capability mirage is the one probe that fires on hosted models, this confound is load-bearing for the taxonomy's capability-stratified claim. I think the phenomenon is real—baseline CSR is zero, and the full-condition dilution argues against a generic more-tools effect—but the phrase \"shows the agent's selection was insensitive to the perturbed property\" in §3 is too strong for the LLM-rewritten types.\n\nTwo other issues. First, the CSR–TSR correlation (ρ=-0.34) is partly circular: a canary call usually causes task failure by design, so this validates the trap but not the diagnosis. The paper handles this better than most, but I would read it as a sanity check, not independent validation. Second, the abstract says \"near-zero frontier rates,\" but Table 1 shows GPT-5.2 at 0.178, which is not near-zero. That overstatement should be fixed. Also, the paper claims \"we release the framework\" but gives no artifact link; given the reproducibility emphasis, that is a concrete omission.\n\nAll told: the method is new, the evaluation is solid, and the limitations are mostly the usual empirical ones. The attribution ambiguity is real but not fatal; it needs a sharper experimental separation, not a redesign. I would send this to a serious referee, and I would cite it if I were working on agent tool-selection evals.","headline":"A genuinely useful new diagnostic for tool-selection failures, carefully evaluated, but the typed-attribution inference is weaker for the two LLM-rewritten canary types than the paper's wording suggests.","tokens_in":12779,"tokens_out":2021,"would_cite":true,"duration_ms":23744,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Canary tools turn a single 'wrong tool' outcome into a six-way diagnosis of which tool-selection reasoning step failed.","keywords":["canary tools","tool selection","LLM agents","MCP","diagnostic evaluation","capability mirage","tool-use reasoning","susceptibility rate"],"falsifier":"Run the eight models on the same 120 tasks with the canary pool reworded by a different LLM and with the real tools' descriptions reworded in the same register; if frontier CSR rises significantly above the reported 0.075, the de-telling result does not generalize and the probes are partially measuring wording rather than reasoning.","tokens_in":11690,"feed_emoji":"🛠️","tokens_out":7934,"duration_ms":78190,"temperature":0.7,"pith_summary":"Standard agent benchmarks score a wrong tool call as one bit: it failed, but not why. This paper introduces canary tools — probe tools planted in an agent's tool set, each engineered as a targeted perturbation of a real tool along a single axis — so that calling a canary reveals which specific reasoning weakness caused the wrong pick. On eight models and 8,640 task runs, susceptibility to these probes spans a 36-fold range, capability tier does not predict safety, and the six probe types fire at different points on the capability curve, with 'capability mirage' the only type that reliably traps strong models. Because softening the give-away phrases leaves frontier trap rates essentially unchanged, the authors argue the probes measure tool-selection reasoning, not phrase-spotting.","feed_headline":"Canary tools turn a wrong-tool pick into a typed diagnosis","feed_subtitle":"One binary failure becomes six reason codes; susceptibility predicts task failure across eight models.","key_machinery":"The load-bearing object is the canary tool: a targeted perturbation c = δ(t*) of a real tool t* along exactly one axis that leaves the tool superficially plausible but wrong for the task, with a deterministic failure semantics — calling c is unambiguous evidence of insensitivity to that property. The six-type taxonomy enumerates one canary per axis of the tool's contract: semantic decoy (paraphrased name, stale-data description), parameter trap (renamed params, unsatisfiable required api_key), capability mirage (inflated 'research-grade' claims), prerequisite blindness (omits required authentication), temporal decoy (outdated date/version), and granularity trap (over-specific hardcoded variant). Four types are generated by deterministic schema transforms; semantic and capability canaries use an LLM to reword names and descriptions. The framework also includes a provider-independent outcome judge and a trap detector that classifies each call as real, canary (with type), or unknown, producing per-task canary susceptibility rate (CSR) and type-specific trap rates.","core_discovery":"The central claim is that a canary — a deterministic perturbation of a real tool along exactly one dimension of its contract (name, description, parameter schema, or scope) — gives a wrong tool call unambiguous, typed failure semantics: observing the agent call the canary shows its selection was insensitive to the perturbed property. The paper builds a six-type taxonomy on this move (semantic decoy, parameter trap, capability mirage, prerequisite blindness, temporal decoy, granularity trap) and shows empirically that the taxonomy is capability-stratified: capability mirages catch mid-tier and weaker frontier models, while the other five probes are near-inert on strong hosted models but fire readily on small open-weight models. Two further findings are that nominal capability tier does not order canary resistance — the most susceptible hosted model is mid-tier and one mid-tier model beats its frontier sibling in the same provider — and that per-task canary susceptibility correlates negatively with task success (Spearman rho = -0.34). The de-telling ablation, which softens the give-away phrase of each probe, leaves frontier CSR essentially unchanged, which the authors take as evidence the probes measure reasoning rather than phrase-spotting.","pith_inferences":["The same perturbation logic could be extended beyond tool selection to other agent reasoning axes — planning, memory retrieval, or subgoal decomposition — where a binary failure could be given typed semantics.","The finding that raw canary density does not increase trap rates on hosted models (and often lowers them) suggests an attention-budget effect: with more competing tools, models read descriptions more carefully; this could be tested directly by varying the number of irrelevant tools while holding the canary constant.","The de-telling ablation does not fully separate a canary's tell from its lure, as the paper notes; a sharper test would reword the real tools in the same style as the softened canaries to rule out style-matching as the discriminator.","The provider-level pattern — the two models from one provider are the most resistant regardless of tier — hints that training-time emphasis on tool-description reading, rather than raw capability, drives canary resistance; a controlled fine-tuning experiment on a small open model could test this."],"forward_implications":["Layering a small panel of typed canaries onto an existing tool-use suite upgrades its binary pass/fail into a reason code, giving developers a low-cost pre-deployment readiness check.","High capability-mirage susceptibility points to specific hardening: guardrails on 'more-powerful' tool variants and description-hygiene rules that avoid superlative capability claims in tool descriptions.","Recovery is a separate, capability-aligned axis: a trapped run that recovers succeeds about half the time, versus about one in six if it stays on the canary, so an explicit verify-and-backtrack step can turn a trap into a near-miss.","Because capability tier does not predict safety, tool-selection safety should be probed directly on each deployment rather than read off general model tier.","Per-task CSR predicts task failure (rho = -0.34), so canary susceptibility is a diagnostic signal for real degradation, not a benchmark artifact."],"supporting_citations":[{"why":"Defines the Model Context Protocol, the tool-set standard into which canaries are planted.","marker":"Anthropic, 2024"},{"why":"AgentBench represents outcome-based agent benchmarks that score failure as one bit, the paradigm canaries diagnose.","marker":"Liu et al., 2024"},{"why":"The Berkeley Function-Calling Leaderboard checks function-call correctness via AST comparison, an outcome-only reference the canary approach extends.","marker":"Patil et al., 2025"},{"why":"MetaTool tests selection among similar tools and reliability issues, the closest distractor baseline that yields only a binary signal.","marker":"Huang et al., 2024"},{"why":"MCPAgentBench adds irrelevant or similar-domain tools to measure anti-interference, another binary-signal baseline.","marker":"Liu et al., 2025"},{"why":"MCP-Atlas provides a large-scale real-MCP-server tool-use benchmark, giving context for the evaluation environment.","marker":"Bandi et al., 2026"},{"why":"τ-bench is a stateful multi-turn tool-agent benchmark with outcome-based scoring, showing the gap the canary diagnosis fills.","marker":"Yao et al., 2024"}],"fun_headline_variants":["Canary tools diagnose why LLMs pick wrong tools","Six canary probes reveal tool-selection reasoning faults","Capability mirages catch frontier models, other traps don't","Canary susceptibility predicts task failure in LLM agents","De-telling canaries prove reasoning, not phrase-spotting"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The method assumes that a canary call is caused by the specific reasoning weakness the canary targets, and not by generic attraction to added tools, tool ordering, formatting, or accidental wording differences introduced when canaries are reworded.","fun_headline_variants_meta":{"raw":{"variants":["Canary tools diagnose why LLMs pick wrong tools","Six canary probes reveal tool-selection reasoning faults","Capability mirages catch frontier models, other traps don't","Canary susceptibility predicts task failure in LLM agents","De-telling canaries prove reasoning, not phrase-spotting"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000303,"raw_usage":{"total_tokens":1840,"prompt_tokens":1141,"completion_tokens":699,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":757,"completion_tokens_details":{"reasoning_tokens":618}},"tokens_in":757,"tokens_out":699,"duration_ms":7860,"temperature":1.0,"reasoning_tokens":618,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:05:35.675945+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the eight models on the same 120 tasks with the canary pool reworded by a different LLM and with the real tools' descriptions reworded in the same register; if frontier CSR rises significantly above the reported 0.075, the de-telling result does not generalize and the probes are partially measuring wording rather than reasoning.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the Model Context Protocol, the tool-set standard into which canaries are planted."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"τ-bench is a stateful multi-turn tool-agent benchmark with outcome-based scoring, showing the gap the canary diagnosis fills."}],"review_version":1}