{"id":"1465443b-0758-422c-ac37-d227deb96fdd","arxiv_id":"2606.12451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ToolSense generates three diagnostic benchmarks revealing a knowledge-retrieval dissociation in parametric tool models on ToolBench, with large performance drops on realistic queries and near-random factual scores.","lead":"ToolSense is an open-source framework that auto-generates realistic, ambiguous queries plus factual probes to test whether LLMs truly understand tools or merely pattern-match. Smart readers should care because current tool-retrieval benchmarks may overestimate reliability for real agent deployments over large catalogs.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"RRB collapse vs ToolBench may conflate ambiguity with removal of constrained decoding","rationale":"Reader's weakest assumption targets potential generation artifacts in the LLM-powered benchmarks. The more immediate load-bearing issue is the mismatched evaluation protocol (constrained vs unconstrained) that undermines direct numerical comparison between RRB and ToolBench, independent of benchmark validity.","tokens_in":1850,"tokens_out":253,"duration_ms":25931,"concrete_test":"Re-run the five parametric configurations on the original fully-specified ToolBench queries using unconstrained generation (no constrained decoding); if accuracy drops by >30 pp relative to the constrained ToolBench numbers, the RRB-specific collapse is not uniquely diagnostic of the claimed dissociation.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"Abstract notes ToolBench benchmarks use constrained decoding restricting outputs to valid token paths, while RRB uses open queries at three ambiguity tiers. The claimed 50-64 pp collapse (and falling below embedding baseline) compares RRB numbers directly to those ToolBench figures. If RRB evaluation omits constrained decoding, the drop cannot be attributed to knowledge-retrieval dissociation or query realism; it could be driven primarily by the change in decoding constraint allowing invalid paths.","agreement_with_reader":"disagree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces ToolSense, an LLM-powered framework that ingests a tool catalog and automatically generates three diagnostic benchmarks: a Realistic Retrieval Benchmark (RRB) at three ambiguity tiers, an MCQ probing benchmark, and a QA probing benchmark. When applied to the ToolBench catalog of ~47k tools, evaluation of five parametric model training configurations shows large performance collapses (50-64 pp) on RRB relative to standard ToolBench retrieval numbers, with some models falling below an embedding baseline, plus near-random scores on factual probes despite strong retrieval, indicating a knowledge-retrieval dissociation.","tokens_in":1937,"tokens_out":499,"duration_ms":25945,"significance":"If the dissociation is shown to be robust, the result would be significant for LLM agent research: it would demonstrate that strong performance on fully-specified, constrained-decoding benchmarks does not imply usable tool knowledge under realistic conditions, motivating changes in how parametric retrievers are trained and evaluated. The open-sourcing of both the framework and the generated ToolBench diagnostics is a clear strength that enables follow-up work.","major_comments":[{"comment":"Abstract (performance comparison paragraph): the claimed 50-64 pp collapse on RRB is obtained by comparing open-generation RRB queries directly against ToolBench numbers that were produced under constrained decoding restricting outputs to valid token paths. Because the decoding regime differs, the drop cannot be unambiguously attributed to query realism or lack of parametric tool knowledge; an ablation that re-evaluates the same models on ToolBench-style queries without constrained decoding is required to isolate the effect.","section":"Abstract"},{"comment":"Abstract (benchmark generation description): the LLM-powered construction of the three benchmarks (RRB at three ambiguity tiers, MCQ, and QA probes) is presented without reported validation (human review of query naturalness, inter-rater reliability, or controls for generation artifacts that might systematically disadvantage parametric models relative to the embedding baseline). This leaves open whether the observed dissociation reflects genuine tool understanding deficits or artifacts of the synthetic test construction.","section":"Abstract"}],"minor_comments":[{"comment":"The abstract states that five parametric configurations were evaluated but does not list their training details or hyper-parameters; a short table or explicit enumeration in the main text would improve reproducibility.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments. We address each major comment below and will revise the manuscript accordingly.","responses":[{"response":"We agree that the reported performance collapse mixes open-generation evaluation on RRB with constrained-decoding results on ToolBench, preventing unambiguous attribution to query realism. In the revised manuscript we will add an ablation that re-evaluates all five parametric configurations on the original ToolBench queries under open generation (no constrained decoding) and report the resulting scores alongside the existing numbers.","revision_made":"yes","referee_comment":"[Abstract] Abstract (performance comparison paragraph): the claimed 50-64 pp collapse on RRB is obtained by comparing open-generation RRB queries directly against ToolBench numbers that were produced under constrained decoding restricting outputs to valid token paths. Because the decoding regime differs, the drop cannot be unambiguously attributed to query realism or lack of parametric tool knowledge; an ablation that re-evaluates the same models on ToolBench-style queries without constrained decoding is required to isolate the effect."},{"response":"We acknowledge that the manuscript does not report human validation of the generated benchmarks. In the revision we will add a validation subsection that describes human review of a stratified sample of RRB queries and probes, reports naturalness ratings, inter-annotator agreement, and explicit checks for generation artifacts that could favor or disfavor parametric models relative to the embedding baseline.","revision_made":"yes","referee_comment":"[Abstract] Abstract (benchmark generation description): the LLM-powered construction of the three benchmarks (RRB at three ambiguity tiers, MCQ, and QA probes) is presented without reported validation (human review of query naturalness, inter-rater reliability, or controls for generation artifacts that might systematically disadvantage parametric models relative to the embedding baseline). This leaves open whether the observed dissociation reflects genuine tool understanding deficits or artifacts of the synthetic test construction."}],"tokens_in":1475,"tokens_out":414,"duration_ms":34863,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main thing to know is that this paper builds an open-source framework that turns a tool catalog into three new test sets—realistic queries at three ambiguity levels plus MCQ and QA factual probes—and shows parametric tool-retrieval models drop 50-64 points on the realistic set while some score near chance on facts despite strong standard-benchmark numbers.\n\nWhat is actually new is the automatic generation pipeline that produces tiered realistic queries and separate factual probes without manual annotation, plus the concrete demonstration that standard ToolBench numbers do not track performance under more open conditions. The scale (47k tools) and the decision to release both code and the generated benchmarks are clear positives; that makes the work immediately usable by others.\n\nThe soft spot is the comparison itself. ToolBench results rely on constrained decoding that forces valid tool paths, while the new RRB uses open queries. The abstract does not say whether they re-ran the parametric models under constrained decoding on the RRB or controlled for invalid outputs, so the size of the collapse cannot yet be cleanly attributed to query ambiguity or missing factual knowledge rather than the decoding change. Benchmark validation details are also thin in the abstract—no mention of human checks on generated query quality or artifact controls—so it is unclear how much the dissociation depends on generation artifacts that might hit parametric and embedding models differently.\n\nThis is the kind of paper that belongs in the tool-use and agent evaluation corner of the field. Readers who build or benchmark retrieval-augmented agents will get immediate value from the released artifacts even if they treat the exact drop numbers as provisional. The work shows clear thinking about the gap between current benchmarks and realistic use, so it deserves a serious referee who can press on the decoding control and the validation of the generated tests.","headline":"ToolSense gives a practical way to generate ambiguity-tiered retrieval tests and factual probes from any tool catalog, and the reported dissociation is worth checking, but the big drops versus ToolBench likely mix query realism with the switch from constrained to open decoding.","tokens_in":2454,"tokens_out":451,"would_cite":false,"duration_ms":23030,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Parametric LLMs trained for tool retrieval collapse on realistic ambiguous queries and score near-random on factual probes.","keywords":["tool retrieval","parametric models","LLM agents","benchmark generation","knowledge dissociation","realistic queries","factual probing","ToolBench"],"falsifier":"Human-authored realistic queries at matching ambiguity levels and human-authored factual probes that show the same performance gaps between parametric configurations and embedding baselines would support the dissociation; the absence of those gaps would undermine it.","tokens_in":2741,"feed_emoji":"🛠","tokens_out":680,"duration_ms":18790,"temperature":0.7,"pith_summary":"The paper presents ToolSense as a diagnostic framework that ingests any tool catalog and automatically produces three new benchmarks: a Realistic Retrieval Benchmark with queries at three ambiguity levels, plus MCQ and QA probes for factual tool knowledge. When applied to a large tool set, five different parametric training setups that had looked strong on standard benchmarks show drops of 50-64 percentage points on the new realistic queries, falling below embedding-model performance. Several models also perform near chance on the factual probes even when retrieval looks solid. A sympathetic reader would care because this points to a gap between apparent retrieval skill and actual tool understanding in agent systems that must handle imperfect user requests.","feed_headline":"Parametric tool retrievers drop 50 points on realistic queries","feed_subtitle":"ToolSense benchmarks expose collapse below embedding baselines and near-random fact scores despite strong standard results.","key_machinery":"ToolSense, the LLM-powered framework that takes a tool catalog as input and generates the Realistic Retrieval Benchmark at three ambiguity tiers together with MCQ and QA probing benchmarks.","core_discovery":"Applying ToolSense to ToolBench reveals a knowledge-retrieval dissociation: parametric model configurations that perform well on fully-specified ToolBench benchmarks with constrained decoding drop sharply on RRB queries at three ambiguity tiers and score near-random on the generated factual probes, indicating that strong retrieval does not imply genuine tool knowledge.","pith_inferences":["Agent builders may need to combine parametric retrieval with embedding methods or add runtime fact-checking to handle ambiguous inputs.","Training objectives could be extended to include explicit factual consistency losses so retrieval and knowledge stay aligned.","The three-tier ambiguity structure offers a way to measure how much specification a model actually requires before its performance falls apart.","Repeated application of the framework across catalogs could reveal whether the dissociation is architecture-specific or training-stage-specific."],"forward_implications":["Standard ToolBench-style benchmarks with verbose queries and constrained decoding overestimate the reliability of parametric tool retrieval.","Some parametric training configurations produce models whose retrieval succeeds without corresponding factual tool knowledge.","Embedding-based retrieval can outperform certain parametric setups once queries become realistic rather than fully specified.","Any new parametric training run should be audited with ambiguity-tiered retrieval tests and factual probes before deployment.","The same dissociation pattern can be checked on other tool catalogs by running the generated benchmarks."],"fun_headline_variants":["Parametric retrievers fail on ambiguous queries","ToolSense shows knowledge-retrieval split in LLMs","Strong ToolBench scores hide fact recall issues","Parametric models lose tool facts on real queries","Diagnostic uncovers near-random tool knowledge scores"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The LLM-generated benchmarks at different ambiguity tiers and the probing sets measure genuine tool understanding and factual knowledge without systematic artifacts that affect parametric models differently from embedding models.","fun_headline_variants_meta":{"raw":{"variants":["Parametric retrievers fail on ambiguous queries","ToolSense shows knowledge-retrieval split in LLMs","Strong ToolBench scores hide fact recall issues","Parametric models lose tool facts on real queries","Diagnostic uncovers near-random tool knowledge scores"]},"model":"grok-4.3","cost_usd":0.005396,"raw_usage":{"total_tokens":2609,"prompt_tokens":686,"num_sources_used":0,"completion_tokens":60,"cost_in_usd_ticks":53962000,"prompt_tokens_details":{"text_tokens":686,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1863,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":686,"tokens_out":60,"duration_ms":15684,"temperature":1.0,"reasoning_tokens":1863,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-28T00:54:34.728702+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Human-authored realistic queries at matching ambiguity levels and human-authored factual probes that show the same performance gaps between parametric configurations and embedding baselines would support the dissociation; the absence of those gaps would undermine it.","supporting_citations":[],"review_version":1}