{"id":"31878bff-9bae-44cf-8274-a0045612924f","arxiv_id":"2608.08467","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Client LLMs systematically prefer calling a search tool over using equivalent data embedded in server instructions, even though they can use the embedded data when the tool is removed.","lead":"This paper measures whether 24 large language models actually use reference data that an MCP server embeds in its instructions, or instead call a search tool to find the same information. It finds most models ignore the embedded data when a search tool is available, and that this is a behavioral preference, not a capability gap.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The behavioral preference finding is solid, but the title's claim of 'inefficient resource utilization' rests on an unvalidated proxy: first-call hit ratio is never checked against actual token or latency cost.","rationale":"The reader's weakest assumption identified the first-call metric as the key risk, and I agree that it is the most load-bearing point. The behavioral part of the claim is well controlled: no_s shows 23/24 models can use the embedded data, and baseline shows many choose search when it exists. The intervention grid and the disclosed limitations support that finding. However, the paper's title and abstract announce 'inefficient resource utilization,' and the only evidence for inefficiency is the first tool call being search. A search-then-fetch sequence is logically more steps than direct fetch, so some inefficiency is likely, but the magnitude and practical significance are not quantified. The paper explicitly says it does not score the final answer and does not report tokens or latency, so a reader cannot tell whether the 0-100% spread corresponds to a meaningful cost difference or a small overhead that users would not notice. Because the authors themselves present the metric as a proxy rather than a measurement, the correct verdict is conditional: accept the behavioral result now, but require direct cost validation or a more narrowly worded title before treating 'inefficient resource utilization' as established. The five-query design is a secondary concern; a single concrete cost measurement would settle the primary issue.","tokens_in":6123,"tokens_out":11519,"duration_ms":118395,"concrete_test":"Re-run the full 24-model grid on the same LexLink server (same harness, same 54,000-trial protocol) while logging per-trial prompt/output tokens and wall-clock latency for every trial that ends in a successful eflaw_service fetch, comparing baseline vs BCD vs no_s. If the median total-token cost of search-first successful trials exceeds the direct-first successful trials by less than 5%, the inefficiency headline is overstated and the paper should reframe to first-call behavior; if the overhead is substantial (or if many search-first trials fail to fetch), the concern is resolved in the paper's favor.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim has two parts: models prefer eflaw_search over the embedded table (strongly supported), and this constitutes inefficient resource utilization (inferred but not measured). Section 2 defines the hit ratio phi as the fraction of trials whose first tool call is eflaw_service with the correct cached ID; Section 4 concedes that the metric captures first-call efficiency rather than end-to-end task success, and the paper never reports token counts, latency, or total cost. A model that first calls eflaw_search and then eflaw_service may complete the lookup correctly; the extra search call is an overhead, but its magnitude is unknown. If the search tool is cheap relative to the LLM's own generation, the 0-100% baseline spread could overstate the resource-utilization problem. Additionally, all 54,000 trials use only five fixed queries; Section 3 shows per-query decompositions only for two models, so the stability of the '9 models below 15%' result across query phrasing is not established. The authors disclose both limits, but the title and abstract do not condition the inefficiency claim on them.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript reports a controlled measurement of whether client LLMs use reference data embedded in MCP server instructions. On a production legal-information server (LexLink), the paper defines a hit ratio phi as the fraction of trials in which the model's first tool call is eflaw_service with the correct cached ID. With a search tool present, baseline hit ratios span 0-100%, with 9 of 24 models below 15%; in a no_s diagnostic condition without the search tool, 23 of 24 models use the embedded data at 98% or higher. A 2^3 factorial over three instruction-level interventions shows strong interactions, with the combined condition restoring at least 86% for 20 of 24 models, while individual interventions can backfire for specific model families. The authors argue that the failure is a behavioral preference rather than a capability deficit and propose host-side mechanisms to place server instructions ahead of tool selection.","tokens_in":6263,"tokens_out":9813,"duration_ms":99668,"significance":"The empirical result is valuable and clearly presented. The no_s control provides a sharp separation between capability and tool-availability-driven behavior; the factorial design reveals non-monotonic, family-specific effects; and the 54,000-trial design, explicit metric, binomial confidence bounds, and released code/data support reproducibility. The main limitation is that the practical magnitude of the inefficiency is inferred from first-call choices rather than measured directly, so the resource-utilization claim is not as strongly supported as the behavioral-preference claim.","major_comments":[{"comment":"The paper's title and abstract claim that the study measures 'inefficient resource utilization,' but the only operationalized quantity is the hit ratio phi, defined in §2 as the fraction of trials whose first tool call is eflaw_service with the correct cached ID. No token counts, latencies, or dollar costs are reported anywhere in the manuscript. As §4 concedes, a model that first calls eflaw_search and then eflaw_service can still complete the lookup correctly; the extra call is an overhead, but its magnitude is never quantified. Because §2 justifies the embedded table by saying it saves 'both tokens and latency,' this central resource-utilization claim currently rests on an unvalidated proxy. The authors should either provide a direct token/latency comparison for at least a subset of models or revise the title, abstract, and conclusion to describe the contribution as measuring avoidable tool-call overhead or first-call efficiency rather than resource utilization generally.","section":"§2, §4 (and title/abstract)"}],"minor_comments":[{"comment":"Table 1 is difficult to parse because rows such as 'cl-3-haiku 2024-03 41100 10088100 100 100 100 100' lack visible column separators; a formatted table with proper delimiters is needed.","section":"Table 1"},{"comment":"Footnote 4 contains an important control result (no model exceeds 10% correct cached IDs when the embedded table is removed) that supports the parametric-memory control; this evidence should be moved to the main text or an appendix.","section":"Footnote 4"},{"comment":"The abstract's '9 models drop below 15%' is computed from five fixed queries, and per-query decompositions are shown only for cl-opus-4.6 and gpt-4.1-nano; the abstract and conclusion should condition this figure on the tested query set.","section":"Abstract and §3"},{"comment":"The one-sample t-tests over model-level effects are computed while models cluster by provider; the p-values should be described as a descriptive screening device rather than population inference for LLMs in general.","section":"§3"}],"recommendation":"major_revision","confidential_remarks":"The empirical core is solid: the no_s control, the factorial design, the 54,000-trial count, and the released artifacts make the behavioral-preference finding credible. My main concern is that the title and abstract overstate what is measured; if the authors add a modest token/latency analysis or clearly label phi as a proxy for resource efficiency, I would be happy to support acceptance. The self-citation to the first author's LexLink server is not an issue because the dependent variable is the client LLM's behavior, which is independent of the server author."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe thing worth knowing: the main empirical claim is solid. With the search tool removed, 23 of 24 models read the embedded table at 98% or higher; with the tool present, 9 models drop below 15%. That is a behavior effect, not a capability gap. The no_s diagnostic is a clean design and worth copying.\n\nThe paper also does two things well beyond the headline. First, the 2^3 factorial across 24 models is a proper way to show instruction-level fixes are not additive: B rescues GPT-5-nano by +80.9pp but drops GPT-5.2 to 0. Second, it publishes the harness and per-trial data for 54,000 trials, with an explicit hit-ratio definition and binomial intervals. The disclosures in Section 4 are honest about scope.\n\nThe soft spots are real but mostly at the edges. The biggest is the title: 'inefficient resource utilization' is inferred from first-call hit ratio, not measured. A client that calls eflaw_search first and then eflaw_service still completes the lookup; the extra call costs tokens and latency, but the paper never reports those costs. It is plausible the overhead is large, but the evidence stops short of quantifying it. The authors acknowledge the first-call metric and single-turn design, so this is more a framing problem than a hidden flaw. I would soften the claim and, if easy, add token-count data on a subset.\n\nOther limits: one server and domain, five fixed queries, one wording per intervention, and per-query decompositions for only two models. These are disclosed and consistent with the paper's own wording, but they mean the precise percentages should be read as testbed-specific. The clustering of trials within five queries also makes the binomial CIs optimistic, though the effect sizes are so large that this does not change the conclusion.\n\nThe self-citation to LexLink is not circular: the result concerns client LLM behavior, not the server's construction.\n\nWho is this for? Anyone building MCP hosts or servers, and anyone evaluating agent tool-use. It deserves a serious referee. I would accept with minor revision, mostly asking for the cost measurement or a reworded title, plus a sentence about query clustering.","headline":"The behavioral-preference finding is real and cleanly demonstrated, but the 'inefficient resource utilization' title reaches slightly beyond what the first-call proxy actually measures.","tokens_in":6851,"tokens_out":2322,"would_cite":true,"duration_ms":24246,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Client LLMs systematically prefer calling a search tool over using equivalent data already embedded in MCP server instructions, and 54,000 trials show this is a behavioral preference rather than a capability deficit.","keywords":["Model Context Protocol","large language models","tool-augmented agents","instruction following","tool selection bias","server instructions","resource utilization","hit ratio"],"falsifier":"Run the same 54,000-trial grid while logging tokens and latency per trial; if search-first models match direct retrievers on end-to-end cost and correctness, the inefficiency claim is falsified.","tokens_in":5886,"feed_emoji":"🔍","tokens_out":5844,"duration_ms":55374,"temperature":0.7,"pith_summary":"The paper sets out to measure whether client LLMs actually use reference data that MCP servers embed in their instructions, and finds that most do not when a search tool is available. On a production legal-information server, 54,000 trials across 24 models show that merely adding a search tool drops the rate of direct cached-ID lookups below 15% for 9 models, even though removing the tool lets 23 of 24 models read the same embedded data at 98% or higher. The authors conclude the failure is a behavioral preference for tool calls rather than a capability deficit, and that per-server prompt engineering is a brittle workaround. The stakes are practical: MCP hosts, not individual servers, should control whether instructions or tool schemas dominate a client model's deliberation.","feed_headline":"Mere presence of a search tool makes 9 LLMs ignore built-in data","feed_subtitle":"Removing the search tool lets 23 of 24 models read instruction-embedded data at 98% or better.","key_machinery":"The load-bearing measure is the hit ratio $\\phi(m,c)$: the fraction of trials, for model $m$ under condition $c$, in which the first tool call is eflaw_service with the exact cached law ID. The experimental machinery is a full $2^3$ factorial over three instruction-level interventions — a directive (B), a few-shot example (C), and a tool-description hint (D) — plus a no_s diagnostic condition that removes the competing search tool entirely. The no_s condition does the causal work: it shows that failure at baseline is not a parsing or comprehension limitation, since nearly every model can use the embedded data once the alternative tool disappears. The factorial design then shows which combinations of instructions restore direct retrieval and which single factors backfire.","core_discovery":"The central claim is that the presence of a search tool overrides instruction-embedded data in client LLMs, producing inefficient resource utilization. Concretely, when the search tool is available, the hit ratio of first-call direct cached-ID retrieval ranges from 0% to 100% across 24 models, with 9 models below 15%; when the search tool is removed, 23 of 24 models retrieve the embedded ID directly in at least 98% of trials. The one exception, a low-capability model at 54% in the no-search condition, shows that a genuine capability deficit can coexist with the dominant preference effect. A $2^3$ factorial of instruction interventions shows that combining a directive, a few-shot example, and a tool-description hint restores at least 86% for 20 of 24 models, while single interventions sometimes backfire for specific families. The authors argue from these results that MCP host applications should provide an explicit mechanism to place server instructions ahead of tool selection.","pith_inferences":["If the behavioral preference is shaped by post-training that rewards successful tool calls, then agentic-benchmark training will keep producing this pattern unless instruction following is explicitly weighted; the authors raise this mechanism as speculation, and it is testable only with training data that providers do not publish.","A direct token- and latency-per-trial measurement would convert the hit-ratio gap into cost terms and could reveal whether models that first call search still finish within a comparable budget; if they do, the inefficiency claim would be overstated for cost but still true for instruction adherence.","A natural extension is to repeat the no_s diagnostic on servers that embed non-ID reference data, such as database schemas or routing tables, to test whether the search-tool override is specific to identifier lookup or is a general property of tool presence.","Because no_s removes the tool and thereby shrinks the action space, a sharper preference probe would add a 'useless' search tool that returns the same data as the embedded table and check whether the presence of a callable tool alone still triggers the bypass."],"forward_implications":["Effective capability of an MCP server cannot be reported without naming the client LLM, because the same server ranges from 0% to 100% efficiency depending on which model consumes it.","Per-server prompt engineering is brittle: a directive that rescues one model can break another, so hosts should offer a system-level consult-instructions-first mechanism rather than relying on server-authored prompts.","For lookup-style workflows, adding a redundant search tool can actively reduce efficiency; tool availability itself is sufficient to override instruction-embedded data.","Server developers can act today by combining directive, few-shot example, and tool-description hint for broadest coverage, and by validating against the weakest client models they expect to serve."],"supporting_citations":[{"why":"Defines the MCP specification with Resources, Tools, and Server instructions, grounding the study's three-channel framing.","marker":"[1]"},{"why":"Documents tool selection bias in LLMs, which the paper's behavioral-preference interpretation extends.","marker":"[2]"},{"why":"Supplies the production legal-information MCP server and embedded law-ID table used for all 54,000 trials.","marker":"[3]"},{"why":"Provides prior evidence that tool preferences in agentic LLMs are unreliable, motivating the preference diagnosis.","marker":"[5]"},{"why":"Shows instruction hierarchies fail under conflict, supporting the claim that instruction-embedded data can be overridden.","marker":"[6]"},{"why":"Reports low single-turn compliance with MCP server instructions, the prior result this study quantifies on its own testbed.","marker":"[10]"}],"fun_headline_variants":["Search tool presence blinds 9 of 24 LLMs to embedded data","LLMs ignore built-in data when search tool is present","9 of 24 LLMs rely on search tool over embedded data","Search tool presence overrides embedded data for 9 LLMs","Tool presence makes LLMs ignore embedded data: 9 fail"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the first tool call being eflaw_service with the exact cached ID is a valid measure of efficient resource utilization; the study does not directly measure token counts, latency, or end-to-end answer correctness, so if search-first trials still yield acceptable results at comparable cost, the inefficiency claim is weakened.","fun_headline_variants_meta":{"raw":{"variants":["Search tool presence blinds 9 of 24 LLMs to embedded data","LLMs ignore built-in data when search tool is present","9 of 24 LLMs rely on search tool over embedded data","Search tool presence overrides embedded data for 9 LLMs","Tool presence makes LLMs ignore embedded data: 9 fail"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":3063,"prompt_tokens":1018,"completion_tokens":2045,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":1957}},"tokens_in":634,"tokens_out":2045,"duration_ms":14891,"temperature":1.0,"reasoning_tokens":1957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:34:17.644285+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 54,000-trial grid while logging tokens and latency per trial; if search-first models match direct retrievers on end-to-end cost and correctness, the inefficiency claim is falsified.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the MCP specification with Resources, Tools, and Server instructions, grounding the study's three-channel framing."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the production legal-information MCP server and embedded law-ID table used for all 54,000 trials."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides prior evidence that tool preferences in agentic LLMs are unreliable, motivating the preference diagnosis."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Shows instruction hierarchies fail under conflict, supporting the claim that instruction-embedded data can be overridden."}],"review_version":1}