{"id":"99dc0475-dbea-409c-8e6c-69db58d123b8","arxiv_id":"2607.11126","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.5,"correctness_risk":"low","formal_verification":"none","parameter_count":4,"one_line_summary":"Provider-side tool memory graphs, expanded by execution-verified frontier probing and queried by adaptive traversal, improve and transfer tool use across agents and environments.","lead":"ToolAtlas builds a reusable memory of tool capabilities, failure modes, and compositions on the tool-provider side rather than inside each agent. Shared MCP servers can then serve many heterogeneous agents without each one re-exploring the same tools.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged coverage/drift assumption.","rationale":"The paper is an empirical systems contribution whose strongest claim is supported by the reported multi-setting experiments, ablations, and released code/prompts. The single softest condition for that claim is that offline seed + fixed R=3 frontier probing produces coverage that stays useful for held-out tasks, new instances, and heterogeneous agents—an assumption already named by the reader and by §6. No stronger load-bearing flaw (e.g., leakage between train/test instances, agent-specific residue that would invalidate the agent-neutral claim, or missing controls that would reverse the ranking vs. ACE/Play2Prompt) appears in the tables or methodology. Therefore the CONDITIONAL verdict with high confidence is appropriate and needs no adjustment.","tokens_in":25513,"tokens_out":513,"duration_ms":8856,"concrete_test":"On one MCPMARK service (e.g., Filesystem), freeze the R=3 memory, then re-run the full same-environment and cross-environment test suites after (a) increasing R to 5 and (b) injecting a controlled API/schema drift (e.g., making list_directory_with_sizes non-recursive sizes recursive, or changing a PostgreSQL column type). If pass@1/pass@4 gains remain within ~3 points of the reported numbers under both (a) and (b), the frozen-memory claim is robust; a drop >10 points under (b) would confirm the drift concern is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a single frozen provider-side tool-memory graph, built once by seed tasks plus R=3 frontier-exploration rounds and retrieved by adaptive traversal, yields the reported pass@1/pass@4 gains and transfers across environment instances and agent frameworks. The manuscript supplies multi-benchmark tables (Tables 1–3, 8–9), ablations isolating G_cap and affordance/boundary probing (Table 4), a same-budget random-seed plateau comparison (Fig. 4), cross-agent and cross-instance transfer, and an honest Limitations section (§6) that already names the fixed exploration budget and the absence of long-term re-verification. No internal inconsistency, circular evaluation, or unsubstantiated leap is present; residual risk is precisely the coverage/validity assumption the reader already identified.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper argues that reusable tool knowledge for LLM agents should live with the tool provider rather than inside individual agents. It introduces ToolAtlas, a three-layer provider-side memory graph (Tool-Trace, Tool-Capability, Tool-Strategy) built offline by seed-task bootstrapping plus execution-verified frontier exploration of affordances, failure boundaries, and cross-tool compositions, then queried at inference by adaptive graph traversal. On MCPMark and MCP-Universe (eight services), the frozen memory improves pass@1/pass@4 by up to 21.61%/18.61% over tool-side optimization and agent-side memory baselines, and transfers across environment instances and agent frameworks (ReAct, CodeAct, SwitchAct) without retraining or task-time exploration. Ablations attribute gains to the capability graph and affordance/boundary probing; inference cost is competitive.","tokens_in":25787,"tokens_out":1290,"duration_ms":31993,"significance":"If the results hold, the work cleanly reframes tool memory ownership for shared MCP-style servers and supplies a concrete, reusable artifact rather than another agent-local skill library. Strengths include multi-benchmark evaluation across three backbones, same-environment / cross-environment / cross-agent protocols, programmatic verifiers, ablations that isolate G_cap and frontier probing (Table 4, Fig. 4), an honest Limitations section on drift and lifecycle cost, and released code. The transfer results are the most consequential part of the contribution: they support amortizing one offline construction across heterogeneous consumers. Residual risk is coverage and long-term validity of a frozen capability map, which the authors already flag rather than overclaim.","major_comments":[{"comment":"The title, abstract, and §5 frame the contribution as amortized “learning once, reusing everywhere,” yet §4.5 / Fig. 3 report only inference tokens. Offline construction cost (k seed tasks × N rollouts per tool, plus R frontier rounds of boundary/affordance probes, Memory Induction LLM calls, and verifier executions) is never quantified. Without that number—or a simple amortization break-even against agent-side re-exploration—the systems claim that provider-side memory is economically preferable remains incomplete. Please add construction-cost accounting (tokens, wall-clock, or API calls) and a short break-even discussion under the reported transfer settings.","section":"§4.5, §5, Appendix B"},{"comment":"pass@1 is defined as the average success rate over four independent runs (§A.1), but Tables 1–3, 8–9 report only point estimates with no standard errors, confidence intervals, or significance tests. Several per-service gains are small (e.g., Notion/PostgreSQL rows in Table 1) on modest test sizes (Table 5). Relative Impro(%) figures can therefore overstate precision. Please report run-level variance or bootstrap CIs for overall and key per-service metrics, and temper claims where intervals overlap the strongest baseline.","section":"Tables 1–3, 8–9; §4.1 Metrics"},{"comment":"The central transfer claim rests on the assumption that seed tasks plus fixed R=3 frontier rounds produce a capability map that remains useful for held-out tasks, new instances, and new agent action formats (§3.2, Appendix B). Fig. 4 shows continued gains at higher R, and §6 correctly notes the lack of re-verification under API drift. The manuscript still lacks any coverage diagnostic (e.g., fraction of tools with non-empty Ui/Bi/Ci, probe success/failure rates, or how often Dynamic Traversal returns empty guidance). A short coverage analysis would make the frozen-memory premise falsifiable rather than only empirically lucky on these splits.","section":"§3.2, Fig. 4, §6, Appendix B"}],"minor_comments":[{"comment":"Figure 1 and Figure 2 contain concatenated or garbled labels (“PerformancenomemwithA’smem”, “Eaxmples”, “ExploreCapabilies”). Redraw for readability.","section":"Fig. 1, Fig. 2"},{"comment":"Typographical issues: “V oyager” (space), “tool-side optimization mainly improves…” paragraph break mid-sentence near the end of §1, and inconsistent capitalization of MCPMARK / MCP-UNIVERSE.","section":"§1–§2"},{"comment":"Hyperparameters k=3, N=4, R=3, kr=3, B=8 are stated in Appendix B but not motivated beyond Fig. 4 for R. A one-sentence sensitivity note for k or B would help reproducibility.","section":"Appendix B"},{"comment":"Baseline re-implementation details (§A.2) are thorough; please also state whether tool-side methods (EasyTool, Play2Prompt, ToolOptimal) were allowed the same training-split interaction budget as ToolAtlas’s exploration, so cost-matched fairness is explicit.","section":"§A.2, §4.1"},{"comment":"Case study in Appendix F is useful; consider moving a shortened version into the main text or pointing to it earlier in §3 so readers see what agent-neutral traces and guidance actually look like.","section":"Appendix F, §3"}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical systems paper with a clear conceptual shift (provider-side vs agent-side memory) and unusually careful transfer protocols for this area. The three major points are fixable with additional reporting rather than new method design; I would not block on them if the authors add cost, variance, and a coverage diagnostic. Fit is good for a methods/systems venue that values reusable infrastructure for tool-using agents. No integrity or circularity concerns."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful move here is ownership: put reusable tool knowledge on the MCP server, not inside each agent. ToolAtlas builds a three-layer graph (agent-neutral traces, per-tool affordances/boundaries/co-usage, and higher-level strategies), expands it with execution-verified frontier probes (outward on failure regimes, inward on missing uses and compositions), and serves it with adaptive traversal instead of static top-k. That is a real shift relative to ExpeL/A-MEM/ACE-style agent memory and EasyTool/Play2Prompt-style description polishing.\n\nWhat they do well is the evaluation design. Same-environment, cross-instance, and cross-agent (SwitchAct → ReAct/CodeAct) splits, two MCP benchmarks, eight services, three backbones, programmatic verifiers, frozen memory after train, and ablations that hit G_cap and the affordance/boundary split hardest. Transfer gains (roughly +24%/+16% cross-env, +17%/+14% cross-agent) are the part that matters for the thesis; agent-side baselines often fall below vanilla after transfer, which is exactly the failure mode they diagnose. Code and prompts are released; limitations section is honest about fixed R=3 and missing long-term re-verification.\n\nSoft spots are real but proportional. Coverage still rests on seed tasks plus a hand-set exploration budget; if APIs drift or the agent-neutral abstraction drops critical context, the frozen atlas ages. Only eight services, no production-scale catalog or continuous refresh study. Those are deployment and longevity questions, not holes in the reported experiments. No circular scoring, no invented math.\n\nThis is for people building shared tool servers or multi-agent MCP stacks who care about amortizing exploration. I would bring it to reading group, cite it when writing on tool memory or MCP, and send it to referees. Solid systems paper with a clear idea and evidence that matches the claim.","headline":"Clean conceptual flip—tool memory lives with the provider—and the multi-setting numbers actually back the transfer claim.","tokens_in":26389,"tokens_out":490,"would_cite":true,"duration_ms":9776,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Reusable tool knowledge belongs with the tool provider, not inside each agent.","keywords":["tool-using agents","provider-side memory","tool memory graph","capability exploration","MCP servers","adaptive graph traversal","cross-agent transfer"],"falsifier":"Freeze a ToolAtlas memory on one set of environment instances and agent style; then evaluate on held-out instances and a different agent framework after the API or backend schema has changed without any re-probing. If pass rates fall to or below the no-memory baseline, the claim that the frozen provider-side map remains reusable fails.","tokens_in":26415,"feed_emoji":"🧰","tokens_out":567,"duration_ms":6934,"temperature":0.7,"pith_summary":"When many different agents call the same shared tool server, each agent currently rediscovers the same capabilities, failure modes, and multi-tool patterns on its own. This paper argues that the waste is structural: agent-side memory cannot be shared cleanly, and it only records the narrow slice of tool behavior that past tasks happened to exercise. ToolAtlas instead builds a persistent memory graph on the provider side. The graph records what each tool can do, where it fails, and how it composes with others, expanded by deliberate execution-verified probes rather than passive logs. At inference time a light navigator walks the graph and returns task-specific guidance. The same memory improves success rates over strong baselines and transfers to new environment instances and different agent frameworks without retraining or extra exploration.","feed_headline":"Tool memory lives with the server, not each agent","feed_subtitle":"One offline graph of capabilities and failures lifts success and transfers across agents","key_machinery":"ToolAtlas: a three-layer tool memory graph (Tool-Trace, Tool-Capability, Tool-Strategy) expanded by frontier exploration that probes untested boundaries and under-recorded affordances, then read at inference by adaptive multi-layer graph traversal that compresses the walk into task-conditioned guidance.","core_discovery":"A single provider-side tool-memory graph, built once through execution-verified probing of affordances, boundaries, and compositions and retrieved by adaptive graph traversal, measurably improves tool-using agents and transfers across environment instances and agent frameworks without retraining or task-time rediscovery.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["Tool memory lives on the server, not each agent","One provider-side graph of tools reused across agents","Build tool capabilities once, reuse without retraining","Shared server memory of tool limits and compositions","Agents query a persistent provider tool-memory graph"],"cache_read_input_tokens":128,"weakest_assumption_plain":"That a fixed offline set of seed tasks and a few rounds of frontier probes produce a capability map that stays accurate and useful when the environment instance, agent action format, or underlying API later changes.","fun_headline_variants_meta":{"raw":{"variants":["Tool memory lives on the server, not each agent","One provider-side graph of tools reused across agents","Build tool capabilities once, reuse without retraining","Shared server memory of tool limits and compositions","Agents query a persistent provider tool-memory graph"]},"model":"grok-4.5","effort":"low","cost_usd":0.002762,"raw_usage":{"total_tokens":1047,"prompt_tokens":784,"num_sources_used":0,"completion_tokens":55,"cost_in_usd_ticks":27620000,"prompt_tokens_details":{"text_tokens":784,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":208,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":784,"tokens_out":55,"duration_ms":2587,"temperature":1.0,"reasoning_tokens":208,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-14T06:52:26.175706+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Freeze a ToolAtlas memory on one set of environment instances and agent style; then evaluate on held-out instances and a different agent framework after the API or backend schema has changed without any re-probing. If pass rates fall to or below the no-memory baseline, the claim that the frozen provider-side map remains reusable fails.","supporting_citations":[],"review_version":1}