{"id":"b679386e-e982-4c87-914e-cab48cc956e7","arxiv_id":"2608.10037","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"DocsChisel adaptively adds, removes, and refines information fields in tool documentation based on failed execution traces, improving LLM agent task success rate by 95.89% over original documentation and 75.15% over prior methods.","lead":"This paper studies how the information fields in tool documentation affect LLM agents' ability to complete tasks, and introduces DocsChisel, a framework that automatically edits those fields to improve agent performance. The authors report large gains in task success rate over original documentation and over two prior optimization methods.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test set independence is not established: RQ5 selects the optimization model and iteration budget on the test set, and per-tool multi-query assignment may let test queries enter optimization sets, so the headline gains may be inflated.","rationale":"The reader's weakest assumption correctly identifies the load-bearing point: the main result depends on the test set being fully independent of all optimization decisions. The paper's own text in Section IV-D shows that the test set was used to select the optimization model and iteration budget, and Section III-B plus Section IV-A leave unclear whether multi-tool queries can straddle the global train/test split. These are not abstract worries; they directly affect the credibility of the 95.89% and 75.15% figures. At the same time, the paper has real strengths: a detailed framework, 3x5 repeated runs, Mann-Whitney U tests with Holm correction, and comparisons across domains, backbones, and agent paradigms. Those support the qualitative direction of the claim even if the precise magnitudes are not yet cleanly established. The reader's CONDITIONAL verdict remains appropriate because the issues are fixable through a clearer split audit and a selection-protocol rerun, not because the framework is fundamentally unsound. I therefore keep the verdict unchanged rather than moving to reject or accept.","tokens_in":18802,"tokens_out":4933,"duration_ms":53096,"concrete_test":"Run a pre-registered audit on the released repository: for every one of the 2,072 queries, record the set of tools it invokes, verify the global optimization/validation/test split, and assert that no query in the 829-query test set appears in any tool's Qopt_c,t or Qval_c,t for any of the 74 tools. Then re-run RQ3 with the RQ5 configuration chosen on a separate model-selection set (or on the validation set only), never on the test set, using the same 3 optimization runs and 5 evaluation runs. If the mean TS improvement over original documentation remains close to 95.89% and the mean TS improvement over the strongest baseline remains close to 46.11%, the concern is resolved. If either drops materially (for example, by more than 20% relative), the headline numbers are inflated by selection or leakage.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that DocsChisel improves task success rate by 95.89% over original documentation and 75.15% over baselines. These numbers are trustworthy only if the test set was never used for any optimization decision. The paper does not establish this. In Section IV-D (RQ5), the authors vary the optimization model and the iteration budget and report TC and TS 'on the test set using the optimized documentation generated under each setting.' They then use exactly those test-set results to justify defaults: Table IV shows Claude Haiku 4.5 performs best, and Fig. 9 is used to 'set the default iteration budget to five.' The main RQ3 results are then reported with that same configuration on the same test set. This is a model/hyperparameter selection step performed on the test set, which breaks the independence the abstract and Section IV-A claim ('the test set is held out throughout the optimization process'). Baselines are not given the same test-based selection opportunity, so the comparison is systematically favorable to DocsChisel. A second, independent ambiguity is in Section III-B: each tool is associated with the queries whose executions involve it, and 'a query involving multiple tools belongs to multiple tool-specific query sets.' Section IV-A describes a global 5:1:4 split, but it is not stated whether a query assigned to the global test set can still appear in the tool-specific optimization set of another tool. If it can, then a test query used for final evaluation may have directly contributed to optimizing a tool's documentation, which is direct leakage. Neither the RQ5 selection issue nor the query-overlap ambiguity necessarily falsifies the approach, but they do mean the reported magnitudes are not yet backed by a clean held-out evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an empirical study of tool documentation for LLM agents, identifying 17 information fields and showing that their effects on task success vary across task domains, LLM backbones, and agent paradigms. Based on this motivation, the authors propose DocsChisel, a framework that iteratively optimizes tool documentation at the information-field level using failed execution traces, field-level operations, and domain-level memory. The evaluation compares DocsChisel against original documentation, EasyTool, and DRAFT across 9 domains, 3 backbones, and 2 agent paradigms, reporting a 95.89% average improvement in task success rate over original documentation and 75.15% over baselines, with 3x5 repeated runs and Mann-Whitney U tests with Holm correction.","tokens_in":19079,"tokens_out":3093,"duration_ms":31339,"significance":"If the reported results hold, the paper makes a useful contribution to tool documentation engineering for LLM agents: it provides a systematic field-level taxonomy, demonstrates domain/backbone/paradigm dependence, and offers a practical adaptive optimization framework with reproducible artifacts and a statistically rigorous main evaluation protocol. The strengths include the 3x5 repeated evaluation, the use of non-parametric significance testing with multiple-testing correction, and the design of memory-guided field-level operations rather than unconstrained rewriting. However, the central empirical claim of consistent gains is currently undermined by the RQ5 practice of selecting the optimization model and iteration budget on the same test set used for the main RQ3 results, and by an unresolved ambiguity in how multi-tool queries are assigned to tool-specific optimization and test splits.","major_comments":[{"comment":"The held-out test set is not independent of the optimization choices. In RQ5, the authors vary the optimization model and iteration budget and report TC and TS 'on the test set' (Section IV-D), and they use those test-set results to select Claude Haiku 4.5 as the optimization model (Table IV) and to set the default iteration budget to five (Fig. 9). The main RQ3 results are then reported with that same configuration on the same test set, directly contradicting the claim in Section IV-A that 'the test set is held out throughout the optimization process.' Because the baselines are not given the same test-based selection opportunity, the comparison is systematically favorable to DocsChisel. The authors should either move the model/iteration selection to the validation set and report RQ3 on a genuinely untouched test set, or justify that the RQ5 decisions were made before any test access and provide evidence that the selected configuration is not an artifact of test-set overfitting.","section":"Section IV-D and Section IV-A"},{"comment":"The query splitting procedure is under-specified with respect to multi-tool queries. Section III-B states that 'a query involving multiple tools belongs to multiple tool-specific query sets,' and Section IV-A describes a global 5:1:4 split over user queries, but it is not stated whether a query assigned to the global test set can still appear in the tool-specific optimization or validation sets of another tool. If a test query can appear in the optimization set of a different tool, then the 'held-out' test queries have influenced documentation optimization indirectly, inflating the reported gains. The authors should clarify whether the split is global over queries or per tool, and if multi-tool queries are assigned to multiple tool-specific sets, they should verify and report that the global optimization, validation, and test sets are disjoint.","section":"Section III-B and Section IV-A"},{"comment":"The RQ2 impact analysis reports point estimates without confidence intervals or per-domain sample sizes, which weakens the load-bearing claim that information-field effects are heterogeneous across domains, backbones, and paradigms. For example, the claim that removing POF improves TS by 7.5 percentage points for Claude Haiku 4.5 but decreases it for GPT-4o and GLM-5 may be within noise if the data-analysis domain has few queries per tool. Since this heterogeneity is the stated motivation for the entire framework, the authors should report the number of queries per domain/bucket and provide uncertainty estimates or significance tests for the field-effect comparisons.","section":"Section II-C and Fig. 1"},{"comment":"The average improvement percentages reported in RQ3 aggregate relative gains across heterogeneous settings, which can be misleading when baseline TS is low. For instance, a relative TS improvement of 194.22% across nine domains is driven largely by settings where the original TS is near zero. The authors should report absolute percentage-point changes alongside relative percentages, and consider reporting per-setting absolute differences in addition to the averaged relative improvements.","section":"Section IV-B and Fig. 8"}],"minor_comments":[{"comment":"The paper states that a stratified split over task domains and tools was performed but does not include the detailed per-domain and per-tool statistics that would allow the reader to assess balance; these details should be provided in the appendix or on the project website.","section":"Section IV-A"},{"comment":"In the boxplot description, the authors claim that DocsChisel reduces the interquartile range of TC by 41.56% and 52.93% compared with EasyTool and DRAFT, but it is unclear whether these IQR reductions are computed over the 15 runs per setting or pooled across settings; this should be clarified.","section":"Section IV-B"},{"comment":"The sentence 'GLM-5 also outperforms GPT-4o, with relative gains of 55.26% in TC and 67.67% in TS' refers to GPT-4o's lower absolute values, but the wording is easy to misread; please rephrase to make the reference point explicit.","section":"Section IV-D"},{"comment":"The ablation statement 'DOCSCHISEL-NoMem reduces TC and TS by 13.64% and 76.47% compared with DOCSCHISEL' is directionally confusing because NoMem is the ablated variant; it would be clearer to say 'DOCSCHISEL outperforms DOCSCHISEL-NoMem by...'.","section":"Section IV-E"},{"comment":"The threats-to-validity section mentions inter-rater agreement for field identification but does not specify the number of tools or fields annotated per rater for the kappa computation; adding this detail would strengthen the reproducibility of the taxonomy.","section":"Section V"}],"recommendation":"major_revision","confidential_remarks":"The core idea is relevant and the paper is generally well written, but the RQ5 test-set reuse is a serious methodological flaw that must be addressed before the headline numbers can be trusted. The fix is well-scoped: move model/iteration selection to the validation set or use a nested split, and clarify the global disjointness of per-tool query sets. If the authors can do that, the paper may well be publishable after revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The useful parts here are real: the 17-field taxonomy, the RQ2 evidence that field effectiveness varies by domain, backbone, and paradigm, and DocsChisel's design—field-level add/remove/refine driven by failed traces, with domain memory. I'd put the empirical study in front of a reading group, and the framework is a sensible step beyond EasyTool and DRAFT, which only rewrite within existing fields.\n\nThe RQ2 experiment is well constructed: controlled addition/removal of individual fields, multiple backbones and paradigms, and the heterogeneity finding is convincing. The main evaluation also has solid aspects: 3x5 repeated runs, Mann-Whitney U with Holm correction, and the ablation for the memory mechanism is informative.\n\nBut the stress-test note lands. RQ5 explicitly reports TC and TS on the test set for different optimization models and iteration budgets, and those numbers are used to pick Claude Haiku 4.5 and a budget of five. Those same settings are then the main RQ3 comparison, and neither baseline gets an equivalent test-set selection opportunity. That breaks the claimed holdout and makes the comparison systematically favorable to DocsChisel. Also, the query splitting is ambiguous: a multi-tool query belongs to multiple tool-specific sets, and the paper never states whether a test-set query can end up in another tool's optimization set. That is direct leakage if it happens, and the authors need to rule it out explicitly.\n\nMissing baselines—Play2Prompt, ToolScope, JTPro—are a weaker concern, but the choice of only two competitors should be justified. None of these issues sinks the core direction; they are fixable by reporting RQ5 on a separate model-selection set or stating that defaults were fixed before test evaluation, and by documenting the per-tool split precisely. As it stands, I'd say the direction is right but the reported percentages should not be taken at face value.\n\nThis deserves a serious referee, but with major revision requested. The empirical study alone merits publication in some form; the framework evaluation needs to be cleaned up before the headline claims are citable.","headline":"A genuinely useful empirical taxonomy and a sensible adaptive-documentation framework, but the headline gains rest on a test set that is double-dipped for model and budget selection, so the magnitudes are not yet solid.","tokens_in":19662,"tokens_out":1822,"would_cite":false,"duration_ms":18106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Adaptively editing tool documentation, field by field, lifts LLM agent task success by 95.89% over the original docs, the paper argues.","keywords":["tool documentation","LLM agents","information fields","execution trace diagnosis","adaptive optimization","task success rate","field-level editing","failed trace feedback"],"falsifier":"Re-run the main comparison under a strict protocol: freeze the optimization model and iteration budget on a development split, exclude every multi-tool query from any test set whose tool also had that query in an optimization set, and then measure DocsChisel's task-success gain; if the 95.89% improvement over original documentation collapses toward the variance of repeated runs, the reported effect is largely selection-driven.","tokens_in":18579,"feed_emoji":"🛠️","tokens_out":6489,"duration_ms":57351,"temperature":0.7,"pith_summary":"The paper sets out to show that tool documentation is not a fixed input but a tunable resource for LLM agents. It surveys 14 tool-use datasets, identifies 17 common information fields, and finds that adding or removing a single field changes task success by 6.34 percentage points on average, with effect directions flipping across task domains, LLM backbones, and agent architectures. To exploit this, the paper proposes DocsChisel, which analyzes failed execution traces, plans add/remove/refine operations on documentation fields, and iteratively revises each tool's documentation using domain-level accumulated memory. The central claim is that this adaptive process improves task success rate by 95.89% over original documentation and by 75.15% over two existing optimizers, while adding modest token overhead and about 12.65 minutes of optimization time per tool. If true, documentation quality should be treated as a first-class optimization target in agent deployment, not a static artifact.","feed_headline":"DocsChisel lifts agent success rates 96%","feed_subtitle":"Adding, removing, and refining documentation fields per agent setting beats fixed rewrites by 75 percent on average.","key_machinery":"The load-bearing mechanism is a closed optimization loop operating at the information-field level. First, domain-aware execution profiling partitions each tool's queries into optimization and validation sets and collects failed execution traces under the original documentation. Then an LLM diagnoses each failed trace against the current documentation and a growing domain memory, mapping failures to responsible fields; a planner merges these diagnoses into add, remove, and refine operations anchored to the original documentation; and a generator applies the operations. Each candidate documentation is re-executed on validation and optimization sets for regression-aware evaluation, and accepted candidates are stored for final selection. Domain-level memories of failure patterns, effective operations, and editing rules are updated after each iteration, which the ablation study credits with a 76.47% relative task-success gain by the fifth iteration over the no-memory variant.","core_discovery":"The central discovery is heterogeneity: tool documentation across ecosystems supplies very different information fields, and the usefulness of each field depends on the agent setting. Among 17 identified fields, only tool name and functionality description appear in all 14 datasets, while usage guidance and invocation constraint appear in only two. Controlled experiments show that the same field can help in one task domain and hurt in another, help one LLM backbone and hurt another, and reverse its effect when switching from a single-agent to a multi-agent execution flow. The paper therefore concludes that no fixed documentation convention generalizes across agent settings, and that an effective optimizer must adapt field composition to the target agent. DocsChisel is the proposed instantiation of that idea, and the paper reports that it consistently improves both tool invocation correctness and task success rate across all evaluated domains, backbones, and agent architectures.","pith_inferences":["[Editorial inference] The 17-field taxonomy could be reused as an audit schema: measuring which fields a documentation set lacks would predict which agent settings are most likely to fail, before any optimization run.","[Editorial inference] The domain-memory mechanism resembles transfer learning across tools; if it generalizes, one could pretrain memories on a cheaper backbone and apply them to an expensive target agent, cutting optimization cost substantially.","[Editorial inference] A testable extension is to let the optimizer learn when to merge similar tools or split overloaded documentation, operations outside the current add/remove/refine vocabulary.","[Editorial inference] The 6.34 percentage-point average swing from a single field is comparable in size to typical prompt-engineering effects, suggesting documentation and prompt optimization should be co-optimized rather than tuned in isolation."],"forward_implications":["Tool documentation should become an explicit optimization artifact in agent deployment pipelines, alongside prompts, retrieval indices, and model selection.","Documentation built for one agent setting should not be assumed portable; teams would likely need per-domain, per-backbone, and per-architecture field choices rather than a single template.","Failed execution traces are a recoverable signal: organizations already logging agent failures could feed them into a similar diagnose-plan-generate loop without requiring new labeled data.","Because the method trades roughly 12.65 minutes of offline optimization per tool for higher success, it is practical for moderate-sized tool sets, with the main cost being compute rather than human curation.","The reported consistency across repeated runs suggests the optimization reduces variance in agent outcomes, not just the average success rate."],"supporting_citations":[{"why":"Supplies the EasyTool baseline that standardizes documentation into concise tool instructions, one of the two approaches DocsChisel must beat.","marker":"[58]"},{"why":"Supplies the DRAFT baseline that iteratively refines documentation from tool-execution feedback, the other main comparison point.","marker":"[42]"},{"why":"Supplies the WorkBench benchmark with five task domains and the realistic workplace execution environment used for most experiments.","marker":"[48]"},{"why":"Supplies the API-Bank benchmark with four additional task domains, expanding the evaluation to nine domains total.","marker":"[27]"},{"why":"Supplies the reasoning-and-acting single-agent paradigm used as one of the two target agent architectures.","marker":"[56]"},{"why":"Supplies the multi-agent conversation architecture used as the second target agent paradigm.","marker":"[51]"},{"why":"Supplies one of the three LLM backbones used in the empirical study and as a target agent.","marker":"[35]"},{"why":"Supplies one of the three LLM backbones used to test whether field effects vary across models.","marker":"[59]"},{"why":"Supplies the Claude Haiku 4.5 backbone used as the optimization model in the main RQ3 results.","marker":"[3]"},{"why":"Supplies the Mann-Whitney U test used to establish statistical significance of the reported improvements.","marker":"[31]"}],"fun_headline_variants":["DocsChisel adapts tool docs, boosting agent success by 96%","Tool docs matter: DocsChisel customizes them per agent, +96% success","No one-size-fits-all tool docs: DocsChisel adapts, boosting success 96%","DocsChisel: Adaptive tool docs lift LLM agent success 96%","DocsChisel beats fixed tool doc rewrites by 75% on average"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim assumes the held-out test set remained truly independent of the optimizer's design choices: the same test set was used in the sensitivity analysis to select the optimization model and iteration budget, and multi-tool queries appear in several tools' query sets, so any leakage there would inflate the reported gains.","fun_headline_variants_meta":{"raw":{"variants":["DocsChisel adapts tool docs, boosting agent success by 96%","Tool docs matter: DocsChisel customizes them per agent, +96% success","No one-size-fits-all tool docs: DocsChisel adapts, boosting success 96%","DocsChisel: Adaptive tool docs lift LLM agent success 96%","DocsChisel beats fixed tool doc rewrites by 75% on average"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000849,"raw_usage":{"total_tokens":3710,"prompt_tokens":980,"completion_tokens":2730,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":596,"completion_tokens_details":{"reasoning_tokens":2622}},"tokens_in":596,"tokens_out":2730,"duration_ms":17306,"temperature":1.0,"reasoning_tokens":2622,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T04:14:23.686589+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the main comparison under a strict protocol: freeze the optimization model and iteration budget on a development split, exclude every multi-tool query from any test set whose tool also had that query in an optimization set, and then measure DocsChisel's task-success gain; if the 95.89% improvement over original documentation collapses toward the variance of repeated runs, the reported effect is largely selection-driven.","supporting_citations":[{"cited_title":"Easytool: Enhancing llm-based agents with concise tool instruction,","cited_arxiv_id":null,"evidence_quote":"Supplies the EasyTool baseline that standardizes documentation into concise tool instructions, one of the two approaches DocsChisel must beat."},{"cited_title":"From exploration to mastery: Enabling llms to master tools via self-driven interactions,","cited_arxiv_id":null,"evidence_quote":"Supplies the DRAFT baseline that iteratively refines documentation from tool-execution feedback, the other main comparison point."},{"cited_title":"Api-bank: A comprehensive benchmark for tool-augmented llms,","cited_arxiv_id":null,"evidence_quote":"Supplies the API-Bank benchmark with four additional task domains, expanding the evaluation to nine domains total."},{"cited_title":"(2024) Gpt-4o","cited_arxiv_id":null,"evidence_quote":"Supplies one of the three LLM backbones used in the empirical study and as a target agent."}],"review_version":1}