{"id":"fa39e43c-54b3-4ed4-a27d-4fae4c3f71cf","arxiv_id":"2412.12151","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SMARTCAL reduces tool-misuse and overconfidence in LLM question answering by injecting self-evaluation instructions and a confidence-accuracy lookup table into step-wise reasoning.","lead":"This paper studies how often large language models misuse tools during question answering and proposes SMARTCAL, a three-part framework that adds self-evaluation and confidence calibration to existing tool-use pipelines. The framework reports an average 8.6 percent accuracy gain and a 21.6 percent lower calibration error across three datasets and three models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported 21.6% ECE reduction may be an artifact of the convention that assigns confidence 0 whenever no valid score is extracted; the calibration claim needs recomputation before it can be trusted.","rationale":"The reader's weakest assumption concerned transfer from the small dev-set lookup table and faithfulness of the calibration model's edits. Those are real concerns, but the more load-bearing problem is in the evaluation itself: the ECE metric as implemented is distorted by the zero-default for missing confidence scores. This is not an external design choice; it is explicitly stated in Appendix A.3.2, and the paper even acknowledges the misleading edge case but still includes those numbers in the headline average. The recalibration claim is therefore not secure without recomputation. The QA accuracy claim (8.6%) is less affected by this particular artifact, though it may have its own issues. The right response is to keep the verdict conditional but make the central condition a reanalysis of ECE with a valid or separate bin for unextracted confidence scores. This is a concrete, checkable fix, so the paper should not be rejected outright, but its central calibration result cannot be accepted as reported.","tokens_in":14883,"tokens_out":5310,"duration_ms":57447,"concrete_test":"Recompute Table 3 and the CPC lookup table after excluding all steps/questions for which no valid confidence score was extracted, or alternatively assign them to a separate 'unstated' bin that is excluded from ECE. Report per-cell ECE and the average percentage reduction under this correction. In particular, recompute the gpt-3.5-turbo/PopQA ART(V) cell (currently 0.010); if its corrected ECE rises substantially, the 21.6% aggregate decrease is partly an artifact. Also compare confidence-extraction coverage between ART(V) and SMARTCAL across all cells to determine whether SMARTCAL's apparent improvement is driven by changes in regex-extraction success rather than true recalibration.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's central calibration claim rests on ECE numbers that are contaminated by the zero-fill convention for missing confidence scores. Section A.3.2 explicitly states: \"When calculating average tool confidence, we default the confidence score to zero when we fail to extract tool usage.\" In ECE, this places every unextracted step or question into the lowest confidence bin. The authors themselves note the misleading consequence: when many wrong answers are binned at zero confidence, the model can appear \"perfectly\" calibrated. That is exactly what happens for gpt-3.5-turbo on PopQA, where the model frequently refuses to answer (A.3.1): all those refusals are wrong, all are assigned confidence 0, and ART(V) shows ECE 0.010 in Table 3. SMARTCAL's ECE of 0.087 for that cell is actually worse, yet the average 21.6% reduction is computed over a table including this artifact. The same zero-fill is used when building the confidence-accuracy table in the CPC component, so the prior itself may be distorted by extraction failures. Until ECE is recomputed without this convention, the main calibration result is not a trustworthy measure of calibration quality.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies tool-use calibration in LLM-based agents. It first reports that LLMs exhibit 'tool-abuse' — using unnecessary tools with overconfident verbalized confidence scores — and that calibration errors remain high across model families. It then proposes SMARTCAL, a multi-agent framework with three components: Self-Evaluation (SE), which generates tool-use instructions from task familiarity and example similarity; Confidence Prior Collection (CPC), which builds a confidence-accuracy lookup table on a heldout dev set; and Augmented Reasoning (AR), in which a calibration model edits step-wise confidence scores according to that table. The authors report that SMARTCAL improves QA accuracy by an average of 8.6 percentage points and reduces ECE by 21.6% across two tool-use frameworks (ART, DSP), three datasets (Mintaka, PopQA, Entity Questions), and three model families (gpt-3.5-turbo, gpt-4, llama-3-70b-instruct). An ablation study on Mintaka isolates the contributions of the SE and CPC components.","tokens_in":15102,"tokens_out":3617,"duration_ms":36793,"significance":"If the reported improvements are robust, SMARTCAL addresses a real and understudied problem: LLM agents often use tools inappropriately and with poorly calibrated confidence. The framework is methodologically transparent — the authors release code and data, provide detailed prompts in the appendix, and present ablations of each component. The use of two tool-use frameworks and three model families is a strength, as is the focus on a heldout confidence prior rather than an in-sample fit. However, the paper's central calibration claim is currently weakened by a known artifact in the ECE computation (zero-filling unextracted confidence scores) and by the absence of a deterministic recalibration baseline. With a recomputation of ECE and a stronger baseline comparison, the contribution would be a useful step toward self-aware tool-use evaluation.","major_comments":[{"comment":"The ECE results are contaminated by the convention, stated in Section A.3.2, of defaulting unextracted confidence scores to zero. Under ECE, all such cases fall into the lowest confidence bin; if those cases are also wrong, the model appears nearly perfectly calibrated. This is exactly what happens for gpt-3.5-turbo on PopQA, where refusals are common (A.3.1): ART(V) receives ECE 0.010 while SMARTCAL receives 0.087, meaning SMARTCAL looks worse precisely because it avoids the zero-fill artifact. The 21.6% average ECE reduction in Section 4.2 is computed over a table that includes this cell, so the headline calibration improvement is not trustworthy. Please recompute ECE without the zero-fill convention (e.g., excluding unextracted cases, or treating them as missing) and report whether the average reduction persists.","section":"A.3.2, Table 3"},{"comment":"The paper does not compare SMARTCAL's calibration editing to a deterministic recalibration baseline. Since the CPC module produces an empirical confidence-accuracy table, a natural baseline is to replace each step's verbalized confidence with the corresponding bin's empirical accuracy (or to apply standard histogram binning / isotonic regression on the dev set). Without such a baseline, it is unclear whether the improvement comes from the table itself or from the calibration model's prompt-driven edits. Please add a deterministic recalibration baseline to Table 3 and to the ablation in Table 5.","section":"4.2, Appendix A.2.2"},{"comment":"The introductory statistics on tool-abuse — 'over 20% of total reasoning steps' involve tool misuse and 'more than 90%' of stated confidence falls in an overconfident bin — are not backed by a dedicated result table. Section 4.1 gives aggregate accuracy, and Figure 4 shows tool-frequency for one model/dataset, but there is no quantitative definition of 'misuse' or a per-model, per-dataset table supporting the over-20% and over-90% claims. Similarly, the claim in Figure 3 that calibration error remains stable regardless of model capability is based on a small set of points with no error bars or significance test. Please add a table with precise metrics (e.g., fraction of steps where the chosen tool is not among the teacher-selected useful tools; fraction of confidence scores where confidence exceeds bin accuracy) across all models and datasets.","section":"1, 4.1, 4.3"},{"comment":"The confidence-accuracy lookup table is the core of the calibration mechanism, but it is constructed from only 200–500 dev questions and uses coarse confidence bins (stepsize 10, as shown in the CPC block of Figure 1). The paper does not report the number of samples per bin or any uncertainty estimate (e.g., bootstrap confidence intervals). Given that the table is used to edit test confidences, the reader cannot assess how sensitive the ECE results are to bin-boundary choices or to small bin counts. Please report per-bin sample sizes and bootstrap intervals, and discuss how the dev distribution transfers to the test set.","section":"2.2, A.1"}],"minor_comments":[{"comment":"The abstract says '8.6 percent increase in QA performance'; the underlying numbers in Table 2 are absolute percentage-point increases (e.g., from 45.4% to 53.0%). Please clarify whether the claim is relative or absolute, and use consistent terminology.","section":"Abstract, 4.1"},{"comment":"The DSP (V) prompt in Table 6 contains a typo: 'confidence socre' should be 'confidence score'.","section":"Table 6"},{"comment":"The calibration prompt in Table 11 contains a typo: 'resaoning process' should be 'reasoning process'.","section":"Table 11"},{"comment":"The caption says 'accessed between Feburary 2024 to June 2024'; 'Feburary' is a typo for 'February'.","section":"Table 2 caption"},{"comment":"In the AR description, the phrase 'the calibration model h(x, d) interacts with both tool-use agent result and confidence prior' would benefit from stating explicitly which model (gpt-3.5-instruct-0914) is used, as is done in A.2.1, to avoid ambiguity.","section":"2.3"},{"comment":"In the Figure 5 caption and its referring text, the zero-confidence interval is attributed to 'regular expressions failed to extract a valid confidence score'; this should be stated in the main text as well, since it directly affects interpretation of the calibration curves.","section":"4.3, Figure 5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central calibration claim currently rests on ECE numbers that include a known zero-fill artifact, which can both inflate and deflate ECE depending on the cell. I strongly recommend asking the authors to recompute ECE after excluding or properly handling unextracted confidence scores, and to add a deterministic recalibration baseline. The tool-abuse statistics in the introduction are also insufficiently supported. If these points are addressed, the paper could be a solid contribution; as it stands, the headline 21.6% ECE reduction is not yet substantiated."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear colleague,\n\nWhat you should know: SMARTCAL is a three-part wrapper for ART/DSP that adds teacher-model self-evaluation, a dev-set confidence-accuracy prior, and a calibration-model edit step. On low-popularity open-domain QA subsets it reports an average 8.6% EM gain and 21.6% ECE reduction. The paper is among the first to study calibration specifically in multi-step tool use, and the framework is simple and modular.\n\nWhat's good: The authors ship code and prompts. The ablation isolates the SE and CPC components. The paper is honest about its own limitations: appendix A.3.2 openly describes the zero-fill convention for missing confidence scores and notes that it can make a model look perfectly calibrated when refusals are binned at zero. That transparency is valuable.\n\nThe soft spots: The headline ECE number is computed under that same zero-fill. Until ECE is recomputed with missing confidences handled properly (excluded, or as a separate bin), the calibration claim is not fully trustworthy. The stress-test note cites the gpt-3.5-turbo/PopQA cell (ART(V) ECE 0.010 vs SMARTCAL 0.087) as evidence the average is inflated; actually that cell makes SMARTCAL look worse, so dropping it would raise the average improvement. The real problem is broader: extraction failures across all cells get zero-confidence credit, which can distort the reliability curve and the comparison. Also missing: error bars, repeated runs, a standard recalibration baseline like temperature scaling or binning-only, and a dedicated table quantifying the tool-abuse statistics claimed in the introduction. The low-popularity subsets are small (151 Mintaka test questions) and the popularity thresholds are somewhat arbitrary.\n\nNone of this is fatal. The EM gains are consistent across models and frameworks, and the ablation shows both components do something. The calibration piece needs rework before the numbers can be taken at face value.\n\nWho this is for: anyone working on tool-use reliability, selective RAG, or verbalized confidence. It deserves a serious referee, but a revision should recompute ECE without the zero-fill, add variance estimates, and compare against a plain recalibration baseline. I'd send it out.","headline":"A transparent, early-stage calibration method for tool-use LLMs; the headline ECE gains rest on a questionable zero-fill convention that the paper itself flags.","tokens_in":15637,"tokens_out":2243,"would_cite":true,"duration_ms":21289,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SMARTCAL shows that LLM tool-use agents overconfidently misuse tools, and that a self-evaluation plus a confidence–accuracy lookup table can reduce this, improving QA accuracy and calibration.","keywords":["large language models","tool use","calibration","Expected Calibration Error","verbalized confidence","self-evaluation","retrieval-augmented generation","agent reasoning"],"falsifier":"Run SMARTCAL with a deliberately shuffled confidence–accuracy table (pairing each confidence bin with the accuracy of a different bin). If the reported ~21.6% Expected Calibration Error reduction is still reproduced, the calibration model is not genuinely using the prior, and the claimed mechanism collapses.","tokens_in":14668,"feed_emoji":"🛠️","tokens_out":8886,"duration_ms":77055,"temperature":0.7,"pith_summary":"This paper argues that large language models given tools to answer questions routinely fall into 'tool-abuse': they call tools that do not help, and they do so with confidence scores that overstate the chance that their reasoning is correct. The same pattern appears across model families and persists as models get more capable. To counter it, the paper proposes SMARTCAL, a fully prompt-based loop with three parts—self-evaluation that restricts which tools may be used, a confidence–accuracy lookup table built from a small held-out dev set, and an augmented reasoning step in which a calibration model edits step-wise confidence scores according to that table. Across two tool-use frameworks, three datasets, and three model families, the loop raises exact-match QA accuracy by an average of 8.6 percentage points and reduces Expected Calibration Error by an average of 21.6% relative to baselines.","feed_headline":"Tool-abusing LLMs gain 8.6% accuracy after recalibration","feed_subtitle":"A dev-set confidence table plus self-evaluation edits cut Expected Calibration Error by 21.6%.","key_machinery":"The load-bearing mechanism is the confidence–accuracy lookup table produced by the Confidence Prior Collection step. Verbalized confidence scores from a held-out dev set are averaged per reasoning trace, binned, and paired with the empirical accuracy of each bin; during Augmented Reasoning, a calibration model is prompted to replace each step's confidence score with the accuracy that the table associates with its confidence bin. The self-evaluation (SE) step supplies the complementary machinery: a teacher model issues task-familiarity and example-similarity verdicts that are compiled into tool-use instructions, telling the agent which tools to use, which to avoid, and when to rely on internal knowledge. Together the table and the instructions turn a single-agent tool-use loop into a multi-agent pipeline with explicit calibration feedback.","core_discovery":"The central discovery is that LLM tool-use agents systematically overconfident-misuse tools: in over 20% of reasoning steps a model calls at least one tool that does not help, and more than 90% of stated confidence falls in bins where reported confidence exceeds actual accuracy. SMARTCAL addresses this by constraining tool selection through a teacher-model self-evaluation, grounding verbalized confidence in an empirical confidence–accuracy table computed on a small dev set, and then having a calibration model edit each step's confidence score toward the accuracy associated with its bin. The paper reports that this three-stage loop is compatible with two existing frameworks—ART and DSP—and yields an average 8.6% increase in QA accuracy and a 21.6% decrease in ECE against baselines, with ablations showing both the self-evaluation and the confidence prior contribute.","pith_inferences":["The dev-set confidence–accuracy table is a lightweight, model-specific prior; a natural extension is to refresh it online as new confident answers accumulate, which would make SMARTCAL adaptive to distribution shift.","The same three-stage loop—constrain tool choice, build an empirical confidence prior, and edit step-wise scores—could apply to other agentic settings such as code execution or web navigation, where overconfident tool calls also erode trust.","The paper's finding that calibration error stays roughly flat as model capability grows implies that simply scaling models will not fix overconfidence; calibration-aware prompting or training may need to be a separate axis of improvement."],"forward_implications":["SMARTCAL raises exact-match QA accuracy by an average of 8.6 percentage points over baseline ART and DSP frameworks.","It lowers Expected Calibration Error by an average of 21.6% compared to verbalized-confidence baselines, so agents' stated confidence better tracks their true correctness.","The gains appear across three model families and three datasets, suggesting the calibration loop is model- and framework-agnostic rather than a one-off prompt tuning.","Ablations show self-evaluation alone contributes about 2.9% accuracy and 21.6% lower ECE, while the confidence prior alone lowers ECE by about 39.4%, implying both components work partly independently."],"supporting_citations":[{"why":"Supplies the ART tool-use framework that SMARTCAL builds on and compares against.","marker":"Paranjape et al., 2023"},{"why":"Supplies the DSP framework, the second baseline SMARTCAL is adapted to.","marker":"Khattab et al., 2023"},{"why":"Provides the low-popularity sampling rationale and the PopQA dataset used in experiments.","marker":"Mallen et al., 2023"},{"why":"Introduces verbalized confidence elicitation, which SMARTCAL adapts to step-wise tool confidence.","marker":"Lin et al., 2022"},{"why":"Provides the Entity Questions dataset used in experiments.","marker":"Sciavolino et al., 2021"},{"why":"Provides the Mintaka dataset used in experiments.","marker":"Sen et al., 2022"},{"why":"Defines Expected Calibration Error, the calibration metric SMARTCAL is evaluated on.","marker":"Naeini et al., 2015"},{"why":"Motivates the confidence prior collection design with hybrid confidence elicitation approaches.","marker":"Xiong et al., 2024"}],"fun_headline_variants":["LLMs overconfidently misuse tools; SMARTCAL recalibrates","Tool-abusing LLMs gain 8.6% accuracy after SMARTCAL","SMARTCAL reduces tool overconfidence, raises QA accuracy","Overconfident LLM tool use corrected by SMARTCAL calibration","Self-aware tool-use calibration yields 8.6% LLM accuracy gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a confidence–accuracy lookup table computed from 200 to 500 held-out dev questions accurately represents the confidence–accuracy relationship on the test set, and that the calibration model's prompt-following edits faithfully apply that table to test confidence scores.","fun_headline_variants_meta":{"raw":{"variants":["LLMs overconfidently misuse tools; SMARTCAL recalibrates","Tool-abusing LLMs gain 8.6% accuracy after SMARTCAL","SMARTCAL reduces tool overconfidence, raises QA accuracy","Overconfident LLM tool use corrected by SMARTCAL calibration","Self-aware tool-use calibration yields 8.6% LLM accuracy gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000269,"raw_usage":{"total_tokens":1592,"prompt_tokens":887,"completion_tokens":705,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":606}},"tokens_in":503,"tokens_out":705,"duration_ms":6466,"temperature":1.0,"reasoning_tokens":606,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T18:11:16.098720+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SMARTCAL with a deliberately shuffled confidence–accuracy table (pairing each confidence bin with the accuracy of a different bin). If the reported ~21.6% Expected Calibration Error reduction is still reproduced, the calibration model is not genuinely using the prior, and the claimed mechanism collapses.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the Mintaka dataset used in experiments."}],"review_version":1}