{"id":"0259f2f2-c035-4307-bb72-86eca3325cf8","arxiv_id":"2504.20168","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A classifier trained on layerwise decodes and BERTScore similarities to the final output gives better-calibrated confidence for tool calls and improves expected utility at medium and high risk levels.","lead":"To decide when a language model should be trusted to call a tool, the authors extract signals from the model's internal layers, combine them with the model's own confidence score, and train a small classifier. The method improves the balance between getting useful actions and avoiding harmful ones in simulated tool-use tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Exact-match labels may inflate MICE's utility gains; need semantic-equivalence relabeling to check whether the advantage over HRE/NWKR reflects real tool-call correctness.","rationale":"The paper is a careful empirical study with open code, significance tests, ablations, and honest limitations. I considered several alternative objections: the simplified ETCU utilities (tn=fn=0) are explicit and applied equally across methods; the lack of direct isotonic/Platt baselines is partially mitigated by the confidence-only ablation, which shows that logistic regression on raw confidence approximates those baselines; and random forest's ability to emit confidence values above 0.9 is exactly what the high-risk threshold requires and is checked by smECE. None of these is as load-bearing as the correctness label. The exact-match definition is the interface between the method and the real-world safety claim: every training label, every ETCU outcome, and every significance test is computed against it. The paper provides no estimate of how often exact mismatch corresponds to functional mismatch. Because the STE data are synthetic and the few-shot prompts are drawn from the same dataset, exact match may be partly a stylistic property. A confidence estimator that exploits layerwise lexical stability can track that stylistic property better than raw-confidence calibrators, so the measured utility gap could be an artifact of the surrogate label. This concern is external to the MICE construction but is falsifiable and central to interpreting the results. It does not refute the paper; it makes the headline claim conditional on label validity, which is exactly the reader's conditional verdict. No change to the reader's verdict is needed.","tokens_in":15350,"tokens_out":6564,"duration_ms":77653,"concrete_test":"Sample 200 test examples from STE and produce semantic-equivalence labels for the generated tool calls (e.g., execute both the generated call and the gold call in the STE simulator and compare resulting effects, or have a second LLM judge functional equivalence). Recompute MICE RF, MICE LR, HRE, and NWKR utility values in Table 1 at medium and high risk using these relabeled outcomes. If MICE RF's margin over HRE/NWKR shrinks below significance or reverses, the exact-match assumption is load-bearing.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—MICE confidences improve expected tool-calling utility at medium and high risk—rests on the correctness label used for both training and evaluation. Section 4.1 defines correct as \"exactly matches the one given by STE,\" and no analysis quantifies how often a generated call is semantically correct but string-mismatched. Tool calls admit multiple valid serializations (argument order, quoting, whitespace, aliases), so this is not a merely stylistic footnote. If such mismatches are common, the MICE training signal is noisy and, more importantly, the ETCU calculation in Section 3.2 assigns the large negative utility fp to calls that would actually succeed in the environment. Because MICE RF is the variant most able to exploit surface-level layerwise BERTScore regularities, its utility advantage over HRE and NWKR could partly reflect skill at predicting the exact-match surrogate rather than at assessing true correctness. The Limitations section acknowledges single-dataset evaluation but does not address label validity. Without label validation, the headline utility numbers are uninterpretable for real tool-calling safety.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes MICE, a family of confidence estimators for tool-calling LLM agents. For each generated tool call, MICE decodes per-token predictions from intermediate layers via logit lens, computes BERTScore between each layer's decoding and the final greedy output, adds the raw token-level confidence as an extra feature, and trains either a logistic regressor or a random forest to predict whether the call is correct. The paper also introduces expected tool-calling utility (ETCU), a normalized decision-theoretic metric with a Bayes-optimal threshold, and evaluates on the simulated trial-and-error (STE) dataset with three Llama-3 models, comparing against raw confidence, histogram regression (HRE), and Nadaraya-Watson kernel regression (NWKR). The reported results show that MICE RF matches or improves smECE, yields significantly higher ETCU than HRE/NWKR at medium risk and often at high risk, remains competitive in zero-shot API generalization, and needs only a few hundred training examples.","tokens_in":15558,"tokens_out":6883,"duration_ms":69306,"significance":"The contribution is potentially useful: if confirmed, MICE gives a way to exploit model internals for safer call/no-call decisions, and ETCU is a sensible metric for comparing confidence estimators under asymmetric risk. The paper is transparent in several ways: code is released, statistical significance is assessed by permutation tests, feature ablations are reported, and zero-shot generalization is examined. These are real strengths. The main caveat is that the empirical support rests on a single synthetic dataset with labels defined by exact string match, and the baseline set omits the most standard recalibration methods, so the breadth of the claims in the title and abstract is not yet fully supported.","major_comments":[{"comment":"Section 4.1 states that 'we label a generated tool call as correct if and only if it exactly matches the one given by STE.' Tool calls generally admit many string-distinct but semantically equivalent serializations (argument order, quoting, whitespace, aliases), so this label is a surrogate for true correctness. Because the same labels are used both to train MICE and to compute the ETCU values in Table 1 and Figure 4, the reported utility improvements over HRE and NWKR could reflect skill at predicting the exact-match surrogate rather than at identifying calls that would succeed in the environment. The Limitations section does not discuss this. I ask for a label-validation analysis: at least a human-annotated or execution-based sample quantifying the mismatch rate, and a rerun of Table 1 under a semantically normalized correctness label, to show that the MICE advantage survives.","section":"§4.1"},{"comment":"Section 4.4 compares MICE against raw confidence, HRE, and NWKR, but not against Platt scaling or isotonic regression, even though these are the standard recalibration baselines and are cited in §7. Since MICE LR is described as an extension of Platt scaling, the experiments do not currently isolate the contribution of the model-internal BERTScore features from the contribution of ordinary logistic recalibration. Adding Platt scaling and isotonic regression to Table 1 and Figure 4 would make the claim that MICE 'beats or matches strong baselines' directly supported.","section":"§4.4, Table 1"},{"comment":"The abstract and conclusion state that MICE 'significantly improves' ETCU in medium- and high-risk regimes, but the permutation tests summarized in the text show that MICE RF is not significant at high risk for Llama3.2 and that MICE LR is only significant at medium risk for Llama3.1. The paper's own summary is more cautious, but the abstract overstates uniformity; the claims should be aligned with the significance pattern.","section":"§5, Table 1"}],"minor_comments":[{"comment":"The phrase 'with loss of generality' before assuming tn=fn=0 is not accurate; this is a modeling simplification, and the following sentence's 'slightly less general' should be the only framing.","section":"§3.2"},{"comment":"The significance markers are placed immediately before negative numbers (e.g., '*-0.015∗†'), which is easy to misread as part of the value; the caption should clarify the marker placement.","section":"Table 1"},{"comment":"The phrase 'as suggested by reviewers' is an artifact of the review process and should be removed or replaced with a neutral motivation.","section":"§6"},{"comment":"Since Platt scaling and isotonic regression are mentioned in §7, a sentence explaining why only HRE and NWKR are used as main baselines would help readers understand the design choice.","section":"§7"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper is solid but narrow. The exact-match label issue is the one I would want resolved before publication; a revision that adds semantic-equivalence validation would materially strengthen the paper. The omission of Platt scaling and isotonic regression from the main comparison is also important, because it directly affects how the contribution is framed. I did not find any indication of unsound derivations or intentional overclaiming beyond the abstract-level generalization."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid empirical paper on confidence estimation for tool-calling LLMs. The core idea is to decode from every intermediate layer via logit lens, compute BERTScore similarity to the final output, and feed those features plus raw token probability into a small logistic regression or random forest to predict whether a generated tool call is correct. They also introduce ETCU, a decision-theoretic metric that converts calibrated confidences into call/no-call decisions under different risk levels. On the STE dataset with three Llama3 models, MICE RF consistently beats the HRE and NWKR calibration baselines on ETCU at medium and high risk, while matching them on smoothed ECE. The work is careful: held-out test sets, permutation tests, ablations, sample-efficiency curves, zero-shot API generalization, and open-source code. That's real evidence and deserves a serious look.\n\nWhat's genuinely new is the combination of layerwise logit-lens features with a learned classifier for tool calls, and the ETCU metric. The metric is simple but useful: it forces you to think about the cost of false positives. The finding that BERTScore similarity to the final layer helps calibration beyond raw confidence is plausible and supported by the ablations, though the effect is mainly in the random forest variant.\n\nSoft spots, in proportion. The biggest is the label definition: a call is 'correct' only if it exactly matches the STE gold string. As you know, tool calls can have multiple valid serializations. So semantically correct but differently formatted calls are counted as errors. This is a real concern for external validity, because both training and ETCU evaluation rest on that label. The paper states it plainly in Section 4.1, but never quantifies how often the mismatch is merely cosmetic. If it's common, MICE RF's utility advantage could partly be an artifact of learning the exact-match surrogate. I would have liked a small human-annotated or semantic-equivalence subset to calibrate this. The paper's own Limitations section mentions single-dataset evaluation but not label validity, which is a gap.\n\nThe second issue is that the strongest gains are concentrated in MICE RF. MICE LR is often not significantly better than the baselines. The advantage over HRE/NWKR is not uniform across models: the high-risk gain is not significant for Llama3.2. That tempers the headline claim but doesn't sink it.\n\nThe missing isotonic regression / Platt scaling baselines are less of a problem, since footnote 11 and the ablation show confidence-only LR is essentially Platt scaling, and it is competitive in some settings.\n\nOverall: the central claim holds on the STE dataset and the three models tested, with appropriate caveats. The exact-match label issue is the main thing I'd want revised or at least quantified. This deserves peer review. I'd send it out.","headline":"A careful empirical paper on layerwise-internal confidence for tool-calling agents; the utility metric is useful and the experiments are solid, but exact-match labels on a single synthetic dataset leave the headline gains vulnerable to a semantic-equivalence artifact.","tokens_in":16110,"tokens_out":3211,"would_cite":true,"duration_ms":31083,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Model-internal confidence estimators (MICE) decode every transformer layer, compare each layer's provisional answer to the final tool call, and use those similarities to produce calibrated confidence scores that increase expected…","keywords":["confidence calibration","tool-using agents","model internals","logit lens","BERTScore","expected calibration error","expected tool-calling utility","large language models"],"falsifier":"Re-annotate the STE test set with semantic-equivalence judgments (for example, two annotators or a judge model decide whether each generated call would accomplish the same action as the gold call), then retrain and re-evaluate MICE against HRE and NWKR on that relabeled set. If the expected-tool-calling-utility advantage at medium and high risk disappears or shrinks to insignificance, the central claim is an artifact of exact-match labeling rather than a property of internal-layer confidence.","tokens_in":15157,"feed_emoji":"🛠️","tokens_out":10607,"duration_ms":98466,"temperature":0.7,"pith_summary":"Tool-using agents need to know when to trust the API call they just generated, because a wrong call can be cheap or catastrophic. This paper proposes MICE, a family of confidence estimators that read the model's own internal trajectory: it decodes a preliminary answer from each transformer layer, compares each layer's answer to the final one with BERTScore, and feeds those similarities plus the model's raw token confidence into a small trained classifier. On the simulated trial-and-error tool-calling dataset with three Llama3 models, MICE matches or beats histogram and kernel recalibration baselines on smooth expected calibration error and, more importantly, improves a new utility metric that charges a cost for incorrect calls. The advantage appears as the cost of a wrong call rises: at medium and high risk levels, executing a call only when calibrated confidence exceeds a Bayes-optimal threshold yields higher expected utility than the baselines. A random-forest version is the strongest, and it retains its edge with only a few hundred training examples and generalizes to unseen APIs.","feed_headline":"Decoding every layer catches bad tool calls at high risk","feed_subtitle":"MICE reads every layer's provisional answer plus raw confidence, improving the risk-aware call/no-call decision.","key_machinery":"The load-bearing object is the per-layer decoding feature: each transformer layer's hidden state is mapped through the unembedding matrix to produce a provisional token, so the whole layer stack yields a sequence of provisional strings $y^{(1)}, \\dots, y^{(\\ell)}$ that gradually converge to the final output $y$ (logit lens). BERTScore, a string-similarity measure built from a separate BERT-family embedder, turns each provisional string into a number measuring how close that layer's answer already is to the final answer. The hypothesis the method tests is that a tool call whose answer was already stable across the later layers is more likely to be correct than one that snaps into place only at the final layer, even when both end with identical token probabilities; the learned logistic-regression or random-forest classifier is what reads that trajectory.","core_discovery":"At the paper's core is the claim that the way a language model's predictions evolve across its layers tells you whether its final tool call can be trusted. MICE operationalizes this by taking, for each layer $i$, the layerwise hidden state $h^{(i)}_{t-1}$, multiplying by the unembedding matrix $W_{\\mathrm{out}}$ to get logits, argmax-decoding a preliminary string $y^{(i)}$, and scoring the similarity of each $y^{(i)}$ to the final string $y$ with BERTScore. A trained classifier maps these layer-similarity features, together with the product of non-formatting token probabilities, to a probability that the call is correct. The paper's central empirical finding is that when these probabilities drive a minimum-Bayes-risk call/no-call decision, they beat both a histogram regressor and a Nadaraya–Watson kernel regressor on expected tool-calling utility at medium risk for all three tested models and at high risk for two of them, while staying statistically tied on smooth ECE. The random-forest version is the strongest configuration; it is also sample-efficient, matching the full-data kernel baseline with about one-fifth of the training set, and zero-shot transfer to held-out APIs.","pith_inferences":["One extension the paper does not pursue is using MICE-style features for continual calibration at deployment time, where the classifier is updated online as new tool calls arrive; the sample-efficiency result suggests this could work.","A sharper test of the mechanism would be to artificially perturb only the final layers of a model, making the decoded trajectory disagree with the final output; MICE should then withhold confidence even when raw confidence is high, which would isolate whether the layer-trajectory signal is causal rather than correlational.","Because MICE requires hidden states, an implicit consequence for closed models is that similar estimators would need to be trained on logprobs or other exposed signals; the gap between MICE and raw confidence suggests what closed APIs lose by hiding internals."],"forward_implications":["With MICE confidence and the risk-adjusted threshold of Section 3.2, an agent can abstain from executing a tool call when confidence is low, improving expected utility in medium- and high-risk settings relative to recalibrated raw confidence.","Because MICE matches the calibration-focused baselines on smooth ECE while improving utility, calibration alone is not sufficient: the paper's ETCU metric rewards estimators that separate correct from incorrect calls, and MICE does so better.","MICE is sample-efficient: the random-forest variant reaches or exceeds the full-data kernel regressor with roughly 300 training examples, so the internal-layer features are informative even with small labeled sets.","MICE generalizes across tools: models trained on 49 of the 50 STE APIs perform indistinguishably from baselines trained on all APIs when evaluated on the held-out API.","Raw token confidence is the weakest estimator in high-risk regimes and can yield negative utility; MICE's core value is recovering safe behavior from an overconfident base generator."],"supporting_citations":[{"why":"Supplies the logit lens technique: decoding from each layer with the unembedding matrix to get the preliminary strings $y^{(i)}$ that MICE compares.","marker":"(nostalgebraist, 2020)"},{"why":"Provides BERTScore, the similarity measure between each layer's decoded string and the final output that forms MICE's main features.","marker":"(Zhang et al., 2020)"},{"why":"Supplies the STE dataset, the gold tool calls, and the few-shot demonstration and in-context selection used to generate and label the tool calls.","marker":"(Wang et al., 2024)"},{"why":"Defines smooth ECE, the calibration metric used in the headline comparison, and the kernel-regression procedure underlying the NWKR baseline.","marker":"(Błasiok and Nakkiran, 2024)"},{"why":"Histogram regression estimator baseline (HRE) against which MICE is compared.","marker":"(Nobel, 1996)"},{"why":"Foundations of Nadaraya–Watson kernel regression used by the NWKR baseline.","marker":"(Nadaraya, 1964; Watson, 1964)"},{"why":"Llama3-8B-Instruct, Llama3.1-8B-Instruct, and Llama3.2-3B-Instruct, the three LLMs on which MICE and baselines are evaluated.","marker":"(Dubey et al., 2024)"}],"fun_headline_variants":["Layer-decoded confidence improves tool-call decisions at risk","Internal layer signals calibrate tool-using agents for safety","MICE: Model-internal confidence for safer tool calls","Layer-wise confidence estimation boosts risk-aware tool use"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire experiment depends on the STE dataset's binary label, which marks a generated tool call correct only if it exactly equals the gold call string; if many correct calls differ in wording, formatting, or argument order, the labels are noisy and the measured utility gains may partly reflect surface-form matching rather than true tool-calling ability.","fun_headline_variants_meta":{"raw":{"variants":["Layer-decoded confidence improves tool-call decisions at risk","Internal layer signals calibrate tool-using agents for safety","MICE: Model-internal confidence for safer tool calls","Layer-wise confidence estimation boosts risk-aware tool use"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000384,"raw_usage":{"total_tokens":2072,"prompt_tokens":1028,"completion_tokens":1044,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":644,"completion_tokens_details":{"reasoning_tokens":979}},"tokens_in":644,"tokens_out":1044,"duration_ms":8443,"temperature":1.0,"reasoning_tokens":979,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T05:36:28.080166+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the STE test set with semantic-equivalence judgments (for example, two annotators or a judge model decide whether each generated call would accomplish the same action as the gold call), then retrain and re-evaluate MICE against HRE and NWKR on that relabeled set. If the expected-tool-calling-utility advantage at medium and high risk disappears or shrinks to insignificance, the central claim is an artifact of exact-match labeling rather than a property of internal-layer confidence.","supporting_citations":[{"cited_title":"Weinberger, and Yoav Artzi","cited_arxiv_id":null,"evidence_quote":"Provides BERTScore, the similarity measure between each layer's decoded string and the final output that forms MICE's main features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines smooth ECE, the calibration metric used in the headline comparison, and the kernel-regression procedure underlying the NWKR baseline."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Foundations of Nadaraya–Watson kernel regression used by the NWKR baseline."}],"review_version":1}