{"id":"9897a18c-dfce-4c8b-9538-b771f2eb16f8","arxiv_id":"2607.18245","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"SAAG decomposes agent-calling evaluation into registry, schema, and grounding stages, and its structured feedback reduces value hallucination and improves solve rate in small models, though the argument-precision claim is contradicted by its own results.","lead":"This paper introduces SAAG, a three-stage diagnostic framework that separates agent-calling failures into function-name registry, argument-schema, and argument-value grounding stages, each with metrics and targeted repair feedback. Tests on 3,000 Glaive queries with three sub-4B local models show structured feedback reduces value hallucination and raises solve rate, but the abstract's precision-improvement claim is contradicted by the paper's own Table 2.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"VHR reduction claim is confounded: SAAG computes Stage-3 metrics on a different, smaller gate-passing population than Direct, so reported hallucination drops may be selection artifacts.","rationale":"The central empirical promise of SAAG is that structured feedback repairs tool calls. The most credible repair signal is the consistent VHR reduction, since the precision claims in §4 are already contradicted by Table 2 (Ministral and Granite SAAG precision is below Direct). But the VHR reduction is at risk of being an artifact of stage-conditional filtering. I chose this as the load-bearing concern because it attacks the one headline result that is consistent across all three models. The reader's weakest assumption (lexical overlap) is real but secondary: even if VHR were semantically valid, the metric would still be confounded by differing gate-passing populations. The matched-subset test is decisive because it controls the conditioning event. If the gap persists on the matched subset, the hallucination-reduction claim is credible; if not, the paper's main evidence disappears. I do not call for outright rejection because the diagnostic decomposition and the solve-rate improvements may survive, but the paper must reanalyze the VHR comparison and correct the precision claim before it can be relied on. This reinforces the reader's CONDITIONAL verdict without changing it.","tokens_in":16943,"tokens_out":10955,"duration_ms":116527,"concrete_test":"Recompute AVEM, QSLO, and VHR on the matched subset of queries that pass RC and SC in all three feedback modes (Direct, Binary, SAAG). If the SAAG-vs-Direct VHR gap is substantially smaller or flips for Ministral/Granite, the reported hallucination reduction is a selection artifact. This requires only per-row predictions, not new inference; authors should release per-row stage outcomes for the 3,000 queries × 3 scales × 3 conditions.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Abstract and §4 claim structured feedback 'reduces value hallucination' for every model. Table 3 reports AVEM/QSLO/VHR as stage-conditional metrics computed only on predictions that pass Registry Conformance and Structural Completeness; the cascade halts at Stage 2 for RPR<1 or SPR>0. Table 3 also shows SAAG substantially lowers RPR relative to Direct for Ministral (0.984→0.886) and Granite (0.958→0.926), and slightly for LLaMA (0.968→0.963). Thus Stage 3 is evaluated on a smaller, structurally cleaner population under SAAG than under Direct. Direct VHR for Ministral is 0.277 averaged over nearly all rows; SAAG VHR of 0.166 is averaged only over the subset that already satisfied structural completeness. If structurally incomplete predictions are also value-hallucinating, excluding them mechanically lowers VHR without any grounding improvement. The paper itself warns these metrics 'should not be interpreted as unconditional population statistics' (§2, §3), yet §4 interprets cross-condition VHR drops as evidence that 'structured feedback consistently reduces hallucination.' The 40% reduction for Ministral is the strongest evidence for the framework's central promise; it is unidentifiable from the reported aggregates because the conditioning event differs by condition.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SAAG, a cascaded diagnostic framework for evaluating tool-calling LLMs. It decomposes a prediction into three stages: registry conformance (FNEM/NCS), structural completeness (RPR/PDS/SPR), and argument grounding (AVEM/QSLO/VHR). On failure at any stage, a deterministic, templated correction message is returned to the model, enabling iterative self-repair without revealing ground truth. The framework is evaluated on 3,000 filtered Glaive function-calling queries, three sub-4B models (Llama 3.2:3B, Ministral 3B, Granite 3.1:3B), and three registry scales (5, 10, 15 functions), comparing direct single-pass inference, binary feedback, and structured SAAG feedback. The paper's central claims are that structured feedback consistently improves argument precision and reduces value hallucination relative to the other regimes, while end-to-end F1 gains are modest and model-dependent.","tokens_in":17237,"tokens_out":7076,"duration_ms":73411,"significance":"If the empirical claims held, the paper would make a useful contribution: stage-decomposed diagnostics are a clean and practically motivated idea, and the use of deterministic fix templates plus external ground-truth checks for extrinsic metrics are strengths. The core diagnostic decomposition—separating function-name errors, schema errors, and argument-grounding errors—is likely valuable independent of the correction loop. However, the paper's headline empirical claims are not currently supported by its own reported tables. The precision claim is contradicted by Table 2, and the value-hallucination reduction is confounded by the stage-conditional population shift. The paper explicitly warns about the latter in §2 and §3 but then interprets the conditional numbers as unconditional improvements in §4. These issues are load-bearing for the central contribution as stated, though they are addressable with revised claims and additional analysis.","major_comments":[{"comment":"The abstract and §4 claim that structured feedback 'consistently improves argument precision' and that 'SAAG improves argument precision for all three models.' Table 2 reports extrinsic Precision: LLaMA 0.524→0.540, Ministral 0.705→0.661, Granite 0.747→0.717. Two of the three models decrease. If the intended claim is about the intrinsic AVEM metric in Table 3, the term 'argument precision' is ambiguous, and the §4 discussion of Table 2 precision is still inconsistent with the data. The central claim must be restated to match the reported metric, and the abstract should be revised accordingly.","section":"Abstract, §4, Table 2"},{"comment":"The claimed reduction in value hallucination is not identifiable from the reported aggregates because the Stage-3 population differs by condition. Table 3 states that later-stage metrics are computed only on predictions that passed earlier stages, and Stage 2 requires RPR=1 and SPR=0. SAAG lowers Stage 2 pass probability: RPR for Ministral falls from 0.984 to 0.886, Granite from 0.958 to 0.926, and LLaMA from 0.968 to 0.963. Hence SAAG's VHR of 0.166 for Ministral is averaged over a smaller, structurally cleaner subset than Direct's 0.277. If structurally incomplete predictions tend to contain value hallucinations, the drop is a selection artifact, not evidence of reduced hallucination. The paper's own §2 and §3 warnings make this clear, yet §4 interprets the drops as evidence that 'structured feedback consistently reduces hallucination.' Report unconditional VHR over all predictions, or","section":"§4, Table 3"},{"comment":"The headline VHR/AVEM improvements are based on a lexical-overlap definition of grounding. The Q3 fix templates instruct the model to 'use the exact numbers from the user's request' and 'use the precise wording or value from the user's request' (Table 4). This makes improvements in AVEM and VHR partly definitional: copying query spans will raise the scores regardless of whether the prior value was actually wrong relative to the ground-truth argument. The paper itself acknowledges in §6 that 'values that are semantically correct but lexically distant from the query will be incorrectly flagged.' Before interpreting VHR reductions as reductions in hallucination, the intrinsic metric should be validated against ground-truth arguments—for example, by reporting the extrinsic argument-accuracy of the subset flagged as hallucinated.","section":"§2.3, Appendix D, §6"}],"minor_comments":[{"comment":"Model naming is inconsistent: §3 says 'Granite 3.1:3B', Appendix A uses 'granite4:3b', and the tables/figures refer to 'Granite 4'. Clarify which model was actually run.","section":"§3, Appendix A, Tables/Figures"},{"comment":"The LLaMA F1 direct value is reported as 0.527±.370, which appears to be a typo for 0.527±.033; the ±.370 value is inconsistent with the other entries in the row.","section":"Table 2"},{"comment":"The text says structured feedback converges within 2–3 iterations (Figure 4), but Figure 4 shows solve rate across feedback modes, not convergence; the cumulative attempt plot is Figure 3. Please fix the cross-reference.","section":"§4, Figures 3–4"},{"comment":"The Yao et al. ReAct reference lacks a year and venue. Other references (e.g., Liu et al., 2026 for Ministral 3) appear to be preprint-only; please provide complete bibliographic data.","section":"References"},{"comment":"Typos and small inconsistencies: 'Firs' in §6 Limitations; 'a agent' in the Abstract; 'Tmax = 15' in Appendix A versus 'Dmax = 15' in §2.4. Please proofread.","section":"§2, §3, Appendix A"},{"comment":"Solve rate is defined as passing all three SAAG stages and matching the ground-truth function name; it does not check argument values against ground truth. Since Stage 3 is a lexical, not semantic, check, the label 'solve rate' may overstate the fraction of truly correct calls. Consider a stricter definition or an explicit caveat.","section":"§3, Solve Rate definition"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about stage-conditional confounding is valid and should be the first issue the authors address. The precision contradiction in Table 2 is a straightforward but central wording problem. The paper has a useful diagnostic core, and the empirical claims can likely be repaired by reporting unconditional metrics and qualifying the lexical-grounding limitation. I would not recommend acceptance in the current form."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know: SAAG is worth reading for the framework, but not for the results as written. The three-stage decomposition (registry conformance, structural completeness, argument grounding) and the eight-metric suite are a genuinely useful addition to the tool-calling evaluation conversation. Prior work like ToolBeHonest and Reliability Alignment separated selection from usage hallucination, but the cascaded gate design with stage-conditional metrics and a deterministic, ground-truth-free repair loop is a practical package that goes beyond what I've seen. The paper is also honest in places: it explicitly warns that later-stage metrics are conditional, and it names the lexical-overlap limitation.\n\nThat honesty makes the overclaims more frustrating. The abstract says structured feedback \"consistently improves argument precision,\" but Table 2 shows the opposite for Ministral (0.705 to 0.661) and Granite (0.747 to 0.717). Only LLaMA improves, and modestly. That is a load-bearing contradiction, not a quibble.\n\nThe bigger issue is the VHR claim. The stress-test note is right: SAAG lowers RPR substantially for Ministral (0.984→0.886) and Granite (0.958→0.926), meaning many more predictions halt at Stage 2. Stage 3 metrics are then computed on a smaller, structurally cleaner population. Comparing Direct VHR (computed on nearly all rows) to SAAG VHR (computed only on rows that pass earlier gates) is apples-to-oranges. The 40% VHR reduction for Ministral is the paper's strongest evidence, and it is unidentifiable from the reported aggregates. The paper even states these metrics \"should not be interpreted as unconditional population statistics,\" then interprets them exactly that way.\n\nThe lexical-grounding definition is a limitation, not a fatal flaw. The authors acknowledge it, and the framework would still be useful for diagnosing surface-level grounding failures. But calling these \"value hallucination\" reductions overstates what lexical overlap can establish.\n\nNo code, data, or prompt templates are released, which matters because the fix templates and the fuzzy-match thresholds are load-bearing. Without them, the reproducibility claim is unverifiable.\n\nWho gets value from this: anyone building tool-calling evaluation pipelines or doing inference-time repair. The conceptual decomposition is solid and the metric definitions are concrete enough to implement. But I would not cite the empirical results as they stand.\n\nMy recommendation: send it to peer review. The framework deserves referee time, and the problems are fixable — correct the abstract and §4, handle stage-conditional metrics without cross-population comparisons, report uncertainty properly, and release artifacts. This is a useful contribution buried under overclaiming, and a good referee can help dig it out.","headline":"Useful diagnostic framework; the empirical section overclaims — the 'consistent improvement in precision' is contradicted by the paper's own Table 2, and the headline hallucination reductions are confounded by stage-conditional population shifts.","tokens_in":17773,"tokens_out":1690,"would_cite":true,"duration_ms":18584,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Three-stage diagnosis of AI tool calls improves grounding and reduces hallucinated argument values.","keywords":["function calling","tool use evaluation","LLM hallucination","diagnostic evaluation","structured feedback","argument grounding","registry conformance","self-repair"],"falsifier":"Construct a held-out set where the correct argument value is a synonym, paraphrase, or inferred entity not appearing verbatim in the query (e.g., a ticker symbol derived from a company name). If corrected calls that SAAG flags as hallucinations actually execute correctly against the real API, then the VHR reduction is partly an artifact of the lexical proxy rather than a true grounding improvement.","tokens_in":1296,"feed_emoji":"🧭","tokens_out":5033,"duration_ms":80580,"temperature":0.7,"pith_summary":"The paper argues that exact-match scoring of tool calls hides distinct failure modes: choosing the wrong registered function, violating the argument schema, or inventing values not grounded in the user's query. SAAG cascades three gates: registry conformance, structural completeness, and argument grounding. Each failure triggers a deterministic, ground-truth-free correction message, enabling iterative self-repair. Across three local sub-4B model families and registry sizes of 5, 10, and 15 functions, structured feedback consistently improves argument precision and reduces value hallucination compared to single-pass inference or uninformative binary feedback. The central message is that telling a model what failed is substantially more effective than telling it that it failed.","feed_headline":"Stage-by-stage feedback cuts AI tool-call hallucination","feed_subtitle":"Structured diagnostics let small models repair bad function calls at inference time, cutting hallucinated argument values.","key_machinery":"The load-bearing mechanism is the strict three-stage cascade of intrinsic evaluators: Registry Conformance (FNEM, NCS), Structural Completeness (RPR, PDS, SPR), and Argument Grounding (AVEM, QSLO, VHR). The cascade halts at the first failing stage, so later-stage metrics are conditioned on earlier passes. Deterministic templates generate stage-specific corrections without leaking ground truth, and the loop repeats up to 15 attempts. The VHR definition, which counts a numeric argument as hallucinated unless its exact floating-point token appears in the query and scores strings by lexical PartialRatio, carries the paper's diagnosis of hallucination.","core_discovery":"The paper claims that agent-calling reliability is best understood through a strict cascade of three gates: does the function name exist in the registry, does the argument set satisfy the schema, and are argument values grounded in the user's query? Using 3,000 filtered queries across registry sizes 5, 10, and 15 and three sub-4B models, the authors find argument grounding is the dominant failure mode. Structured feedback reduces the value hallucination rate for all models, with a 40% reduction for one reasoning-oriented model, and improves argument precision across the board. Binary feedback sometimes increases hallucination. The paper also documents an overcorrection effect in reasoning mo","pith_inferences":["Because grounding is defined by lexical overlap, semantically valid values that are paraphrased or inferred will be flagged as hallucinations; the reported VHR reductions may therefore overstate true hallucination on real-world queries.","A testable extension is to add a semantic-similarity check as a fourth gate or soft override to QSLO; this would likely recover many false-positive hallucinations and turn mixed F1 results into consistent gains.","The random-distractor registry construction means real deployments with semantically confusable tools will likely show lower grounding performance, but the diagnostic taxonomy should transfer since it is query-dependent.","The overcorrection effect suggests a cheap safeguard: apply the grounding correction only when VHR is non-zero over the contextual subset, or gate corrections on the model's confidence, an intervention the paper leaves for future work."],"forward_implications":["Argument grounding, not function-name selection, is the main bottleneck; benchmarks that collapse failure modes into one score misdirect improvement effort.","Telling a model what failed is substantially more effective than telling it that it failed, across model families and registry scales.","Structured feedback consistently reduces value hallucination, with the largest drop being a 40% VHR reduction for a reasoning-oriented model.","Iterative correction converges within 2-3 attempts and reaches a higher ceiling than binary feedback.","Reasoning-oriented models can overcorrect initially correct predictions after feedback, motivating confidence-gated correction."],"fun_headline_variants":["Three-stage diagnosis cuts AI tool-call hallucination","Staged feedback reduces hallucinated argument values","Cascaded checks improve small model function calls","Binary feedback fails, staged signals repair AI tools"],"cache_read_input_tokens":19072,"weakest_assumption_plain":"The framework's grounding stage assumes that lexical overlap between predicted argument values and the user query (exact match or PartialRatio) is a valid proxy for operational correctness; semantically correct but lexically distant values are therefore counted as hallucinations.","fun_headline_variants_meta":{"raw":{"variants":["Three-stage diagnosis cuts AI tool-call hallucination","Staged feedback reduces hallucinated argument values","Cascaded checks improve small model function calls","Binary feedback fails, staged signals repair AI tools"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000322,"raw_usage":{"total_tokens":1638,"prompt_tokens":723,"completion_tokens":915,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":467,"completion_tokens_details":{"reasoning_tokens":868}},"tokens_in":467,"tokens_out":915,"duration_ms":10542,"temperature":1.0,"reasoning_tokens":868,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T15:05:00.295558+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Construct a held-out set where the correct argument value is a synonym, paraphrase, or inferred entity not appearing verbatim in the query (e.g., a ticker symbol derived from a company name). If corrected calls that SAAG flags as hallucinations actually execute correctly against the real API, then the VHR reduction is partly an artifact of the lexical proxy rather than a true grounding improvement.","supporting_citations":[],"review_version":1}