{"id":"b2f61343-9115-4bb4-aab7-8c977ccea697","arxiv_id":"2501.05255","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"CallNavi is a new benchmark for LLM function calling with unfiltered, nested, multi-step API tasks; GPT-4o leads the leaderboard and a two-step routing pipeline improves fine-tuned models.","lead":"The paper introduces CallNavi, a new benchmark of 729 questions and 579 API functions for testing how well large language models select and call APIs. It also reports an empirical comparison of 18 models and proposes two prompting strategies that improve performance on complex calls.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GPT-4o generated the benchmark data and also serves as the LLM-as-a-judge scorer, creating a circularity that may inflate its top ranking; the two-step routing improvement is reported with an undefined 'overall' metric and no baseline rows.","rationale":"The reader's weakest assumption—that the synthetic GPT-4o-generated dataset may not represent real-world API collections—is a legitimate external-validity concern. However, the more immediate and testable risk is internal: the benchmark generation and evaluation are both performed by GPT-4o, and the headline improvement metric in Table 6 is undefined. These problems do not merely threaten generalizability; they undermine the specific empirical conclusions that GPT-4o ranks highest and that two-step routing and backward inference provide large gains. The paper itself flags LLM-judge subjectivity only as a minor threat, but it sits at the center of the abstract's claims. A held-out, independently authored validation set would settle whether the advantage is real, and a re-reported Table 6 with baselines and defined metrics would settle whether the proposed optimization actually improves performance. Because these are addressable in revision, the appropriate verdict remains conditional rather than rejection, and my read does not change the reader's overall conditional stance.","tokens_in":17239,"tokens_out":7304,"duration_ms":70796,"concrete_test":"Construct a held-out set of 100–200 new questions on the same 579-API corpus, authored independently (human-written or generated by a non-OpenAI model such as Llama-3.1-70B) and manually validated; rerun all Table 4, Table 6, and Table 7 experiments. If GPT-4o's lead shrinks or the two-step/backward gains decrease, the original results are confounded by the GPT-4o generation/judging pipeline. In parallel, re-report Table 6 with explicit AST Exact Match baselines (fine-tuned model on the full API list), define 'overall' as the question-count-weighted mean, and include standard errors; if the gain over baseline falls within one standard error, the improvement claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claims—model ranking and the value of two-step routing/backward inference—rest on a benchmark whose API functions and easy questions were generated by GPT-4o (Section 4) and whose primary 'Overall GPT Score' is assigned by GPT-4o-as-judge (Section 5.2). This double role can systematically favor GPT-4o: it is evaluated on tasks it wrote and scored by itself. The paper's Threats to Validity admits 'LLM-as-a-judge introduces potential subjectivity in evaluation', but this is not a peripheral caveat—it directly affects the headline finding that GPT-4o is best and that the proposed hybrid approach improves routing. Separately, Table 6 presents the key two-step generation improvement using an 'overall' column that is never defined in Sections 5 or 7, and it omits the no-routing baseline rows needed to verify the claimed gain. Table 7's backward-thinking comparison likewise reports single-point estimates with no variance or significance testing. Until these metrics are clarified and the GPT-4o confound is tested, the abstract's assertion that the proposed methods 'significantly improve API execution' is not supported by the reported evidence.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CallNavi, a new benchmark for evaluating LLMs on API function calling and routing. The benchmark contains 729 questions and 579 API functions across 10 domains, with easy, medium, and hard difficulty levels that require single calls, multiple sequential calls, and nested/dependent calls with up to five steps. The authors evaluate 18 commercial, general-purpose, and fine-tuned LLMs using several metrics: API routing exact match, AST-based syntax/structural/exact-match scores, an LLM-as-a-judge score using GPT-4o, and a newly proposed stability score. They also propose two optimizations for zero/few-shot settings: a two-step generation pipeline that separates API selection (routing) from parameter generation, and a backward-inference strategy for multi-step API planning. The paper reports that GPT-4o and GPT-4o mini perform best overall, that most open models are competitive at routing but weaker at parameter generation, and that the two proposed optimizations improve hard-level performance.","tokens_in":17469,"tokens_out":6410,"duration_ms":61457,"significance":"If the empirical claims hold, CallNavi would be a useful addition to the function-calling benchmark landscape. Unlike API-Bank, ToolEyes, and BFCL, CallNavi presents large unfiltered candidate pools, multi-call sequences, and nested dependency scenarios, which better reflect realistic chatbot deployments. The released dataset and the breadth of evaluated models are strengths: the authors provide detailed statistics, manual validation procedures, and stability measurements, and they compare against existing benchmarks. The proposed routing recipes (two-step generation and backward reasoning) are simple and potentially practical. However, the empirical evaluation currently has confounds that prevent unambiguous interpretation: GPT-4o generated part of the benchmark and also served as the judge, the key optimization tables omit baselines and define their headline metric only implicitly, and no variance or significance information is reported for the main optimization results. These issues are addressable, and the benchmark itself remains valuable, but the reported improvements and model rankings should be treated as provisional until they are resolved.","major_comments":[{"comment":"GPT-4o has a double role that directly affects the headline ranking and the optimization claims. Section 4 states that GPT-4o generated the API functions and the easy questions, and Section 5.2 uses GPT-4o as the LLM-as-a-judge scorer for the 'Overall GPT Score' in Table 4. Thus GPT-4o is evaluated on data it helped create and is scored by itself. The Threats to Validity paragraph only acknowledges 'potential subjectivity in evaluation' and does not address this specific lifecycle. The Pearson correlation with AST match reported in Section 6.4 does not remove the confound, because both metrics can share the same model-specific bias. Please provide a concrete test: e.g., score a random sample with a different judge model or with human annotators and report agreement/reranking, or present model rankings using only the AST metrics. Without this, RQ1 and the claim that GPT-4o is the best model are not fully supported.","section":"Section 4 and Section 5.2"},{"comment":"The 'overall' column in Table 6 is never defined in Section 7.1 or in the text around the table. From the numbers it appears to be the difficulty-weighted average of the easy/medium/hard columns (e.g., for NexusRaven: (0.657*456 + 0.457*188 + 0.188*85)/729 = 0.551), but the text should state this explicitly and specify which underlying metric is being averaged (presumably AST match). More importantly, Table 6 omits the corresponding one-step baseline rows from Table 4 (the 'without routing' condition). The claim that separating API selection from parameter generation 'significantly improves performance' cannot be verified without these baselines. Please add the baseline rows for each model and state the metric and aggregation formula.","section":"Table 6 / Section 7.1"},{"comment":"The optimization results are reported as single-point estimates with no variance or significance testing, despite the paper itself using five runs for stability experiments in Section 6.4.1. Section 7.2 claims a '30% improvement in hard-level API calling tasks', but the two rows in Table 7 show absolute improvements of 0.283 and 0.341, so the source of the 30% figure is unclear. LLM outputs are stochastic, so a single run can produce misleading gains. Please report mean and standard deviation (or confidence intervals) over at least the same five runs used for the stability scores, and apply a paired significance test when claiming that the proposed methods improve routing or parameter generation.","section":"Tables 6 and 7 / Sections 7.1-7.2"},{"comment":"The proposed Levenshtein stability score measures the average normalized similarity to the first output x0, not agreement among all outputs. If the first sample is an outlier and the remaining n-1 samples are identical, the score will be low even though the model is highly stable. This undermines the claim in Section 6.4.1 that the score measures 'the stability of the model's outputs across multiple runs'. Please replace it with a symmetric measure, such as the average pairwise similarity or similarity to the majority output. Also, the Election Stability Score definition does not specify F2 when there is only one unique output; the example sets F2=0, but this convention should be made part of the formal definition.","section":"Section 5.3"}],"minor_comments":[{"comment":"The column headings 'All Avg.' and 'Macro Avg.' are not defined. Please clarify whether they are weighted by question count or are simple means across the difficulty columns.","section":"Table 4"},{"comment":"NexusRaven and Gorilla v2 have 'N/A' for Syntax Validity. Please explain why these fine-tuned models are not evaluated on that metric, since they do have entries for the AST-match columns.","section":"Table 4"},{"comment":"The model selection criteria are described loosely ('based on their performance, architecture, and relevance'). Please state the inclusion criteria more concretely, especially because the paper excludes some models such as Firefunction v2 but does not give a complete list of considered models.","section":"Section 6.1"},{"comment":"Table 8 is difficult to read because the rows for different models are run together in the text; please format it as a normal table with separate rows and clear column alignment.","section":"Section 8.5 and Table 8"},{"comment":"The conclusion says '500 APIs and 700 questions', while Table 2 reports 579 API functions and 729 questions. Please use consistent numbers.","section":"Section 9.2"},{"comment":"There are several minor typos and formatting inconsistencies, e.g., 'Mistal models' in Section 6.4 and 'Y AML' in Section 8.5 and Table 8. Please proofread the final version.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The benchmark dataset and the breadth of the empirical comparison are genuinely useful for the software engineering community. However, the GPT-4o-as-generator-and-judge confound and the undefined/under-specified optimization metrics are load-bearing for the paper's main empirical claims. These issues are fixable within the paper's scope, so I recommend major revision rather than rejection. The authors should also consider releasing the evaluation scripts to facilitate reproduction."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the CallNavi paper. Short version: the benchmark is real and worth knowing about; the empirical results need more work before you quote them.\n\nWhat's new: CallNavi gives you 729 questions over 579 APIs across 10 domains, with a difficulty split (easy/medium/hard) and, importantly, candidate pools of 100+ APIs and nested multi-step calls. That is a genuine gap relative to API-Bank, ToolEyes, and BFCL. The dataset is released, the statistics are detailed, and the manual validation is described plausibly. The Election Stability Score is a small but sensible addition—it measures majority dominance across repeated runs, which freq@topk misses. The two-step routing idea (general LLM selects APIs, fine-tuned model generates parameters) is also reasonable and worth testing.\n\nThe load-bearing weakness is the GPT-4o lifecycle: GPT-4o generated the API functions and the easy questions, and GPT-4o also acts as the judge for the 'Overall GPT Score.' That doesn't invalidate the benchmark, but it means the headline ranking (GPT-4o first) and the claimed improvements from the proposed pipelines are confounded. A self-authored, self-judged test can favor the author model in ways that have nothing to do with routing quality. The paper's threats section mentions LLM-as-a-judge subjectivity, but it treats it as a side note rather than a threat to the main conclusion.\n\nSecond, Table 6's 'overall' column is never defined, and the table omits the no-routing baseline rows. You can't verify the claimed gain from 2-step generation without seeing the same models without routing. Table 7's backward-thinking comparison is also single-point numbers with no variance or significance testing. Minor: the abstract says 'significantly improve' but there are no significance tests anywhere in the paper.\n\nWorth noting: the stability score derivation and examples are clear; the case studies are useful. The paper would be strengthened by adding error bars and statistical tests, clarifying the overall metric, reporting baselines for Table 6, and ideally running a human or independent judge to decouple generation from evaluation.\n\nVerdict: this deserves a serious referee. The core benchmark contribution is solid and the dataset release makes it checkable. But the paper should not be accepted in its current form—the empirical claims need to be either backed by proper comparisons and statistics or softened.","headline":"CallNavi is a genuinely useful new benchmark for LLM function calling, but the empirical claims are softer than the abstract suggests because GPT-4o generated the data, judged the outputs, and tops the leaderboard.","tokens_in":18028,"tokens_out":1989,"would_cite":true,"duration_ms":17927,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CallNavi shows that realistic API routing needs unfiltered candidate pools, multi-step calls, and nested dependencies.","keywords":["function calling","API routing","benchmark","large language models","nested API calls","AST match","stability score","multi-step reasoning"],"falsifier":"Independently sample, say, 50 CallNavi questions and have fresh annotators verify ground-truth API calls against the stated API list, checking for missing or invented functions and wrong parameter names; a substantial error rate would invalidate the benchmark's conclusions. As a second check, run the two-step and backward-inference pipelines on an existing independent function-calling benchmark with a large candidate pool; if the gains do not reproduce there, the paper's recipe is dataset-specific.","tokens_in":17074,"feed_emoji":"🧭","tokens_out":6514,"duration_ms":58044,"temperature":0.7,"pith_summary":"The paper introduces CallNavi, a benchmark that tests whether large language models can pick the right API from an unfiltered list of more than 100 candidates, then build correct JSON calls, including multi-step and nested call sequences. Eighteen commercial, general-purpose, and fine-tuned models were evaluated on 729 questions across 10 domains, with difficulty levels from single calls to five-step dependent chains. The paper argues that existing benchmarks such as API-Bank, ToolEyes, and BFCL understate this challenge because they use smaller candidate pools or omit nested dependencies. It also proposes two inexpensive improvements: a two-step pipeline that lets a general LLM do routing while a parameter-focused model fills in arguments, and a backward-inference prompting strategy that plans API calls from the final goal backward. If the benchmark is accepted, it gives chatbot builders a more realistic stress test and simple recipes for improving API routing and parameter generation.","feed_headline":"GPT-4o tops API-call benchmark; all models stumble on nested calls","feed_subtitle":"CallNavi tests 18 models on 100+ APIs and multi-step calls; splitting routing from parameters lifts scores.","key_machinery":"The load-bearing object is the CallNavi dataset itself: 579 manually validated API functions and 729 questions, with an unfiltered candidate list of up to 115 APIs, organized into easy, medium, and hard levels, where hard questions require up to five sequential calls and nested dependencies. The evaluation pipeline includes syntax-validity, structural-accuracy, and AST exact-match checks on generated JSON, plus an LLM-as-a-judge score and a stability score computed as $(F_1-F_2)/(N-F_2)$ over repeated runs. The two proposed mechanisms are a two-step generation pipeline, which separates API routing from parameter generation, and a backward-inference prompting procedure that plans from the final answer's API backward through its parameter dependencies.","core_discovery":"CallNavi's central claim is that function-calling ability should be measured under realistic selection pressure: a model facing over 100 plausible APIs must name the right ones in order, fill in parameters, and chain calls where later parameters depend on earlier outputs. On this test, OpenAI's GPT-4o and GPT-4o mini lead, but even they drop sharply on hard questions, and most open-weight models that route correctly fail to produce valid parameter JSON. The paper shows that decomposing the task, first letting a general-purpose LLM select the API list and then letting a fine-tuned or smaller model generate the full call, improves overall accuracy over end-to-end generation, and that a backward-inference prompt, which starts from the final API and recursively searches for missing dependencies, raises GPT-4o's hard-level routing from 0.611 to 0.894 exact match and parameter accuracy from 0.388 to 0.729. A proposed election-style stability score quantifies run-to-run consistency across five repeated samples, complementing AST exact-match metrics.","pith_inferences":["Because the dataset's API specs and questions are synthetic, a natural next test is to run the same routing protocols against live, versioned API documentation from a real product and see whether the model rankings and the gains from two-step routing persist.","The benchmark's hard-level improvements from backward inference might transfer to other long-horizon tool-use settings, such as code repair workflows where a final edit depends on earlier retrieval steps, but that transfer is not established by the paper.","The election stability score could be adopted alongside standard consistency metrics in other structured-generation benchmarks, since it directly penalizes ties and rewards one dominant output; its behavior with very small sample sizes deserves a check.","A practical deployment implication is that chatbot systems should cache or validate the most frequent output across repeated runs rather than trusting a single generation, because stability and accuracy are only weakly aligned for many models."],"forward_implications":["API routing accuracy and parameter-generation accuracy are distinct capabilities: several open models route well but collapse on JSON parameter generation, so benchmarks should report both separately.","Splitting routing from parameter generation improves fine-tuned models like NexusRaven, Gorilla v2, and xLAM when GPT-4o supplies the routing, bringing them close to general LLMs.","Backward-inference prompting yields large gains on hard questions, suggesting that dependency-aware planning rather than larger context windows is the lever for multi-step API calls.","Context windows below roughly 6K tokens truncate CallNavi inputs and cause failures, while larger context alone does not fix reasoning errors.","The stability score reveals consistent-output differences among models that overall accuracy metrics miss, which matters for trust in finance and healthcare chatbots."],"supporting_citations":[{"why":"API-Bank is the prior benchmark with fewer than five API candidates per task that CallNavi positions itself against.","marker":"[18]"},{"why":"ToolEyes is a tool-learning benchmark that lacks nested API interactions, and CallNavi uses it as a comparison baseline.","marker":"[40]"},{"why":"BFCL supplies the evaluation format for JSON API calls and the AST-style metrics that CallNavi adapts.","marker":"[38]"},{"why":"This prior study runs five repeated generations to assess model stability, which motivates CallNavi's repeated-run design.","marker":"[34]"},{"why":"Reverse Chain contributes the backward multi-API planning idea that the paper's Backward Inference approach builds on.","marker":"[44]"},{"why":"CauseJudger is a second abductive-reasoning inspiration behind the backward-thinking prompt.","marker":"[14]"},{"why":"This work supplies the LLM-as-a-judge method used to compute the GPT score that correlates with AST match.","marker":"[45]"},{"why":"Gorilla OpenFunctions v2 is one of the fine-tuned function-calling models benchmarked and improved by GPT-4o routing.","marker":"[17]"}],"fun_headline_variants":["Backward-inference prompt lifts GPT-4o hard API routing to .894","Split routing from params: hybrid beats end-to-end on API calls","GPT-4o tops API benchmark but open models fail on parameter JSON","Election score measures stability in LLM function calling","Decompose API calls: general LLM selects, fine-tuned fills params"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the synthetic API list and questions, generated by GPT-4o and then manually checked, faithfully represent realistic API collections and user queries; if that representativeness fails, the benchmark rankings and the measured gains of the two proposed methods may not carry over to production systems.","fun_headline_variants_meta":{"raw":{"variants":["Backward-inference prompt lifts GPT-4o hard API routing to .894","Split routing from params: hybrid beats end-to-end on API calls","GPT-4o tops API benchmark but open models fail on parameter JSON","Election score measures stability in LLM function calling","Decompose API calls: general LLM selects, fine-tuned fills params"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000245,"raw_usage":{"total_tokens":1521,"prompt_tokens":918,"completion_tokens":603,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":508}},"tokens_in":534,"tokens_out":603,"duration_ms":5182,"temperature":1.0,"reasoning_tokens":508,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:13:19.446911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Independently sample, say, 50 CallNavi questions and have fresh annotators verify ground-truth API calls against the stated API list, checking for missing or invented functions and wrong parameter names; a substantial error rate would invalidate the benchmark's conclusions. As a second check, run the two-step and backward-inference pipelines on an existing independent function-calling benchmark with a large candidate pool; if the gains do not reproduce there, the paper's recipe is dataset-specific.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Reverse Chain contributes the backward multi-API planning idea that the paper's Backward Inference approach builds on."},{"cited_title":"Patil Fanjia Yan, Tianjun Zhang, Ion Stoica, and Joseph E","cited_arxiv_id":null,"evidence_quote":"Gorilla OpenFunctions v2 is one of the fine-tuned function-calling models benchmarked and improved by GPT-4o routing."}],"review_version":1}