{"id":"d61e0b46-7c65-483c-96d2-a1f6637d31d4","arxiv_id":"2411.14054","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new Korean benchmark shows that single-turn tool-call accuracy does not predict a model's multi-turn conversational tool-use performance.","lead":"FunctionChat-Bench is a Korean-language benchmark that tests language models on four kinds of outputs in tool-use dialogs: tool calls, answers, slot questions, and relevance detection. It finds that strong single-turn tool-call accuracy does not predict strong multi-turn conversational performance, based on evaluations of eight models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The single-turn vs multi-turn attribution is confounded: the two datasets differ in item content and difficulty, so Solar's dramatic drop in pass rate is not clean evidence for the central claim.","rationale":"The reader's weakest_assumption correctly identifies the lack of calibrated difficulty between the single-call and dialog subsets as the load-bearing issue. My analysis agrees: the paper's evidence for the central claim is a confounded comparison, and the proposed concrete test (re-evaluating dialog Tool Call items as single-turn queries, or adding history to single-call items) would directly settle whether the multi-turn environment or item content drives the observed drops. The benchmark itself is a valuable contribution with a documented human-corrected LLM-judge protocol, and the error analysis is useful. The central claim that 'high single-turn accuracy does not necessarily translate' is a weak existential statement and may survive even under the confound, but the paper's stronger causal inference about multi-turn influence is not established. This supports the reader's CONDITIONAL verdict. I recommend no change: the paper should be accepted conditionally, with the authors asked to run the matching control or soften the causal language.","tokens_in":11504,"tokens_out":4534,"duration_ms":45498,"concrete_test":"Extract the 70 Tool Call turns from FunctionChat-Dialog. For each, present the final user utterance alone (without the preceding conversation history) along with the same available function list as in the original dialog, and score with the same Tool Call rubric and LLM judge. If Solar's pass rate on these 'single-turn-ized' dialog items remains near 0.63, the low score is due to item difficulty, not multi-turn context. If it rises toward 83.6, multi-turn context is implicated. Conversely, take the 500 single-call items and prepend a short neutral multi-turn history (e.g., chit-chat) before each user utterance, keeping the same function list and ground truth; a significant drop in pass rate would confirm a genuine multi-turn effect.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that high single-turn tool-call accuracy does not translate to multi-turn generative performance—is supported primarily by Solar's drop from 83.6 on FunctionChat-Singlecall to 0.63 on Tool Call and 0.40 macro pass rate on FunctionChat-Dialog (Tables 3 and 4). The paper attributes this drop to the single-turn versus multi-turn distinction (Section 6.1). However, the two subsets are not matched: they use different functions, different items, different tool-list compositions, and the dialog subset adds conversation history and other output types. No calibration or item-difficulty model is presented. Thus the observed difference is confounded by content and difficulty. If the dialog Tool Call items are inherently harder—e.g., requiring reasoning over history to extract arguments—then Solar's low score reflects item difficulty, not the multi-turn format per se. The fact that GPT-4o and GPT-4-turbo score higher on dialog Tool Call (0.94 and 0.96) than on single-call (87.6 and 89.6) further suggests the dialog items are not uniformly harder; rather, there is a model-by-difficulty interaction. Without controlling for item content, the paper's causal statement that 'the performance of generating Tool Call type outputs is influenced by whether the input environment is single-turn or multi-turn' is unsupported. The dataset and evaluation methodology are still useful, but the headline empirical claim rests on an uncontrolled comparison.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces FunctionChat-Bench, a Korean-language benchmark for evaluating language models' generative capabilities in tool-use dialogs. It defines four output types (Tool Call, Answer Completion, Slot Question, Relevance Detection), constructs a single-turn dataset of 500 items and a multi-turn dialog dataset of 200 model-generation turns, and evaluates eight function-calling models. Evaluation uses gpt-4-0125-preview as an automated judge with human correction of its decisions; the dataset and evaluation code are publicly released. The headline finding is that high single-turn tool-call accuracy does not necessarily transfer to multi-turn generative performance, exemplified by Solar's 83.6 Singlecall average versus a 0.40 macro pass rate on FunctionChat-Dialog.","tokens_in":11744,"tokens_out":5443,"duration_ms":50221,"significance":"If the central claim is substantiated, this benchmark would be a useful resource for the community, broadening function-calling evaluation beyond exact JSON matching to include conversational outputs. The paper's strengths include the public release of the dataset and scripts, transparent reporting of LLM-judge error rates (Table 5, 5.5% FP+FN), a human-verified correction stage, a clear four-way taxonomy of outputs, and a detailed qualitative error analysis with concrete examples. The Korean-language focus addresses an under-resourced setting. However, the headline empirical claim currently rests on an uncontrolled comparison between two different item sets, and the statistical evidence is thinner than the text suggests; the paper needs either a controlled comparison or a substantially hedged interpretation.","major_comments":[{"comment":"The central claim that single-turn tool-call accuracy does not transfer to multi-turn environments is supported by comparing the Singlecall and Dialog subsets, but these subsets are not matched for item difficulty. They differ in functions, tool-list composition, conversation history, and the presence of other output types; no calibration, item-difficulty model, or matched-pair design is provided. The observed model-by-difficulty interaction (e.g., GPT-4o and GPT-4-turbo achieve higher Dialog Tool Call pass rates of 0.94 and 0.96 than their Singlecall averages of 87.6% and 89.6%) suggests that the dialog items are not uniformly harder, so the difference cannot be attributed cleanly to the single-turn versus multi-turn distinction. Please either add an item-difficulty control (e.g., evaluating the same functions and queries in both protocols, or reporting per-item difficulty annotations) or explicitly reframe the findings as a description of this dataset rather than a general causal claim about the multi-turn environment.","section":"6.1, Tables 3 and 4"},{"comment":"No confidence intervals or significance tests are reported for any pass rate. With only 70 Tool Call, 36 Slot Question, and 23 Relevance Detection dialog items, the rates have substantial sampling error; for example, Solar's 0.08 Slot Question rate (3/36) has a 95% Wilson interval of approximately [0.03, 0.21], and several inter-model differences fall within this noise. The text uses 'significantly' (Section 6.1) and compares models without accounting for this uncertainty. Reporting confidence intervals, exact tests (e.g., Fisher's exact or bootstrap), or a correction for multiple comparisons is needed to support the ranking claims.","section":"5, Tables 3 and 4"},{"comment":"The human-judge correction stage is a strength, but Table 5 reports that 5.5% of all LLM decisions were judged incorrect and adjusted. The reported pass rates are point estimates based on the corrected labels, with no sensitivity analysis. Since the corrections are partly driven by the authors' rubric interpretations, the headline comparisons should be accompanied by the original LLM-based rates or a discussion of how the corrections affect the final numbers.","section":"5, Appendix C"}],"minor_comments":[{"comment":"The abstract and Section 1 state that the benchmark comprises 700 evaluation items; the dialog subset's 200 items are model-generation turns, which is fine, but the sentence 'All dialogs contain 70 Tool Calls' in Section 3.2 is ambiguous and should clarify that the 70 Tool Calls are across the whole dialog set, not within each dialog.","section":"Abstract, Section 3.2"},{"comment":"The table title 'Pass Count' is misleading because the last column is an average (AVG) and the per-condition numbers are counts out of 100 items; please specify the denominator and label the columns more precisely (e.g., 'Pass count / 100').","section":"Table 3"},{"comment":"There is a recurring typo 'Aacceptable arguments' in the Tool Call rubric; it should be 'Acceptable arguments'.","section":"Appendix B"},{"comment":"The sentence 'These format refers to Zhou et al. (2023)' attributes the evaluation-prompt template to the LIMA paper, but the LLM-as-judge template with this structure is from Zheng et al. (2023), which is already cited in Section 4. Please verify the citation.","section":"Appendix B"},{"comment":"The classification diagram shows two AI outputs labeled 'Relevance Detection' that appear to belong to different conversation states; clarifying the dialogue flow (e.g., with numbered turns) would make the taxonomy easier to follow.","section":"Figure 1"}],"recommendation":"major_revision","confidential_remarks":"The benchmark and code release are valuable and the paper is likely publishable after the comparison is made rigorous or appropriately hedged. The central claim currently overreaches the evidence; the authors should either add a controlled comparison or substantially temper the abstract and Section 6.1. The small item counts and absence of uncertainty quantification are also worth raising in a revision request."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read, mostly. The paper contributes a public Korean-language function-calling benchmark with a sensible four-type taxonomy (Tool Call, Answer Completion, Slot Question, Relevance Detection), 700 items, an LLM-as-judge pipeline, and human correction of judge errors. The error analysis is genuinely informative: the Korean-specific numeric mistakes, fabricated arguments, and relevance-detection failures are the kind of detail practitioners need. They also ship code and data, so the work is reproducible.\n\nThe problem is the central empirical claim. The paper argues that high single-turn tool-call accuracy does not translate to multi-turn performance, pointing to Solar's 83.6% single-call score versus 0.63 pass rate on dialog Tool Call and 0.08 on Slot Question. But the two datasets are not matched: different functions, different items, different system prompts, different tool-list lengths. No item-difficulty calibration is offered. GPT-4o and GPT-4-turbo actually score slightly higher on dialog Tool Call (0.94, 0.96) than on single-call (87.6, 89.6), which shows the dialog items are not universally harder. There's a model-by-condition interaction, and the observed drop for Solar could reflect those specific items rather than single-turn versus multi-turn per se. The statement that Tool Call performance is 'influenced by whether the input environment is single-turn or multi-turn' is too strong for these data.\n\nThat said, the qualitative error analysis stands on its own. The slot-question hallucinations and relevance-detection failures are real behaviors that do not depend on the confounded comparison. The benchmark itself is a solid contribution for Korean tool-use evaluation, and the field needs more non-English test sets.\n\nOther soft spots: the dialog set is small (200 turns), no confidence intervals or significance tests, and the authors' own Limitations section concedes the dataset is narrow and relatively easy. None of this disqualifies the paper, but the headline should be framed as an observation with a call for more controlled follow-up rather than a demonstrated causal effect.\n\nRecommendation: yes, send this to peer review. A serious referee can push on the comparison design and the claim. The benchmark and error taxonomy deserve to be in the literature.","headline":"Worth a read: solid Korean tool-use benchmark and useful error taxonomy, but the headline single-turn vs multi-turn claim is confounded by unmatched item sets.","tokens_in":12290,"tokens_out":2158,"would_cite":true,"duration_ms":21717,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"High single-turn tool-call accuracy does not transfer to multi-turn tool-use dialogs, and FunctionChat-Bench, a 700-item Korean benchmark, demonstrates the gap","keywords":["function calling","tool-use dialogs","multi-turn dialog evaluation","LLM-as-a-judge","Korean language models","benchmark","slot filling","relevance detection"],"falsifier":"Use a matched-pair benchmark in which every user request appears once as a standalone utterance and once embedded in a multi-turn dialog with identical available functions; if dialog-context tool-call pass rates equal single-turn pass rates on the matched items, the paper's central contrast would be refuted.","tokens_in":11295,"feed_emoji":"💬","tokens_out":10818,"duration_ms":97369,"temperature":0.7,"pith_summary":"This paper aims to establish that a language model's single-turn tool-call accuracy is not a reliable guide to how well it will converse when tools are involved. It introduces FunctionChat-Bench, a 700-item Korean evaluation set whose 200 dialog turns are classified into four output types: Tool Call, Answer Completion, Slot Question, and Relevance Detection. An automated judge based on a large language model, using per-type rubrics and human review of its decisions, scores eight function-calling models on both a single-call subset and a dialog subset. A model that scored 83.6 on the single-call subset dropped to a 0.40 macro dialog pass rate (the average of the four per-type pass rates), with 0.08 on slot-question turns. If the paper is right, assistant quality in tool-use dialogs must be measured on conversational turns, not just on correctly formatted tool calls.","feed_headline":"Tool-call accuracy doesn't predict real assistant quality","feed_subtitle":"Korean dialog tests show top single-turn tool-call models failing on slot questions and relevance checks","key_machinery":"The central object is FunctionChat-Bench itself, made of two subsets: FunctionChat-Singlecall, 500 one-utterance requests that must trigger exactly one tool call, with tool-list lengths of 1, 4, or 8 and random or similar function sets; and FunctionChat-Dialog, 200 model-generated turns across 45 Korean dialogs, each turn labeled as Tool Call, Answer Completion, Slot Question, or Relevance Detection. The evaluation mechanism is an LLM judge: a large language model grades each turn pass or fail against a per-type rubric, a ground-truth response, and lists of acceptable arguments, and human reviewers correct the judge's false positives and false negatives (about 5.5% of decisions in both subsets). This design is what allows the paper to attribute performance differences not just to function selection and argument extraction but to each conversational capability separately.","core_discovery":"The central claim is that high single-turn function-calling accuracy does not transfer to multi-turn tool-use dialogs. The paper demonstrates this by comparing eight models on two subsets of the same benchmark: 500 isolated requests that each require one tool call, and 200 turns inside 45 multi-turn Korean dialogs that require one of four behaviors. The dialog pass rates do not track the single-call pass rates: for instance, a model that aced the single-call set at 83.6 fell to 0.40 macro in dialogs, with only 0.08 on slot-question turns, while another model with lower single-call accuracy showed comparatively strong conversational behavior. The authors conclude that function-calling competence has four separable components, and that evaluations that test only the tool-call component will overstate the quality of an assistant that must also relay tool results, ask for missing slot values, and judge whether a tool is relevant at all.","pith_inferences":["A matched-pair version of FunctionChat-Bench, where identical requests appear alone and inside a dialog with the same functions, would isolate the multi-turn environment's effect from item difficulty; the current subsets differ in functions, items, and system prompts, so part of the observed gap could be hardness rather than context.","If the four-type taxonomy generalizes, assistant fine-tuning could target slot-question and relevance-detection behavior directly, since these are exactly where models hallucinate tool calls or invent arguments.","The single-call-to-dialog gap suggests that for Korean and other lower-resource languages, conversational turn handling, not tool-call formatting, will be the binding constraint for deploying function-calling assistants."],"forward_implications":["Single-turn tool-call pass rates cannot be treated as a proxy for deployed assistant quality in multi-turn settings; dialog-based evaluation is the relevant test.","The four-type taxonomy produces a per-model error profile, since models fail differently on slot questions, answer completion, relevance detection, and tool calls, and aggregate scores hide which capability needs work.","Similarity between available functions lowered pass rates more than increasing the list from four to eight functions, so confusable function sets matter more than list size for selection difficulty.","Korean-language evaluation surfaces failures that English benchmarks would miss, such as misreading Korean number words, so multilingual tool-use evaluation can change model rankings.","The benchmark's scope is single function calls per turn, without parallel calls or multi-step planning, so the multi-turn finding applies to this basic tool-use setting."],"supporting_citations":[{"why":"ToolTalk: prior multi-turn tool-usage evaluation that the dialog subset builds on and contrasts with.","marker":"(Farn and Shin, 2023)"},{"why":"APIBench/Gorilla: canonical single-turn function-calling evaluation, the main target of the paper's insufficiency argument.","marker":"(Patil et al., 2023)"},{"why":"ToolBench: large-scale real-API tool-use benchmark representing the single-turn-oriented approach the paper extends.","marker":"(Qin et al., 2023)"},{"why":"BFCL: earlier function-calling leaderboard that already included relevance detection and tool-call accuracy, giving the paper a reference point for scope.","marker":"(Yan et al., 2024)"},{"why":"Supplies the prompt format for LLM-as-a-judge used in the evaluation program.","marker":"(Zhou et al., 2023)"},{"why":"Supplies the LLM-as-a-judge methodology and the MT-Bench/Chatbot-Arena basis for grading model outputs.","marker":"(Zheng et al., 2023)"}],"fun_headline_variants":["Tool-call leaders lag in multi-turn dialog tests","Why high single-call scores don't make a good assistant","Function calling is four skills, not one — new benchmark shows","Korean tool-use dialogs unmask hidden assistant gaps","Don't trust single-turn tool scores for real-world chat"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's main comparison assumes the single-call and dialog subsets are comparable in per-item difficulty, so the lower dialog pass rates can be attributed to the multi-turn environment rather than to the dialog items simply being harder.","fun_headline_variants_meta":{"raw":{"variants":["Tool-call leaders lag in multi-turn dialog tests","Why high single-call scores don't make a good assistant","Function calling is four skills, not one — new benchmark shows","Korean tool-use dialogs unmask hidden assistant gaps","Don't trust single-turn tool scores for real-world chat"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000315,"raw_usage":{"total_tokens":1736,"prompt_tokens":848,"completion_tokens":888,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":464,"completion_tokens_details":{"reasoning_tokens":807}},"tokens_in":464,"tokens_out":888,"duration_ms":9233,"temperature":1.0,"reasoning_tokens":807,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T15:34:25.783384+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Use a matched-pair benchmark in which every user request appears once as a standalone utterance and once embedded in a multi-turn dialog with identical available functions; if dialog-context tool-call pass rates equal single-turn pass rates on the matched items, the paper's central contrast would be refuted.","supporting_citations":[],"review_version":1}