FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs

Byeongil Ko; Daniel Lee; Gaeun Seo; Myeongcheol Shin; Shinbok Lee; Sunghee Jung

REVIEW 3 major objections 5 minor 11 references

FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs

T0 review · 3 major / 5 minor · reviewed 2026-08-12 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read High single-turn tool-call accuracy does not transfer to multi-turn tool-use dialogs, and FunctionChat-Bench, a 700-item Korean benchmark, demonstrates the gap

desk verdict Worth a read: solid Korean tool-use benchmark and useful error taxonomy, but the headline single-turn vs multi-turn claim is confounded by unmatched item sets. read the letter →

arxiv 2411.14054 v1 pith:N3MUIJFJ submitted 2024-11-21 cs.CL cs.AI

Shinbok Lee , Gaeun Seo , Daniel Lee , Byeongil Ko , Sunghee Jung , Myeongcheol Shin This is my paper · ORCID

classification cs.CLcs.AI

keywords functioncallingtool-usedialogsmulti-turndialogevaluationLLM-as-a-judgeKoreanlanguagemodelsbenchmarkslotfillingrelevancedetection

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper aims to establish that a language model's single-turn tool-call accuracy is not a reliable guide to how well it will converse when tools are involved. It introduces FunctionChat-Bench, a 700-item Korean evaluation set whose 200 dialog turns are classified into four output types: Tool Call, Answer Completion, Slot Question, and Relevance Detection. An automated judge based on a large language model, using per-type rubrics and human review of its decisions, scores eight function-calling models on both a single-call subset and a dialog subset. A model that scored 83.6 on the single-call subset dropped to a 0.40 macro dialog pass rate (the average of the four per-type pass rates), with 0.08 on slot-question turns. If the paper is right, assistant quality in tool-use dialogs must be measured on conversational turns, not just on correctly formatted tool calls.

What carries the argument

The central object is FunctionChat-Bench itself, made of two subsets: FunctionChat-Singlecall, 500 one-utterance requests that must trigger exactly one tool call, with tool-list lengths of 1, 4, or 8 and random or similar function sets; and FunctionChat-Dialog, 200 model-generated turns across 45 Korean dialogs, each turn labeled as Tool Call, Answer Completion, Slot Question, or Relevance Detection. The evaluation mechanism is an LLM judge: a large language model grades each turn pass or fail against a per-type rubric, a ground-truth response, and lists of acceptable arguments, and human reviewers correct the judge's false positives and false negatives (about 5.5% of decisions in both subsets). This design is what allows the paper to attribute performance differences not just to function selection and argument extraction but to each conversational capability separately.

What would settle it

Use a matched-pair benchmark in which every user request appears once as a standalone utterance and once embedded in a multi-turn dialog with identical available functions; if dialog-context tool-call pass rates equal single-turn pass rates on the matched items, the paper's central contrast would be refuted.

Watch

Extended reading notes

Core claim

The central claim is that high single-turn function-calling accuracy does not transfer to multi-turn tool-use dialogs. The paper demonstrates this by comparing eight models on two subsets of the same benchmark: 500 isolated requests that each require one tool call, and 200 turns inside 45 multi-turn Korean dialogs that require one of four behaviors. The dialog pass rates do not track the single-call pass rates: for instance, a model that aced the single-call set at 83.6 fell to 0.40 macro in dialogs, with only 0.08 on slot-question turns, while another model with lower single-call accuracy showed comparatively strong conversational behavior. The authors conclude that function-calling competence has four separable components, and that evaluations that test only the tool-call component will overstate the quality of an assistant that must also relay tool results, ask for missing slot values, and judge whether a tool is relevant at all.

Load-bearing premise

The paper's main comparison assumes the single-call and dialog subsets are comparable in per-item difficulty, so the lower dialog pass rates can be attributed to the multi-turn environment rather than to the dialog items simply being harder.

Editorial extensions

If this is right

Single-turn tool-call pass rates cannot be treated as a proxy for deployed assistant quality in multi-turn settings; dialog-based evaluation is the relevant test.
The four-type taxonomy produces a per-model error profile, since models fail differently on slot questions, answer completion, relevance detection, and tool calls, and aggregate scores hide which capability needs work.
Similarity between available functions lowered pass rates more than increasing the list from four to eight functions, so confusable function sets matter more than list size for selection difficulty.
Korean-language evaluation surfaces failures that English benchmarks would miss, such as misreading Korean number words, so multilingual tool-use evaluation can change model rankings.
The benchmark's scope is single function calls per turn, without parallel calls or multi-step planning, so the multi-turn finding applies to this basic tool-use setting.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

A matched-pair version of FunctionChat-Bench, where identical requests appear alone and inside a dialog with the same functions, would isolate the multi-turn environment's effect from item difficulty; the current subsets differ in functions, items, and system prompts, so part of the observed gap could be hardness rather than context.
If the four-type taxonomy generalizes, assistant fine-tuning could target slot-question and relevance-detection behavior directly, since these are exactly where models hallucinate tool calls or invent arguments.
The single-call-to-dialog gap suggests that for Korean and other lower-resource languages, conversational turn handling, not tool-call formatting, will be the binding constraint for deploying function-calling assistants.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Worth a read, mostly. The paper contributes a public Korean-language function-calling benchmark with a sensible four-type taxonomy (Tool Call, Answer Completion, Slot Question, Relevance Detection), 700 items, an LLM-as-judge pipeline, and human correction of judge errors. The error analysis is genuinely informative: the Korean-specific numeric mistakes, fabricated arguments, and relevance-detection failures are the kind of detail practitioners need. They also ship code and data, so the work is reproducible.

The problem is the central empirical claim. The paper argues that high single-turn tool-call accuracy does not translate to multi-turn performance, pointing to Solar's 83.6% single-call score versus 0.63 pass rate on dialog Tool Call and 0.08 on Slot Question. But the two datasets are not matched: different functions, different items, different system prompts, different tool-list lengths. No item-difficulty calibration is offered. GPT-4o and GPT-4-turbo actually score slightly higher on dialog Tool Call (0.94, 0.96) than on single-call (87.6, 89.6), which shows the dialog items are not universally harder. There's a model-by-condition interaction, and the observed drop for Solar could reflect those specific items rather than single-turn versus multi-turn per se. The statement that Tool Call performance is 'influenced by whether the input environment is single-turn or multi-turn' is too strong for these data.

That said, the qualitative error analysis stands on its own. The slot-question hallucinations and relevance-detection failures are real behaviors that do not depend on the confounded comparison. The benchmark itself is a solid contribution for Korean tool-use evaluation, and the field needs more non-English test sets.

Other soft spots: the dialog set is small (200 turns), no confidence intervals or significance tests, and the authors' own Limitations section concedes the dataset is narrow and relatively easy. None of this disqualifies the paper, but the headline should be framed as an observation with a call for more controlled follow-up rather than a demonstrated causal effect.

Recommendation: yes, send this to peer review. A serious referee can push on the comparison design and the claim. The benchmark and error taxonomy deserve to be in the literature.

Referee Report

3 major / 5 minor

Summary. The paper introduces FunctionChat-Bench, a Korean-language benchmark for evaluating language models' generative capabilities in tool-use dialogs. It defines four output types (Tool Call, Answer Completion, Slot Question, Relevance Detection), constructs a single-turn dataset of 500 items and a multi-turn dialog dataset of 200 model-generation turns, and evaluates eight function-calling models. Evaluation uses gpt-4-0125-preview as an automated judge with human correction of its decisions; the dataset and evaluation code are publicly released. The headline finding is that high single-turn tool-call accuracy does not necessarily transfer to multi-turn generative performance, exemplified by Solar's 83.6 Singlecall average versus a 0.40 macro pass rate on FunctionChat-Dialog.

Significance. If the central claim is substantiated, this benchmark would be a useful resource for the community, broadening function-calling evaluation beyond exact JSON matching to include conversational outputs. The paper's strengths include the public release of the dataset and scripts, transparent reporting of LLM-judge error rates (Table 5, 5.5% FP+FN), a human-verified correction stage, a clear four-way taxonomy of outputs, and a detailed qualitative error analysis with concrete examples. The Korean-language focus addresses an under-resourced setting. However, the headline empirical claim currently rests on an uncontrolled comparison between two different item sets, and the statistical evidence is thinner than the text suggests; the paper needs either a controlled comparison or a substantially hedged interpretation.

major comments (3)

[6.1, Tables 3 and 4] The central claim that single-turn tool-call accuracy does not transfer to multi-turn environments is supported by comparing the Singlecall and Dialog subsets, but these subsets are not matched for item difficulty. They differ in functions, tool-list composition, conversation history, and the presence of other output types; no calibration, item-difficulty model, or matched-pair design is provided. The observed model-by-difficulty interaction (e.g., GPT-4o and GPT-4-turbo achieve higher Dialog Tool Call pass rates of 0.94 and 0.96 than their Singlecall averages of 87.6% and 89.6%) suggests that the dialog items are not uniformly harder, so the difference cannot be attributed cleanly to the single-turn versus multi-turn distinction. Please either add an item-difficulty control (e.g., evaluating the same functions and queries in both protocols, or reporting per-item difficulty annotations) or explicitly reframe the findings as a description of this dataset rather than a general causal claim about the multi-turn environment.
[5, Tables 3 and 4] No confidence intervals or significance tests are reported for any pass rate. With only 70 Tool Call, 36 Slot Question, and 23 Relevance Detection dialog items, the rates have substantial sampling error; for example, Solar's 0.08 Slot Question rate (3/36) has a 95% Wilson interval of approximately [0.03, 0.21], and several inter-model differences fall within this noise. The text uses 'significantly' (Section 6.1) and compares models without accounting for this uncertainty. Reporting confidence intervals, exact tests (e.g., Fisher's exact or bootstrap), or a correction for multiple comparisons is needed to support the ranking claims.
[5, Appendix C] The human-judge correction stage is a strength, but Table 5 reports that 5.5% of all LLM decisions were judged incorrect and adjusted. The reported pass rates are point estimates based on the corrected labels, with no sensitivity analysis. Since the corrections are partly driven by the authors' rubric interpretations, the headline comparisons should be accompanied by the original LLM-based rates or a discussion of how the corrections affect the final numbers.

minor comments (5)

[Abstract, Section 3.2] The abstract and Section 1 state that the benchmark comprises 700 evaluation items; the dialog subset's 200 items are model-generation turns, which is fine, but the sentence 'All dialogs contain 70 Tool Calls' in Section 3.2 is ambiguous and should clarify that the 70 Tool Calls are across the whole dialog set, not within each dialog.
[Table 3] The table title 'Pass Count' is misleading because the last column is an average (AVG) and the per-condition numbers are counts out of 100 items; please specify the denominator and label the columns more precisely (e.g., 'Pass count / 100').
[Appendix B] There is a recurring typo 'Aacceptable arguments' in the Tool Call rubric; it should be 'Acceptable arguments'.
[Appendix B] The sentence 'These format refers to Zhou et al. (2023)' attributes the evaluation-prompt template to the LIMA paper, but the LLM-as-judge template with this structure is from Zheng et al. (2023), which is already cited in Section 4. Please verify the citation.
[Figure 1] The classification diagram shows two AI outputs labeled 'Relevance Detection' that appear to belong to different conversation states; clarifying the dialogue flow (e.g., with numbered turns) would make the taxonomy easier to follow.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the benchmark claim is an empirical comparison, not a derivation from its own inputs.

full rationale

The paper's central claim, that high single-turn Tool Call accuracy does not necessarily translate to superior generative performance in multi-turn environments, is an empirical observation based on comparing model outputs on two human-authored datasets (Tables 3 and 4). The four output types are a taxonomy, not a derivation, and the rubric-based LLM judge is an external measurement instrument rather than a fitted parameter. Human correction of judge decisions (Appendix C) is measurement calibration, not a definitional shortcut. The paper contains no equations in which an output variable is defined in terms of the claimed result, no fitted parameter is renamed as a prediction, and no load-bearing self-citation or imported uniqueness theorem is invoked. A potential concern about unequal difficulty between the single-call and dialog subsets is a validity threat to the causal interpretation of observed differences, but it is not a form of circularity under the stated criteria.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on three domain assumptions: the meaningfulness of the four-type taxonomy, the validity of the LLM-judge-with-human-correction protocol, and the comparability of the Singlecall and Dialog subsets. The last is the most fragile and is listed as the weakest assumption. No fitted free parameters or invented entities are used.

assumptions (3)

domain assumption The four output types (Tool Call, Answer Completion, Slot Question, Relevance Detection) are an exhaustive and meaningful taxonomy of model outputs in tool-use dialogs.
The entire evaluation is built on this classification (Section 2.1). If the taxonomy is not appropriate, the rubrics and pass/fail criteria lose their foundation.
domain assumption An LLM judge (gpt-4-0125-preview) with human correction provides a valid assessment of whether model outputs meet the rubrics.
The evaluation method (Section 4) relies on LLM-as-judge. Human correction of 5.5 percent of decisions is acknowledged, but the corrected judgments are still treated as ground truth.
domain assumption Pass rates on the Singlecall and Dialog subsets are directly comparable, and differences reflect single-turn vs multi-turn capability rather than item difficulty.
The central claim (Section 6.1) compares scores across the two subsets. The subsets use different functions, items, and system prompts, and no calibration is provided for item difficulty.

how reviews work

0 comments

Cite this review

Pith. "Pith review of FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs." pith.science (2026). https://pith.science/paper/N3MUIJFJ

@misc{pith2026241114054,
  author       = {Pith},
  title        = {Pith review of: FunctionChat-Bench: Comprehensive Evaluation of Language Models' Generative Capabilities in Korean Tool-use Dialogs},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N3MUIJFJ}},
  note         = {Machine review of arXiv:2411.14054}
}

read the original abstract

This study investigates language models' generative capabilities in tool-use dialogs. We categorize the models' outputs in tool-use dialogs into four distinct types: Tool Call, Answer Completion, Slot Question, and Relevance Detection, which serve as aspects for evaluation. We introduce FunctionChat-Bench, comprising 700 evaluation items and automated assessment programs. Using this benchmark, we evaluate several language models that support function calling. Our findings indicate that while language models may exhibit high accuracy in single-turn Tool Call scenarios, this does not necessarily translate to superior generative performance in multi-turn environments. We argue that the capabilities required for function calling extend beyond generating tool call messages; they must also effectively generate conversational messages that engage the user.

Figures

Figures reproduced from arXiv: 2411.14054 by the authors.

**Figure 2.** A simplified sample of the content of the eval [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗

**Figure 3.** Examples of Errors in "Answer Completion" [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Figures from the paper (2 more)

**Figure 5.** Figure 5: An Example of Errors in "Relevance Detec [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]

**Figure 4.** Figure 4: Examples of Errors in "Slot Question" • Errors in Relevance Detection. In this part, language models fail to properly detect inputs that are unrelated to tool-use, and instead, it excessively associates all outputs with tool usage. This error is observed as a pattern …

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

11 extracted references · 7 linked inside Pith

[1]

Nicholas Farn and Richard Shin. 2023. https://arxiv.org/abs/2311.10775 Tooltalk: Evaluating tool-usage in a conversational setting . Preprint, arXiv:2311.10775

arXiv 2023
[2]

Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. https://arxiv.org/abs/2310.03128 Metatool benchmark for large language models: Deciding whether to use tools and which to use . Preprint, arXiv:2310.03128

arXiv 2024
[3]

Patil, Tianjun Zhang, Xin Wang, and Joseph E

Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. https://arxiv.org/abs/2305.15334 Gorilla: Large language model connected with massive apis . Preprint, arXiv:2305.15334

arXiv 2023
[4]

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://arxiv.org/abs/2307.16789 Toolllm: Facilitating large language models to master 16000+ real-world apis . Preprint, ...

arXiv 2023
[5]

Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. 2023. https://arxiv.org/abs/2306.06624 Restgpt: Connecting large language models with real-world restful apis . Preprint, arXiv:2306.06624

arXiv 2023
[6]

Patil, Ion Stoica, and Joseph E

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html

2024
[7]

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. https://arxiv.org/abs/2305.18752 Gpt4tools: Teaching large language model to use tools via self-instruction . Preprint, arXiv:2305.18752

arXiv 2023
[8]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf Judging llm-as-a-judge with mt-bench and chatbot ...

2023

Show all 11 references

[9]

Chunting Zhou, Pengfei Liu, Puxin Xu, Srini Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, Lili Yu, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://arxiv.org/abs/2305.11206 Lima: Less is more for alignment . Preprint, arXiv:2305.11206

2023 arXiv
[10]

online" 'onlinestring :=

ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...
[11]

write newline

" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 12, 2026 · model on record in the stance chip above.

[1] [1]

Nicholas Farn and Richard Shin. 2023. https://arxiv.org/abs/2311.10775 Tooltalk: Evaluating tool-usage in a conversational setting . Preprint, arXiv:2311.10775

arXiv 2023

[2] [2]

Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. https://arxiv.org/abs/2310.03128 Metatool benchmark for large language models: Deciding whether to use tools and which to use . Preprint, arXiv:2310.03128

arXiv 2024

[3] [3]

Patil, Tianjun Zhang, Xin Wang, and Joseph E

Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. https://arxiv.org/abs/2305.15334 Gorilla: Large language model connected with massive apis . Preprint, arXiv:2305.15334

arXiv 2023

[4] [4]

Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://arxiv.org/abs/2307.16789 Toolllm: Facilitating large language models to master 16000+ real-world apis . Preprint, ...

arXiv 2023

[5] [5]

Yifan Song, Weimin Xiong, Dawei Zhu, Wenhao Wu, Han Qian, Mingbo Song, Hailiang Huang, Cheng Li, Ke Wang, Rong Yao, Ye Tian, and Sujian Li. 2023. https://arxiv.org/abs/2306.06624 Restgpt: Connecting large language models with real-world restful apis . Preprint, arXiv:2306.06624

arXiv 2023

[6] [6]

Patil, Ion Stoica, and Joseph E

Fanjia Yan, Huanzhi Mao, Charlie Cheng-Jie Ji, Tianjun Zhang, Shishir G. Patil, Ion Stoica, and Joseph E. Gonzalez. 2024. Berkeley function calling leaderboard. https://gorilla.cs.berkeley.edu/blogs/8_berkeley_function_calling_leaderboard.html

2024

[7] [7]

Rui Yang, Lin Song, Yanwei Li, Sijie Zhao, Yixiao Ge, Xiu Li, and Ying Shan. 2023. https://arxiv.org/abs/2305.18752 Gpt4tools: Teaching large language model to use tools via self-instruction . Preprint, arXiv:2305.18752

arXiv 2023

[8] [8]

Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric Xing, Hao Zhang, Joseph E Gonzalez, and Ion Stoica. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/91f18a1287b398d378ef22505bf41832-Paper-Datasets_and_Benchmarks.pdf Judging llm-as-a-judge with mt-bench and chatbot ...

2023