Pith. sign in

REVIEW 4 major objections 5 minor 56 references

DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues

T0 review · 4 major / 5 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read DICE-BENCH shows that AI assistants fail when key facts are scattered across a multi-party group chat, and it introduces a metric, DICE-SCORE, to quantify that difficulty.

desk verdict A genuinely useful multi-party tool-calling benchmark whose headline dispersion-vs-length claim is undercut by a flawed metric, but which deserves review with revisions. read the letter →

arxiv 2506.22853 v2 pith:B6NTKMES submitted 2025-06-28 cs.CL cs.AI

Kyochul Jang , Donghyeon Lee , Kyusik Kim , Dongseok Heo , Taewhoo Lee , Woojeong Kim , Bongwon Suh This is my paper
classification cs.CLcs.AI
keywords functioncallingtoolusemulti-partydialoguemulti-roundbenchmarklargelanguagemodelsinformationdispersionDICE-SCORE
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

DICE-BENCH is a proposed benchmark for evaluating whether large language models can call the right function when the required information is spread across a multi-party, multi-round dialogue rather than contained in one instruction. The paper argues that existing function-calling benchmarks miss this real-world condition, and it introduces a metric, DICE-SCORE, to measure how dispersed tool-related details are. On 1,607 validated dialogue instances, the authors find that model accuracy falls steadily as DICE-SCORE rises, with even the strongest systems dropping from about 74% exact match in the simplest single-round setting to about 59% in four-round dialogues. The point of the work is that current assistants are not yet deployable in realistic group-chat scenarios, and that dispersion, not sheer input length, is the main obstacle.

What carries the argument

The load-bearing objects are DICE-SCORE and the Tool Graph. DICE-SCORE takes a vector S of per-utterance counts of tool-related items and a total item count T, and computes $$\text{DICE}(S,T) = \frac{\min(|S_{\ne 0}|, T) \cdot \sqrt{|S| \cdot T}}{\sum_{i \in S} \ln(1 + \$\alpha$ S_i)},$$ with $\alpha = e^2$; a higher score means the items needed for the function call are spread across more turns with less repetition. The Tool Graph is a directed graph over 124 tool functions whose edges encode that one tool's output or context is needed to call another, and sampling paths from this graph forces the generated dialogues to carry dependencies across rounds. A multi-agent system with distinct personas and an orchestrator produces the dialogues, and a three-stage validation pipeline (LLM-based scoring, rule checks, and human criteria) filters them to 1,607 instances.

What would settle it

Compare two sets of dialogues matched in total length and number of facts but differing in whether the facts are concentrated in one utterance or spread across turns; the paper's claim predicts a large accuracy gap, and a null result would falsify it.

Watch

Extended reading notes

Core claim

The paper's central claim is that DICE-BENCH is the first function-calling benchmark that combines multi-round structure, multiple parties, and explicit dependencies between tools across rounds, and that current large language models perform poorly on it. Using DICE-SCORE, defined as a measure of how spread out function-related items are over dialogue turns, the authors report a strong inverse correlation between the metric and both model performance and human performance: human exact-match accuracy falls from 80.5% on round-1 dialogues (DICE-SCORE 1.42) to 49.3% on round-4 dialogues (DICE-SCORE 5.36), with Pearson r ≈ −0.984, and the 19 evaluated models show the same trend. The paper concludes that retrieving sparse and fragmented tool-related details, rather than long-context handling, is what degrades function-calling performance.

Load-bearing premise

The paper assumes that DICE-SCORE measures how spread out the needed facts are, and not just how long the conversation is, so that the observed performance drop can be attributed to dispersion.

Editorial extensions

If this is right

  • If DICE-BENCH is an accurate testbed, then single-turn function-calling benchmarks overstate model readiness for real assistant use.
  • If DICE-SCORE captures what makes these tasks hard, benchmark designers can use it to label difficulty rather than relying on round count alone.
  • If dispersion rather than length is the bottleneck, then extending context windows will not by itself fix multi-party tool use; models need architectures or training that aggregate information across speakers.
  • If tool-specific models fine-tuned on single instructions generalize poorly, then future tool-use training should include multi-party dialogue data.
  • The reported human accuracy drop from 80.5% to 49.3% provides an upper-bound reference: even people find round-4 dialogues hard, so benchmark scores must be interpreted against human performance.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The dispersion effect DICE-SCORE measures is likely not unique to function calling: any task that requires aggregating facts from multiple speakers, such as meeting summarization or multi-source question answering, may show the same degradation, and DICE-SCORE could serve as a difficulty probe there.
  • A testable extension is to use DICE-SCORE as a training signal, for instance by generating curricula that start with low-dispersion dialogues and progress to high-dispersion ones.
  • The paper's attribution of the performance drop to dispersion rather than length relies on the specific form of DICE-SCORE; an independent test could directly regress model error on utterance count while controlling for dispersion.
  • DICE-BENCH's human validation suggests the benchmark is not trivially solvable, so it could be adopted as a stress test for dialogue-state tracking components in assistant pipelines.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces DICE-BENCH, a benchmark for evaluating LLM function-calling in multi-round, multi-party dialogues, built through a tool graph and a multi-agent simulation pipeline, filtered by automated, rule-based, and human criteria into 1,607 instances. It also proposes DICE-SCORE (Eq. 1), a metric intended to quantify the dispersion of tool-related information across utterances, and reports evaluations of 19 LLMs showing that performance declines as DICE-SCORE increases. The paper argues that current models struggle specifically because critical information is scattered across rounds and speakers, rather than merely because dialogues are long.

Significance. If the claims hold, DICE-BENCH would be a valuable and genuinely novel resource: it is the first multi-round, multi-party function-calling benchmark, it ships public code and data, and its construction pipeline with persona-based multi-agent simulation and multi-stage human validation is a useful template. The proposed DICE-SCORE, if valid, would give the community a difficulty label for dispersion. However, the current validation of DICE-SCORE is not sufficient to support the paper's central explanatory claim, and the evaluation protocol has a known format-mismatch issue. The benchmark itself is likely salvageable and useful, but the metric and its validation need substantial revision.

major comments (4)
  1. [§3.4, Eq. (1)] The 'Scale Adjustment' term sqrt(|S|·T) does not normalize for dialogue length; it is monotonically increasing in |S|, the total number of utterances. If one appends extra utterances that mention no tool-related items, |S≠0|, T, and the denominator remain unchanged, but |S| grows, so DICE-SCORE increases even though information dispersion is identical. This contradicts the claim in §4.3.2 that DICE-SCORE 'effectively isolates information dispersion from utterance length.' The paper should either revise the formula so that length is truly controlled, or restrict the claim to what the metric actually measures. A concrete diagnostic would be to hold T and |S≠0| fixed while varying the number of zero-item utterances and show that the score is invariant.
  2. [Table 4 and Appendix B] The human-validation evidence for DICE-SCORE consists of only four round-level aggregates (accuracy and DICE-SCORE per round). Across these four rounds, both dialogue length and information dispersion increase together, so the reported correlation r ≈ -0.984 cannot separate the effect of dispersion from the effect of length. The paper needs per-instance human judgments, or at least a controlled comparison that fixes length while varying dispersion, to support the metric's validity. Without such evidence, the claim that DICE-SCORE captures difficulty beyond length is not established.
  3. [Appendix A] The proof of the bound α ≥ e² is invalid as written. The derivative computation in §A.1 gives f′(c) = α³c², but the correct derivative of f(c) = ln(1+αc) - 2αc/(1+αc) is f′(c) = α(αc-1)/(1+αc)². In addition, the base case in §A.2 invokes the inequality ln(1+x) > 2x/(1+x) for x ≥ e², which is exactly the inequality being proved. Thus the monotonicity property of DICE-SCORE as the number of distinct tools grows is not proven by this appendix. The proof should be corrected or the claim should be verified numerically and stated as an empirical property.
  4. [§4.2 and Limitations] The evaluation uses only Exact Match (EM), and the Limitations section acknowledges that models sometimes produce outputs that fail the required JSON format, causing correct semantic content to be scored as wrong. Because format-compliance failures may become more frequent in longer and more complex rounds, the observed performance decline with round number could be partially an artifact of strict EM scoring. The paper should report a lenient scoring variant, such as parsing the best-effort JSON or matching after format normalization, and show that the main performance trends persist.
minor comments (5)
  1. [Figure 3 caption] The caption says 'Lower DICE-SCORE indicates that the input instruction is more challenging,' but the text and Figure 4 consistently treat higher DICE-SCORE as more challenging; the caption should be corrected.
  2. [§4.3.2] The sentence 'because the numerator of DICE-SCORE employs a logarithmic scale' is inaccurate: the logarithmic term appears in the denominator, not the numerator. The argument should be reworded.
  3. [§2.2] The text says academic research categorizes conversations into six types, but seven categories are listed (Persuasion, Inquiry, Discovery, Negotiation, Information-Seeking, Deliberation, and Eristic). The count should be reconciled.
  4. [Appendix K heading] The heading 'Dialgue Type' contains a typo and should read 'Dialogue Type.'
  5. [§4.1 and Limitations] The model selection says all evaluated models support at least an 8k context window, but the Limitations section mentions tool-based models with roughly 4k token limits that could not be tested. This is not contradictory if those models were excluded from the 19, but the wording should be clarified.

Circularity Check

1 steps flagged · score 3.0 of 10

Appendix A's proof of the α bound assumes the very inequality it sets out to prove; the benchmark construction and model/human correlations are otherwise independent of the metric.

  1. other [Appendix A, Section A.2 Base Case Verification]
    "Using the property ln(1+x)>2x/(1+x) for x≥e^2, we confirm that f(1)>0."

    Appendix A aims to prove inequality (2), ln(1+αc)>2αc/(1+αc) for all c≥1. Setting c=1 gives exactly ln(1+α)>2α/(1+α), i.e., the quoted property with x=α. The base case therefore assumes the theorem it is supposed to establish. The preceding derivative analysis does not rescue the proof because the computation is algebraically wrong: the true derivative is f'(c)=α(αc-1)/(1+αc)^2, not α^3 c^2. With the circular base case and incorrect derivative, the claimed guarantee that α=e^2 makes DICE-SCORE strictly increase with the number of distinct tools under balanced dispersal and repetition is not actually derived from independent premises.

full rationale

The central contribution of the paper, DICE-BENCH, is constructed through a tool graph, multi-agent dialogue simulation, and a three-stage validation pipeline; none of these steps is defined in terms of DICE-SCORE, and the benchmark instances are not selected by the metric. The model evaluations are genuine external measurements, and the human-performance correlation, though based on only four round-level aggregates, uses accuracy values that are external to the formula. The length/dispersion confound in Eq. (1) — where sqrt(|S|*T) makes the score grow with total utterance count — is a serious validity threat to the claim that DICE-SCORE 'effectively isolates information dispersion from utterance length,' but that is a correctness problem, not a circular reduction: the model and human accuracies are not derived from the metric. The one true circular step is in Appendix A, where the base case invokes the very inequality being proved; this is load-bearing only for the auxiliary monotonicity property of DICE-SCORE, not for the benchmark's independent content. Self-citations (e.g., Lee et al. 2024, which shares authors) are used only to distinguish the proposed metric from prior long-context coverage work, not to justify the central claims. Overall, the paper has one genuine circular proof step, but the benchmark and its main empirical findings remain independent, so the score is low rather than high.

Assumptions & free parameters 3 free parameters · 5 assumptions · 1 invented entities

The central claims depend on hand-chosen metric parameters and thresholds, the assumed validity of dialogue typology and tool collections, and the use of LLM-as-judge for filtering and counting. No new physical or ontological entities are introduced; the DICE-SCORE metric is an invented instrument whose independent evidence is currently weak.

free parameters (3)
  • alpha (α) = e^2 ≈ 7.389
    DICE-SCORE repetition penalty constant, set by hand to satisfy the inequality in Appendix A; not fitted to performance data.
  • G-Eval average threshold = 4.0 on a 5-point Likert scale
    Dialogues with average G-Eval score below 4.0 are removed; cutoff is arbitrary and no sensitivity analysis is reported.
  • Criteria-based filtering threshold = 10 out of 15
    Human criteria-based filtering requires a score of at least 10/15; cutoff is arbitrary.
assumptions (5)
  • standard math Calculus facts used in Appendix A to bound α
    The proof contains an algebra error, though the inequality itself is valid for α≥e^2.
  • domain assumption Walton and Krabbe's dialogue typology can be condensed into three types that cover realistic multi-party function-calling dialogues
    Section 3.1 Scenario Configuration; no empirical justification is provided for the condensation.
  • domain assumption TaskBench and ToolEyes tool collections represent realistic everyday-life APIs
    Section 3.1 Tool Graph Construction; tools are filtered by authors and LLM, but the underlying collection is assumed adequate.
  • domain assumption GPT-4o-based G-Eval scores and GPT-4o-mini item counting are reliable proxies for human judgments
    Sections 3.2 and 3.4; model bias is acknowledged but not independently validated on this dataset.
  • ad hoc to paper The DICE-SCORE formula with α=e^2 captures task difficulty
    The metric is proposed without derivation from first principles; its validation is correlational only.
invented entities (1)
  • DICE-SCORE metric
    purpose: Quantify the dispersion of tool-related information across dialogue turns and label task difficulty.
    The metric is defined by a handcrafted formula with a hand-set constant α=e^2; its validity rests on a 4-point aggregate human correlation and model performance correlation on the authors' own benchmark, not on an external falsifiable prediction.

how reviews work

0 comments
Cite this review

Pith. "Pith review of DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues." pith.science (2026). https://pith.science/paper/B6NTKMES

@misc{pith2026250622853,
  author       = {Pith},
  title        = {Pith review of: DICE-BENCH: Evaluating the Tool-Use Capabilities of Large Language Models in Multi-Round, Multi-Party Dialogues},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/B6NTKMES}},
  note         = {Machine review of arXiv:2506.22853}
}
read the original abstract

Existing function-calling benchmarks focus on single-turn interactions. However, they overlook the complexity of real-world scenarios. To quantify how existing benchmarks address practical applications, we introduce DICE-SCORE, a metric that evaluates the dispersion of tool-related information such as function name and parameter values throughout the dialogue. Analyzing existing benchmarks through DICE-SCORE reveals notably low scores, highlighting the need for more realistic scenarios. To address this gap, we present DICE-BENCH, a framework that constructs practical function-calling datasets by synthesizing conversations through a tool graph that maintains dependencies across rounds and a multi-agent system with distinct personas to enhance dialogue naturalness. The final dataset comprises 1,607 high-DICE-SCORE instances. Our experiments on 19 LLMs with DICE-BENCH show that significant advances are still required before such models can be deployed effectively in real-world settings. Our code and data are all publicly available: https://snuhcc.github.io/DICE-Bench/.

Figures

Figures reproduced from arXiv: 2506.22853 by the authors.

Figure 1
Figure 1. Illustration of a Single-Round, Four-Party Dialogue in DICE-BENCH. LLMs must identify function-related information from multi-party dialogue. Relevant values in the dialogue are color-coded to match their function call components. Despite advancements, most function-calling benchmarks assume all API parameters are present in a single user utterance, overlooking real-world group chat scenarios (Chen et al., 2024; Zhu… view at source ↗
Figure 2
Figure 2. DICE-BENCH data-generation pipeline. (1) In the Tool Graph Construction phase, we build a tool graph from tool collections. (2) In the Scenario Configuration step, we sample tool chains and configure dialogue types, personas, and the target number of rounds. (3) In the Dialogue Simulation phase, we iteratively generate parameter values for each tool and simulate corresponding multi-party dialogues across N rounds. 2… view at source ↗
Figure 3
Figure 3. Inverse Correlation between DICE￾SCORE and Model Performance. Lower DICE￾SCORE indicates that the input instruction is more chal￾lenging, suggesting that the LLM is capable of handling complex scenarios. coverage by quantifying how much of the input context is necessary for answering queries, none have proposed a metric that explicitly captures how dispersed or fragmented these details are within a dialogue for func… view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: EM Performance Scores vs DICE￾SCORE. DICE-SCORE has been inverted to highlight its correlation with LLMs performance. The "DICE" in the legend represents the DICE-SCORE, and the purple-shaded region indicates ±1 standard deviation of DICE-SCORE. Thus, a high DICE-SCORE…
Figure 5
Figure 5. Figure 5: Multi-round Dialogue Example. User utterances and instructions are shown; highlights mark function￾call arguments [PITH_FULL_IMAGE:figures/full_fig_p015_5.png]
Figure 6
Figure 6. Figure 6: Tool Graph of DICE-BENCH. The graph comprises 124 nodes and 270 edges representing the dependencies among tool functions [PITH_FULL_IMAGE:figures/full_fig_p023_6.png]
Figure 7
Figure 7. Figure 7: EM Scores (Log Scale, Linear Scale, and Average Bar Chart) are presented horizontally for each category, Round, Party, and Dialogue Type, which are arranged vertically [PITH_FULL_IMAGE:figures/full_fig_p024_7.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

56 extracted references · 17 canonical work pages

  1. [1]

    Ibrahim Abdelaziz, Kinjal Basu, Mayank Agarwal, Sadhana Kumaravel, Matthew Stallone, Rameswar Panda, Yara Rizk, G. P. Bhargav, Maxwell Crouse, Chulaka Gunasekara, Shajith Ikbal, Sachin Joshi, Hima Karanam, Vineet Kumar, Asim Munawar, Sumit Neelam, Dinesh Raghu, Udit Sharma, Adriana Meza Soria, Dheeraj Sreedhar, Praveen Venkateswaran, Merve Unuvar, David C...

    arXiv 2024

  2. [2]

    Hewett, Mojan Javaheripi, Piero Kauffmann, James R

    Marah Abdin, Jyoti Aneja, Harkirat Behl, S \'e bastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli...

    arXiv 2024

  3. [3]

    Emre Can Acikgoz, Jeremiah Greer, Akul Datta, Ze Yang, William Zeng, Oussama Elachqar, Emmanouil Koukoumidis, Dilek Hakkani-T \"u r , and Gokhan Tur. 2025. https://arxiv.org/abs/2502.08820 Can a Single Model Master Both Multi-turn Conversations and Tool Use ? CALM : A Unified Conversational Agentic Language Model . Preprint, arXiv:2502.08820

    arXiv 2025

  4. [4]

    Kinjal Basu, Ibrahim Abdelaziz, Subhajit Chaudhury, Soham Dan, Maxwell Crouse, Asim Munawar, Sadhana Kumaravel, Vinod Muthusamy, Pavan Kapanipathi, and Luis A. Lastras. 2024. https://arxiv.org/abs/2402.15491 API-BLEND : A Comprehensive Corpora for Training and Benchmarking API LLMs . Preprint, arXiv:2402.15491

    arXiv 2024

  5. [5]

    Douglas Biber, Bethany Gray, and Kornwipa Poonpon. 2011. Should We Use Characteristics of Conversation to Measure Grammatical Complexity in L2 Writing Development ? TESOL Quarterly, 45(1):5--35

    work page 2011

  6. [6]

    Giovanni Campagna, Silei Xu, Mehrad Moradshahi, Richard Socher, and Monica S. Lam. 2019. https://arxiv.org/abs/1904.09020 Genie: A Generator of Natural Language Semantic Parsers for Virtual Assistant Commands . In Proceedings of the 40th ACM SIGPLAN Conference on Programming Language Design and Implementation , pages 394--410

    work page Pith review arXiv 2019

  7. [7]

    Zehui Chen, Weihua Du, Wenwei Zhang, Kuikun Liu, Jiangning Liu, Miao Zheng, Jingming Zhuo, Songyang Zhang, Dahua Lin, Kai Chen, and Feng Zhao. 2024. https://arxiv.org/abs/2312.14033 T- Eval : Evaluating the Tool Utilization Capability of Large Language Models Step by Step . Preprint, arXiv:2312.14033

    arXiv 2024

  8. [8]

    Lutfi Eren Erdogan, Nicholas Lee, Siddharth Jha, Sehoon Kim, Ryan Tabrizi, Suhong Moon, Coleman Hooper, Gopala Anumanchipalli, Kurt Keutzer, and Amir Gholami. 2024. https://arxiv.org/abs/2409.00608 TinyAgent : Function Calling at the Edge . Preprint, arXiv:2409.00608

    arXiv 2024

Show all 56 references
  1. [9]

    Nicholas Farn and Richard Shin. 2023. https://arxiv.org/abs/2311.10775 ToolTalk : Evaluating Tool-Usage in a Conversational Setting . Preprint, arXiv:2311.10775

    2023 arXiv

  2. [10]

    Team GLM, Aohan Zeng, Bin Xu, Bowen Wang, Chenhui Zhang, Da Yin, Dan Zhang, Diego Rojas, Guanyu Feng, Hanlin Zhao, Hanyu Lai, Hao Yu, Hongning Wang, Jiadai Sun, Jiajie Zhang, Jiale Cheng, Jiayi Gui, Jie Tang, Jing Zhang, Jingyu Sun, Juanzi Li, Lei Zhao, Lindong Wu, Lucen Zhong...

    2024 arXiv

  3. [11]

    Omer Goldman, Alon Jacovi, Aviv Slobodkin, Aviya Maimon, Ido Dagan, and Reut Tsarfaty. 2024. https://arxiv.org/abs/2407.00402 Is It Really Long Context if All You Need Is Retrieval ? Towards Genuinely Difficult Long Context NLP . Preprint, arXiv:2407.00402

    2024 arXiv

  4. [12]

    Peiyuan Gong, Jiamian Li, and Jiaxin Mao. 2024. https://arxiv.org/abs/2402.06360 CoSearchAgent : A Lightweight Collaborative Search Agent with Large Language Models . Preprint, arXiv:2402.06360

    2024 arXiv

  5. [13]

    Yanchu Guan, Dong Wang, Zhixuan Chu, Shiyu Wang, Feiyue Ni, Ruihua Song, Longfei Li, Jinjie Gu, and Chenyi Zhuang. 2023. https://arxiv.org/abs/2312.06677 Intelligent Virtual Assistants with LLM-based Process Automation . Preprint, arXiv:2312.06677

    2023 arXiv

  6. [14]

    Yue Huang, Jiawen Shi, Yuan Li, Chenrui Fan, Siyuan Wu, Qihui Zhang, Yixin Liu, Pan Zhou, Yao Wan, Neil Zhenqiang Gong, and Lichao Sun. 2024. https://arxiv.org/abs/2310.03128 MetaTool Benchmark for Large Language Models : Deciding Whether to Use Tools and Which to Use . Prepri...

    2024 arXiv

  7. [15]

    Koji Inoue, Divesh Lala, Mikey Elmers, Keiko Ochi, and Tatsuya Kawahara. 2025. https://arxiv.org/abs/2501.16643 An LLM Benchmark for Addressee Recognition in Multi-modal Multi-party Dialogue . Preprint, arXiv:2501.16643

    2025 arXiv

  8. [16]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, L \'e lio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Tho...

    2023 arXiv

  9. [17]

    Daniel Jurafsky and James H. Martin. 2019. https://web.stanford.edu/ jurafsky/slp3/ Speech and language processing . Draft available online

    2019

  10. [18]

    Anjali Khurana, Hari Subramonyam, and Parmit K. Chilana. 2024. https://arxiv.org/abs/2402.08030 Why and When LLM-Based Assistants Can Go Wrong : Investigating the Effectiveness of Prompt-Based Interactions for Software Help-Seeking . In Proceedings of the 29th International Co...

    2024 arXiv

  11. [19]

    Woojeong Kim, Ashish Jagmohan, and Aditya Vempaty. 2024. https://arxiv.org/abs/2409.15523 SEAL : Suite for Evaluating API-use of LLMs . Preprint, arXiv:2409.15523

    2024 arXiv

  12. [20]

    Taewhoo Lee, Chanwoong Yoon, Kyochul Jang, Donghyeon Lee, Minju Song, Hyunjae Kim, and Jaewoo Kang. 2024. https://arxiv.org/abs/2410.16848 ETHIC : Evaluating Large Language Models on Long-Context Tasks with High Information Coverage . Preprint, arXiv:2410.16848

    2024 arXiv

  13. [21]

    Minghao Li, Yingxiu Zhao, Bowen Yu, Feifan Song, Hangyu Li, Haiyang Yu, Zhoujun Li, Fei Huang, and Yongbin Li. 2023. https://arxiv.org/abs/2304.08244 API-Bank : A Comprehensive Benchmark for Tool-Augmented LLMs . Preprint, arXiv:2304.08244

    2023 arXiv

  14. [22]

    Weiwen Liu, Xu Huang, Xingshan Zeng, Xinlong Hao, Shuai Yu, Dexun Li, Shuai Wang, Weinan Gan, Zhengying Liu, Yuanqing Yu, Zezhong Wang, Yuxian Wang, Wu Ning, Yutai Hou, Bin Wang, Chuhan Wu, Xinzhi Wang, Yong Liu, Yasheng Wang, Duyu Tang, Dandan Tu, Lifeng Shang, Xin Jiang, Rui...

    2024 arXiv

  15. [23]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. https://arxiv.org/abs/2303.16634 G- Eval : NLG Evaluation using GPT-4 with Better Human Alignment . Preprint, arXiv:2303.16634

    2023 arXiv

  16. [24]

    Manning and Hinrich Schütze

    Christopher D. Manning and Hinrich Schütze. 1999. Foundations of statistical natural language processing. Available online at https://nlp.stanford.edu/fsnlp/

    1999

  17. [25]

    OpenAI, Josh Achiam, Steven Adler, Sandhini Agarwal, et al. 2024. https://arxiv.org/abs/2303.08774 GPT-4 Technical Report . Preprint, arXiv:2303.08774

    2024 arXiv

  18. [26]

    Bernstein

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. Generative Agents : Interactive Simulacra of Human Behavior . In Proceedings of the 36th Annual ACM Symposium on User Interface Software and Technology , pages ...

    2023

  19. [27]

    Patil, Tianjun Zhang, Xin Wang, and Joseph E

    Shishir G. Patil, Tianjun Zhang, Xin Wang, and Joseph E. Gonzalez. 2023. https://arxiv.org/abs/2305.15334 Gorilla: Large Language Model Connected with Massive APIs . Preprint, arXiv:2305.15334

    2023 arXiv

  20. [28]

    Yujia Qin, Shengding Hu, Yankai Lin, Weize Chen, Ning Ding, Ganqu Cui, Zheni Zeng, Yufei Huang, Chaojun Xiao, Chi Han, Yi Ren Fung, Yusheng Su, Huadong Wang, Cheng Qian, Runchu Tian, Kunlun Zhu, Shihao Liang, Xingyu Shen, Bokai Xu, Zhen Zhang, Yining Ye, Bowen Li, Ziwei Tang, ...

    2024 arXiv

  21. [29]

    Yujia Qin, Shihao Liang, Yining Ye, Kunlun Zhu, Lan Yan, Yaxi Lu, Yankai Lin, Xin Cong, Xiangru Tang, Bill Qian, Sihan Zhao, Lauren Hong, Runchu Tian, Ruobing Xie, Jie Zhou, Mark Gerstein, Dahai Li, Zhiyuan Liu, and Maosong Sun. 2023. https://arxiv.org/abs/2307.16789 ToolLLM :...

    2023 arXiv

  22. [30]

    Changle Qu, Sunhao Dai, Xiaochi Wei, Hengyi Cai, Shuaiqiang Wang, Dawei Yin, Jun Xu, and Ji-Rong Wen. 2024. https://arxiv.org/abs/2405.17935 Tool Learning with Large Language Models : A Survey . Preprint, arXiv:2405.17935

    2024 arXiv

  23. [31]

    Qwen, An Yang, Baosong Yang, Beichen Zhang, Binyuan Hui, Bo Zheng, Bowen Yu, Chengyuan Li, Dayiheng Liu, Fei Huang, Haoran Wei, Huan Lin, Jian Yang, Jianhong Tu, Jianwei Zhang, Jianxin Yang, Jiaxi Yang, Jingren Zhou, Junyang Lin, Kai Dang, Keming Lu, Keqin Bao, Kexin Yang, Le ...

    2025 arXiv

  24. [32]

    Andrew Reece, Gus Cooney, Peter Bull, Christine Chung, Bryn Dawson, Casey Fitzpatrick, Tamara Glazer, Dean Knox, Alex Liebscher, and Sebastian Marin. 2023. The CANDOR corpus: Insights from a large multimodal dataset of naturalistic conversation. Science Advances, 9(13):eadf3197

    2023

  25. [33]

    LG AI Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Yountae Jung, Hyosang Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil...

    2024

  26. [34]

    Jonan Richards and Mairieli Wessel. 2025. https://arxiv.org/abs/2502.07956 Bridging HCI and AI Research for the Evaluation of Conversational SE Assistants . Preprint, arXiv:2502.07956

    2025 arXiv

  27. [35]

    Gerard Salton and Chris Buckley. 1988. Term-weighting approaches in automatic text retrieval. Technical report discussing downweighting via logarithmic scaling

    1988

  28. [36]

    Timo Schick, Jane Dwivedi-Yu , Roberto Dess \`i , Roberta Raileanu, Maria Lomeli, Luke Zettlemoyer, Nicola Cancedda, and Thomas Scialom. 2023. https://arxiv.org/abs/2302.04761 Toolformer: Language Models Can Teach Themselves to Use Tools . Preprint, arXiv:2302.04761

    2023 arXiv

  29. [37]

    Emre Sezgin. 2024. Redefining Virtual Assistants in Health Care : The Future With Large Language Models . Journal of Medical Internet Research, 26:e53225

    2024

  30. [38]

    Yongliang Shen, Kaitao Song, Xu Tan, Wenqi Zhang, Kan Ren, Siyu Yuan, Weiming Lu, Dongsheng Li, and Yueting Zhuang. 2024. https://arxiv.org/abs/2311.18760 TaskBench : Benchmarking Large Language Models for Task Automation . Preprint, arXiv:2311.18760

    2024 arXiv

  31. [39]

    Qiaoyu Tang, Ziliang Deng, Hongyu Lin, Xianpei Han, Qiao Liang, Boxi Cao, and Le Sun. 2023. https://arxiv.org/abs/2306.05301 ToolAlpaca : Generalized Tool Learning for Language Models with 3000 Simulated Cases . Preprint, arXiv:2306.05301

    2023 arXiv

  32. [40]

    Gemini Team, Rohan Anil, Sebastian Borgeaud, Jean-Baptiste Alayrac, et al. 2020. https://arxiv.org/abs/2005.14165 Gemini: A Family of Highly Capable Multimodal Models . Preprint, arXiv:2005.14165

    2020 arXiv

  33. [41]

    Nexusflow.ai team. 2023. https://nexusflow.ai/blogs/ravenv2 Nexusraven-v2: Surpassing gpt-4 for zero-shot function calling

    2023

  34. [42]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timoth \'e e Lacroix, Baptiste Rozi \`e re, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.1397...

    2023 arXiv

  35. [43]

    Minh Duc Vu, Han Wang, Zhuang Li, Jieshan Chen, Shengdong Zhao, Zhenchang Xing, and Chunyang Chen. 2024. https://arxiv.org/abs/2401.14268 GPTVoiceTasker : Advancing Multi-step Mobile Task Efficiency Through Dynamic Interface Exploration and Learning . In Proceedings of the 37t...

    2024 arXiv

  36. [44]

    Douglas Walton. 2010. Types of Dialogue and Burdens of Proof . In Computational Models of Argument , pages 13--24. IOS Press

    2010

  37. [45]

    Douglas Walton and Erik C. W. Krabbe. 1995. Commitment in Dialogue : Basic Concepts of Interpersonal Reasoning . SUNY Press

    1995

  38. [46]

    Jun Wang, Jiamu Zhou, Muning Wen, Xiaoyun Mo, Haoyu Zhang, Qiqiang Lin, Cheng Jin, Xihuai Wang, Weinan Zhang, Qiuying Peng, and Jun Wang. 2024 a . https://arxiv.org/abs/2412.16516 HammerBench : Fine-Grained Function-Calling Evaluation in Real Mobile Device Scenarios . Preprint...

    2024

  39. [47]

    Pei Wang, Yanan Wu, Zekun Wang, Jiaheng Liu, Xiaoshuai Song, Zhongyuan Peng, Ken Deng, Chenchen Zhang, Jiakai Wang, Junran Peng, Ge Zhang, Hangyu Guo, Zhaoxiang Zhang, Wenbo Su, and Bo Zheng. 2024 b . https://arxiv.org/abs/2410.11710 MTU-Bench : A Multi-granularity Tool-Use Be...

    2024 arXiv

  40. [48]

    Joseph Weizenbaum. 1966. ELIZA ---a computer program for the study of natural language communication between man and machine. Communications of the ACM, 9(1):36--45

    1966

  41. [49]

    Qiantong Xu, Fenglu Hong, Bo Li, Changran Hu, Zhengyu Chen, and Jian Zhang. 2023. https://arxiv.org/abs/2305.16504 On the Tool Manipulation Capability of Open-source Large Language Models . Preprint, arXiv:2305.16504

    2023 arXiv

  42. [50]

    Junjie Ye, Guanyu Li, Songyang Gao, Caishuang Huang, Yilong Wu, Sixian Li, Xiaoran Fan, Shihan Dou, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024 a . https://arxiv.org/abs/2401.00741 ToolEyes : Fine-Grained Evaluation for Tool Learning Capabilities of Large Language Models in Re...

    2024 arXiv

  43. [51]

    Junjie Ye, Yilong Wu, Songyang Gao, Caishuang Huang, Sixian Li, Guanyu Li, Xiaoran Fan, Qi Zhang, Tao Gui, and Xuanjing Huang. 2024 b . https://arxiv.org/abs/2401.08326 RoTBench : A Multi-Level Benchmark for Evaluating the Robustness of Large Language Models in Tool Learning ....

    2024 arXiv

  44. [52]

    Schweitzer, and Alison Wood Brooks

    Michael Yeomans, Maurice E. Schweitzer, and Alison Wood Brooks. 2022. The Conversational Circumplex : Identifying , prioritizing, and pursuing informational and relational motives in conversation. Current Opinion in Psychology, 44:293--302

    2022

  45. [53]

    Chen Zhang, Xinyi Dai, Yaxiong Wu, Qu Yang, Yasheng Wang, Ruiming Tang, and Yong Liu. 2025. https://arxiv.org/abs/2501.09959 A Survey on Multi-Turn Interaction Capabilities of Large Language Models . Preprint, arXiv:2501.09959

    2025 arXiv

  46. [54]

    Yuchen Zhuang, Yue Yu, Kuan Wang, Haotian Sun, and Chao Zhang. 2023. https://arxiv.org/abs/2306.13304 ToolQA : A Dataset for LLM Question Answering with External Tools . Preprint, arXiv:2306.13304

    2023 arXiv

  47. [55]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.