{"id":"73ad4d9d-fa1e-46b9-b218-814e20b57331","arxiv_id":"2506.22853","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark and metric show that large language models still struggle to call tools when the needed details are scattered across multi-party, multi-round group dialogues.","lead":"DICE-BENCH is a new benchmark that tests AI assistants on function-calling in group chats where key details are spread across multiple speakers and rounds. It also proposes DICE-SCORE, a metric for how scattered that information is, and shows that current language models lose accuracy as scattering increases.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DICE-SCORE conflates dialogue length with information dispersion, so the paper's central claim that performance drops due to dispersion rather than length is not established.","rationale":"The reader's weakest assumption is exactly the load-bearing issue. The paper's strongest claim is not just that the benchmark exists, but that DICE-SCORE explains why models degrade; if length is confounded, that explanation is unsupported. We agree with the reader's conditional verdict because the issue is fixable via per-instance analysis or controlled padding, not a fundamental invalidation of the dataset. Independent support: code and data are public, 1,607 instances are human-filtered, and the benchmark is a useful resource regardless. The Appendix proof error and the Figure 3 caption typo ('Lower DICE-SCORE indicates more challenging') are additional inconsistencies but do not change the verdict.","tokens_in":17986,"tokens_out":3985,"duration_ms":45602,"concrete_test":"Take a random sample of DICE-BENCH instances and create padded controls by inserting 3–5 irrelevant utterances (no tool-related items) before the final summary turn, keeping T and |S≠0| fixed. Compute DICE-SCORE and run the same 19 LLMs on original and padded inputs. If DICE-SCORE rises while EM is unchanged, the metric is confounded with length; if EM drops with padding, the observed decline may be a length effect, not dispersion. Either outcome would require revising the Sec. 4.3.2 claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Equation (1) defines DICE-SCORE as [min(|S≠0|, T) * sqrt(|S| * T)] / [sum ln(1 + α S_i)]. The factor sqrt(|S| * T) grows with the total number of utterances |S|, so adding irrelevant turns that contain no tool-related items increases DICE-SCORE without changing dispersion (|S≠0|, T, and the denominator all stay the same). The paper calls this a 'Scale Adjustment' that 'normalizes' the score, but it is a multiplicative term, not a normalization. The validation that DICE-SCORE 'effectively isolates information dispersion from utterance length' rests on Table 4, which contains only four round-level aggregates; rounds vary simultaneously in length and dispersion, so the r ≈ -0.984 correlation cannot separate the two. The Appendix A proof of the α ≥ e^2 bound is also not independent: the base case uses the very inequality it claims to prove, and the derivative computation is algebraically wrong (f'(c) = α^3 c^2 is incorrect). Because the central conclusion—that models fail because information is scattered rather than because inputs are longer—depends entirely on this metric, the metric's length confound is load-bearing. Note that the benchmark itself, with public code and data, can remain useful even if this analysis must be revised.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DICE-BENCH, a benchmark for evaluating LLM function-calling in multi-round, multi-party dialogues, built through a tool graph and a multi-agent simulation pipeline, filtered by automated, rule-based, and human criteria into 1,607 instances. It also proposes DICE-SCORE (Eq. 1), a metric intended to quantify the dispersion of tool-related information across utterances, and reports evaluations of 19 LLMs showing that performance declines as DICE-SCORE increases. The paper argues that current models struggle specifically because critical information is scattered across rounds and speakers, rather than merely because dialogues are long.","tokens_in":18251,"tokens_out":3717,"duration_ms":39951,"significance":"If the claims hold, DICE-BENCH would be a valuable and genuinely novel resource: it is the first multi-round, multi-party function-calling benchmark, it ships public code and data, and its construction pipeline with persona-based multi-agent simulation and multi-stage human validation is a useful template. The proposed DICE-SCORE, if valid, would give the community a difficulty label for dispersion. However, the current validation of DICE-SCORE is not sufficient to support the paper's central explanatory claim, and the evaluation protocol has a known format-mismatch issue. The benchmark itself is likely salvageable and useful, but the metric and its validation need substantial revision.","major_comments":[{"comment":"The 'Scale Adjustment' term sqrt(|S|·T) does not normalize for dialogue length; it is monotonically increasing in |S|, the total number of utterances. If one appends extra utterances that mention no tool-related items, |S≠0|, T, and the denominator remain unchanged, but |S| grows, so DICE-SCORE increases even though information dispersion is identical. This contradicts the claim in §4.3.2 that DICE-SCORE 'effectively isolates information dispersion from utterance length.' The paper should either revise the formula so that length is truly controlled, or restrict the claim to what the metric actually measures. A concrete diagnostic would be to hold T and |S≠0| fixed while varying the number of zero-item utterances and show that the score is invariant.","section":"§3.4, Eq. (1)"},{"comment":"The human-validation evidence for DICE-SCORE consists of only four round-level aggregates (accuracy and DICE-SCORE per round). Across these four rounds, both dialogue length and information dispersion increase together, so the reported correlation r ≈ -0.984 cannot separate the effect of dispersion from the effect of length. The paper needs per-instance human judgments, or at least a controlled comparison that fixes length while varying dispersion, to support the metric's validity. Without such evidence, the claim that DICE-SCORE captures difficulty beyond length is not established.","section":"Table 4 and Appendix B"},{"comment":"The proof of the bound α ≥ e² is invalid as written. The derivative computation in §A.1 gives f′(c) = α³c², but the correct derivative of f(c) = ln(1+αc) - 2αc/(1+αc) is f′(c) = α(αc-1)/(1+αc)². In addition, the base case in §A.2 invokes the inequality ln(1+x) > 2x/(1+x) for x ≥ e², which is exactly the inequality being proved. Thus the monotonicity property of DICE-SCORE as the number of distinct tools grows is not proven by this appendix. The proof should be corrected or the claim should be verified numerically and stated as an empirical property.","section":"Appendix A"},{"comment":"The evaluation uses only Exact Match (EM), and the Limitations section acknowledges that models sometimes produce outputs that fail the required JSON format, causing correct semantic content to be scored as wrong. Because format-compliance failures may become more frequent in longer and more complex rounds, the observed performance decline with round number could be partially an artifact of strict EM scoring. The paper should report a lenient scoring variant, such as parsing the best-effort JSON or matching after format normalization, and show that the main performance trends persist.","section":"§4.2 and Limitations"}],"minor_comments":[{"comment":"The caption says 'Lower DICE-SCORE indicates that the input instruction is more challenging,' but the text and Figure 4 consistently treat higher DICE-SCORE as more challenging; the caption should be corrected.","section":"Figure 3 caption"},{"comment":"The sentence 'because the numerator of DICE-SCORE employs a logarithmic scale' is inaccurate: the logarithmic term appears in the denominator, not the numerator. The argument should be reworded.","section":"§4.3.2"},{"comment":"The text says academic research categorizes conversations into six types, but seven categories are listed (Persuasion, Inquiry, Discovery, Negotiation, Information-Seeking, Deliberation, and Eristic). The count should be reconciled.","section":"§2.2"},{"comment":"The heading 'Dialgue Type' contains a typo and should read 'Dialogue Type.'","section":"Appendix K heading"},{"comment":"The model selection says all evaluated models support at least an 8k context window, but the Limitations section mentions tool-based models with roughly 4k token limits that could not be tested. This is not contradictory if those models were excluded from the 19, but the wording should be clarified.","section":"§4.1 and Limitations"}],"recommendation":"major_revision","confidential_remarks":"The benchmark resource is a solid contribution and the paper is worth pursuing, but the DICE-SCORE validation and the EM-only evaluation need real work. I would not reject outright, because the dataset and generation pipeline are useful even if the metric analysis is revised. Please encourage the authors to address the length confound and the proof issues rather than merely adding caveats."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: the benchmark is real, the metric is shaky, and the paper's main scientific claim—that performance drops because of information dispersion rather than dialogue length—is not supported. Still, I'd send it to review. The dataset is useful enough to warrant referee time, provided the analysis is fixed.\n\nWhat's genuinely new: DICE-BENCH is the first function-calling benchmark I know of that combines multi-party and multi-round dialogues. The data-generation pipeline (tool graph, multi-agent personas, three-stage validation) is more careful than most synthetic benchmark construction. The authors ship code and data, run 19 models, and include human evaluation. For that alone, this is a real contribution to tool-use evaluation.\n\nThe soft spots are concentrated in DICE-SCORE. Equation (1) multiplies by sqrt(|S|*T), where |S| is the total utterance count. Adding irrelevant turns increases the score without changing dispersion. That directly contradicts the paper's claim in Section 4.3.2 that the metric 'isolates information dispersion from utterance length.' It doesn't. The human validation in Table 4 uses only four round-level aggregates, so it cannot separate length from dispersion either. The Appendix A proof is also flawed: the derivative computation is algebraically wrong, and the base case assumes exactly the inequality it claims to prove. These are not fatal to the dataset, but they are fatal to the central explanation for why models degrade.\n\nThe good news is the fix is straightforward: control for length, validate the metric on more granular data, and repair the proof. The benchmark itself remains useful regardless of DICE-SCORE.\n\nWho this is for: anyone building or evaluating tool-calling models, especially for group-chat assistants. It deserves a serious referee, but with a request for substantial revision on the metric and the interpretation of the results.","headline":"A genuinely useful multi-party tool-calling benchmark whose headline dispersion-vs-length claim is undercut by a flawed metric, but which deserves review with revisions.","tokens_in":18805,"tokens_out":3199,"would_cite":true,"duration_ms":29945,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DICE-BENCH shows that AI assistants fail when key facts are scattered across a multi-party group chat, and it introduces a metric, DICE-SCORE, to quantify that difficulty.","keywords":["function calling","tool use","multi-party dialogue","multi-round dialogue","benchmark","large language models","information dispersion","DICE-SCORE"],"falsifier":"Compare two sets of dialogues matched in total length and number of facts but differing in whether the facts are concentrated in one utterance or spread across turns; the paper's claim predicts a large accuracy gap, and a null result would falsify it.","tokens_in":17785,"feed_emoji":"💬","tokens_out":5599,"duration_ms":87046,"temperature":0.7,"pith_summary":"DICE-BENCH is a proposed benchmark for evaluating whether large language models can call the right function when the required information is spread across a multi-party, multi-round dialogue rather than contained in one instruction. The paper argues that existing function-calling benchmarks miss this real-world condition, and it introduces a metric, DICE-SCORE, to measure how dispersed tool-related details are. On 1,607 validated dialogue instances, the authors find that model accuracy falls steadily as DICE-SCORE rises, with even the strongest systems dropping from about 74% exact match in the simplest single-round setting to about 59% in four-round dialogues. The point of the work is that current assistants are not yet deployable in realistic group-chat scenarios, and that dispersion, not sheer input length, is the main obstacle.","feed_headline":"Scattered group-chat facts trip up even the best AI assistants","feed_subtitle":"A 1,607-dialogue benchmark shows models drop sharply as tool-related details spread across rounds and speakers.","key_machinery":"The load-bearing objects are DICE-SCORE and the Tool Graph. DICE-SCORE takes a vector S of per-utterance counts of tool-related items and a total item count T, and computes $$\\text{DICE}(S,T) = \\frac{\\min(|S_{\\ne 0}|, T) \\cdot \\sqrt{|S| \\cdot T}}{\\sum_{i \\in S} \\ln(1 + \\$\\alpha$ S_i)},$$ with $\\alpha = e^2$; a higher score means the items needed for the function call are spread across more turns with less repetition. The Tool Graph is a directed graph over 124 tool functions whose edges encode that one tool's output or context is needed to call another, and sampling paths from this graph forces the generated dialogues to carry dependencies across rounds. A multi-agent system with distinct personas and an orchestrator produces the dialogues, and a three-stage validation pipeline (LLM-based scoring, rule checks, and human criteria) filters them to 1,607 instances.","core_discovery":"The paper's central claim is that DICE-BENCH is the first function-calling benchmark that combines multi-round structure, multiple parties, and explicit dependencies between tools across rounds, and that current large language models perform poorly on it. Using DICE-SCORE, defined as a measure of how spread out function-related items are over dialogue turns, the authors report a strong inverse correlation between the metric and both model performance and human performance: human exact-match accuracy falls from 80.5% on round-1 dialogues (DICE-SCORE 1.42) to 49.3% on round-4 dialogues (DICE-SCORE 5.36), with Pearson r ≈ −0.984, and the 19 evaluated models show the same trend. The paper concludes that retrieving sparse and fragmented tool-related details, rather than long-context handling, is what degrades function-calling performance.","pith_inferences":["The dispersion effect DICE-SCORE measures is likely not unique to function calling: any task that requires aggregating facts from multiple speakers, such as meeting summarization or multi-source question answering, may show the same degradation, and DICE-SCORE could serve as a difficulty probe there.","A testable extension is to use DICE-SCORE as a training signal, for instance by generating curricula that start with low-dispersion dialogues and progress to high-dispersion ones.","The paper's attribution of the performance drop to dispersion rather than length relies on the specific form of DICE-SCORE; an independent test could directly regress model error on utterance count while controlling for dispersion.","DICE-BENCH's human validation suggests the benchmark is not trivially solvable, so it could be adopted as a stress test for dialogue-state tracking components in assistant pipelines."],"forward_implications":["If DICE-BENCH is an accurate testbed, then single-turn function-calling benchmarks overstate model readiness for real assistant use.","If DICE-SCORE captures what makes these tasks hard, benchmark designers can use it to label difficulty rather than relying on round count alone.","If dispersion rather than length is the bottleneck, then extending context windows will not by itself fix multi-party tool use; models need architectures or training that aggregate information across speakers.","If tool-specific models fine-tuned on single instructions generalize poorly, then future tool-use training should include multi-party dialogue data.","The reported human accuracy drop from 80.5% to 49.3% provides an upper-bound reference: even people find round-4 dialogues hard, so benchmark scores must be interpreted against human performance."],"supporting_citations":[{"why":"Supplies the tool-graph construction idea and the initial tool set that DICE-BENCH builds on.","marker":"(Shen et al., 2024)"},{"why":"Provides an additional collection of tools used to construct the 124-node Tool Graph.","marker":"(Ye et al., 2024a)"},{"why":"G-Eval is the LLM-based scorer used in the first validation stage to filter low-quality dialogues.","marker":"(Liu et al., 2023)"},{"why":"Defines the dialogue-type taxonomy (persuasion/negotiation, inquiry, eristic) used to configure scenarios.","marker":"(Walton and Krabbe, 1995)"},{"why":"API-Bank is one of the existing benchmarks compared through DICE-SCORE to show that current datasets have low dispersion.","marker":"(Li et al., 2023)"},{"why":"ToolLLM is another comparison benchmark whose low DICE-SCORE motivates the need for multi-party, multi-round evaluation.","marker":"(Qin et al., 2023)"},{"why":"APIBench is a single-turn baseline benchmark whose DICE-SCORE is reported to contrast with DICE-BENCH.","marker":"(Patil et al., 2023)"}],"fun_headline_variants":["AI assistants trip over scattered tool details in group chats","Scattered tool cues across turns stump even top AI models","When tool details scatter, AI accuracy drops sharply","Group chats expose AI's weakness with scattered tool calls","Multi-party chats scatter tool details, stumping LLMs"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper assumes that DICE-SCORE measures how spread out the needed facts are, and not just how long the conversation is, so that the observed performance drop can be attributed to dispersion.","fun_headline_variants_meta":{"raw":{"variants":["AI assistants trip over scattered tool details in group chats","Scattered tool cues across turns stump even top AI models","When tool details scatter, AI accuracy drops sharply","Group chats expose AI's weakness with scattered tool calls","Multi-party chats scatter tool details, stumping LLMs"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000662,"raw_usage":{"total_tokens":3010,"prompt_tokens":912,"completion_tokens":2098,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":528,"completion_tokens_details":{"reasoning_tokens":2021}},"tokens_in":528,"tokens_out":2098,"duration_ms":17056,"temperature":1.0,"reasoning_tokens":2021,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T21:57:42.965691+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare two sets of dialogues matched in total length and number of facts but differing in whether the facts are concentrated in one utterance or spread across turns; the paper's claim predicts a large accuracy gap, and a null result would falsify it.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the dialogue-type taxonomy (persuasion/negotiation, inquiry, eristic) used to configure scenarios."}],"review_version":1}