Pith. sign in

REVIEW 2 major objections 5 minor 1 cited by

Aggregate tool-use scores hide distinct failures: the best model is only 86% clean, and same-scale models can differ by 89 points on when not to call a tool.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 15:11 UTC pith:FVP6GIBR

load-bearing objection Solid diagnostic benchmark: parametric traps + failure taxonomy cleanly show that aggregate tool scores hide distinct failure modes, with a striking Llama Always-Call vs Qwen discipline gap. the 2 major comments →

arxiv 2607.04686 v1 pith:FVP6GIBR submitted 2026-07-06 cs.CL cs.AIcs.SE

ToolFailBench: Diagnosing Tool-Use Failures in LLM Agents

Harsh Soni This is my paper
classification cs.CL cs.AIcs.SE
keywords tool useLLM agentsfunction callingdiagnostic benchmarkresult faithfulnessunnecessary tool usefailure modesToolFailBench
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Modern language-model agents are expected to call tools and then use what those tools return. Existing benchmarks often collapse that loop into one accuracy number, so a model that never calls a needed tool and a model that calls the tool but ignores the result can look the same. ToolFailBench is a 1,000-task diagnostic suite across finance, medicine, law, cybersecurity, and real estate. Tool-required tasks are built as parametric traps: the mock tool return deliberately contradicts a plausible memorized value, so the model must trust the tool. Control tasks attach the same tools but should be answered directly. Each trace is labeled Tool-Skip, Result-Ignore, Output-Fabrication, or Unnecessary-Tool-Use by a rule classifier plus two LLM judges (majority vote). Across 19 models the best Clean Tool-Use Rate is only 86.33%, so faithful tool use is not saturated. Models with similar aggregate scores fail differently: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same scale Llama-3.1-70B and Qwen2.5-72B differ by 89 points on control accuracy. The paper’s claim is that evaluation must measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Core claim

Aggregate tool-use metrics hide distinct failure modes. On ToolFailBench, the strongest of 19 models reaches only 86.33% Clean Tool-Use Rate. Most models form a disciplined low-UTR cluster on control tasks, while Llama-3.1 models show an Always-Call pattern; at matched scale, Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool discipline is therefore not explained by parameter count alone and depends strongly on model family and training behavior.

What carries the argument

ToolFailBench’s parametric traps plus failure-mode taxonomy: mock tool returns contradict a plausible prior so post-call faithfulness can be scored, and each trace is labeled Tool-Skip, Result-Ignore, Output-Fabrication, or Unnecessary-Tool-Use by a rule classifier and two LLM judges aggregated by majority vote.

Load-bearing premise

That setting mock tool returns against a plausible prior, then majority-voting a surface rule classifier with two LLM judges, cleanly isolates post-call faithfulness and unnecessary tool use rather than format brittleness, judge quirks, or single-turn artifacts.

What would settle it

If models that score high Clean Tool-Use Rate on these traps still ignore or invent tool results at high rates on held-out multi-turn or real-API tool workflows, or if the Always-Call Llama-vs-Qwen control gap shrinks to chance under refreshed traps and human labels, the diagnostic claim would not hold.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Leaderboards that report only final task accuracy or call validity will continue to treat Tool-Skip, Result-Ignore, and Always-Call models as interchangeable.
  • Training and fine-tuning for agents should target tool restraint on no-tool queries as a first-class objective, not only call correctness.
  • Same-scale model families can still have opposite tool discipline; family and training recipe matter more than parameter count alone for when not to call a tool.
  • Developers can use the released harness to get a failure-mode breakdown (TSR, RIR, OFR, UTR, CTRL-Acc) instead of a single pass rate.
  • Domain variation in Result-Ignore (finance highest among the five) means single-domain tool benchmarks can mis-estimate post-call faithfulness.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • If Always-Call is family-specific rather than a scale effect, post-training that rewards any tool call may be teaching the wrong prior for agent reliability.
  • Parametric-trap design could transfer to other agent skills—retrieval faithfulness, multi-hop state, or recovery after a bad tool return—where aggregate success also collapses different breaks in the loop.
  • The 89-point control gap is a natural stress test for whether future alignment or tool-policy methods actually teach restraint, not just more calling.
  • Judge ensembles that overrule surface rules mainly on paraphrases suggest pure string matching will systematically understate faithful tool use as models paraphrase more.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper introduces ToolFailBench, a 1,000-task diagnostic benchmark (five professional domains; 750 tool-required parametric traps and 250 no-tool controls) that labels single-turn agent traces with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use via a rule classifier plus two LLM judges under majority vote. Across 19 headline models under a fixed temperature-0 protocol, the best Clean Tool-Use Rate is 86.33% (unsaturated). Aggregate scores hide distinct profiles: most models form a disciplined low-UTR cluster, Llama-3.1 models show Always-Call behavior, and Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. The authors argue tool-use evaluation should measure not only whether tools are called, but whether outputs are used faithfully and tools are avoided when unnecessary.

Significance. If the reported behavioral contrasts hold under the stated protocol, the work is a useful diagnostic contribution for agent evaluation: it separates post-call faithfulness and tool restraint from call validity and end-task success, areas that BFCL, ToolLLM, τ-bench, and related suites often collapse. Strengths include a released harness (github.com/SoHarshh/ToolFailBench), a fixed decoding protocol, Wilson CIs, a two-proportion test on the Llama–Qwen control gap, inter-rater statistics (Fleiss’ κ ≈ 0.693), a same-family judge check showing nearly identical Qwen-vs-GLM gaps for Qwen and non-Qwen models, and explicit rule-only vs ensemble reporting. These make the failure-profile claim more actionable for model developers than a single pass rate.

major comments (2)
  1. [§3.4, §5.4, Appendix C] §3.4 / §5.4 / Appendix C: The headline taxonomy rests on majority vote of a surface rule classifier and two LLM judges, with no reported human gold labels or human–ensemble agreement. Fleiss’ κ = 0.693 and the same-family check reduce some bias concerns, but they do not establish that Result-Ignore vs Output-Fabrication (and faithful paraphrase vs ignore) match expert intent. For a diagnostic benchmark whose central claim is that aggregate scores hide distinct failure modes, a modest human-labeled subset (or adjudication study) is load-bearing for label validity; without it, the rates remain internally consistent automated scores rather than fully validated failure diagnoses.
  2. [§3.1, Appendix A.2, §5.5] §3.1 / Appendix A.2: Parametric traps are defined so that mock returns “contradict a plausible prior,” yet the manuscript treats “memorized prior” as a design term and gives limited systematic detail on how competing values were chosen, validated as actually competing for the evaluated models, or balanced across domains. Finance’s higher median rule RIR (§5.5) is interpreted as possible prior competition, but without construction criteria or a check that trap offsets are comparable, Result-Ignore rates mix faithfulness failures with trap salience. Clarifying trap generation and, ideally, a small prior-competition check would strengthen the claim that RIR isolates post-call unfaithfulness rather than task design artifacts.
minor comments (5)
  1. [Table 1, Table 2] Table 1 states TSR + CTUR + RIR + OFR sum to 100% on Required; Table 2 reports rates that appear consistent, but denominators for Called-subset metrics (RIR/OFR) vs full Required should be stated once in the table caption to avoid misreading rates as sharing a single base.
  2. [Figure 2] Figure 2’s CTUR–UTR scatter is central to the Always-Call claim; axis scales, exact UTR=0 clustering, and which points are Llama-3.1 should be unambiguous in the caption for readers of the print version.
  3. [§4.1, Appendix D] §4.1 / Appendix B.1: Three non-headline runs are documented carefully; a one-sentence pointer in the main text to why glm-4-9b/mistral-7b are tool-style-without-execution cases would help readers who skip the appendix.
  4. [§2] Related Work is generally well positioned; a slightly sharper contrast with SMART (tool overuse) and ToolBeHonest (honesty/hallucination) on what is newly isolated by controlled parametric returns would help.
  5. [§6 Limitations] Limitations correctly flag single-turn scope and future train-test overlap; stating whether task IDs/values will be versioned or refreshed for leaderboard use would aid long-term use of the harness.

Circularity Check

0 steps flagged

No significant circularity: empirical benchmark metrics are operational labels on held-out traces, not predictions forced by fitted inputs or self-citation.

full rationale

ToolFailBench is an empirical diagnostic evaluation paper, not a first-principles derivation. Its load-bearing claims (unsaturated Clean Tool-Use Rate, disciplined low-UTR cluster vs Llama-3.1 Always-Call, 89-point same-scale control-accuracy gap) are measurements obtained by running fixed tasks, mock tool returns, and a fixed labeling pipeline on model traces. CTUR/TSR/RIR/OFR/UTR/CTRL-Acc are operational definitions of observable behaviors (tool called or not; answer matches controlled return or not; tool used on controls or not), not quantities fitted then re-presented as predictions. The parametric-trap design deliberately sets tool returns to differ from plausible priors as a construction choice for scoring faithfulness; that is experimental control, not circular reduction of a claimed prediction to its fit. Labeling uses a deterministic rule classifier plus two LLM judges with majority vote; agreement and same-family checks are robustness audits, not self-justifying theorems. Related-work citations (BFCL, τ-bench, ToolLLM, SpecTool, SMART, etc.) are external benchmarks and prior diagnostic lines, not load-bearing uniqueness results by the present author. No self-definitional loop, fitted-input-as-prediction, self-citation chain, uniqueness import, smuggled ansatz, or mere renaming of a known result appears in the derivation of the central claims. Score 0 is therefore appropriate.

Axiom & Free-Parameter Ledger

3 free parameters · 4 axioms · 4 invented entities

The paper’s claims rest on operational design choices (parametric traps, mock returns, single-turn protocol, ensemble labeling) rather than fitted physical constants. Free parameters are experimental knobs and construction choices that define the measurement, not theory fits. Axioms are standard evaluation assumptions plus domain-specific notions of faithfulness. Invented entities are the taxonomy, metrics, and trap mechanism that make the diagnostic claim possible.

free parameters (3)
  • temperature / max_tokens / seed decoding settings = temp=0, max_tokens=1024, seed=42
    Fixed at temperature=0, max_tokens=1024, seed=42 where supported; these control reproducibility of traces and are chosen by hand, not learned, but headline rates depend on this protocol.
  • parametric trap offset from plausible prior
    Mock tool returns are intentionally set away from likely memorized values; the magnitude and choice of trap values are design choices that define how hard Result-Ignore is to avoid.
  • task split 150 tool-required / 50 control per domain = 150/50 per domain × 5
    Fixed 750/250 design weights the leaderboard toward required-tool metrics; not fitted to model scores but still a free design parameter of the claim surface.
axioms (4)
  • domain assumption A final answer that follows the controlled mock tool return is faithful tool use; replacing it with an unsupported value is Result-Ignore or Output-Fabrication.
    Core scoring assumption in Sections 3.1–3.2; treats observable agreement with tool return as the ground truth of faithfulness.
  • domain assumption Presence of an executed tool call in the trace is necessary for non-skip / for detecting Unnecessary-Tool-Use on controls.
    Rule classifier and critical judge instructions treat only executed calls as real tool use (Appendix C/E).
  • ad hoc to paper Majority vote of rule classifier + two differently prompted LLM judges yields a usable failure-mode label.
    Labeling protocol of Section 3.4/4.3; justified by κ and same-family checks but still an evaluation axiom.
  • standard math Standard statistical comparisons (Wilson CIs, two-proportion z-test) apply to ensemble-labeled task outcomes treated as independent Bernoulli trials.
    Used for leaderboard CIs and the Llama–Qwen control gap significance claim.
invented entities (4)
  • ToolFailBench parametric-trap task suite independent evidence
    purpose: Force models to reveal whether they use tool returns rather than priors on professional-domain queries.
    New benchmark construction; independent evidence is the released tasks/harness, not an external physical measurement.
  • Failure-mode taxonomy (Tool-Skip, Result-Ignore, Output-Fabrication, Unnecessary-Tool-Use) no independent evidence
    purpose: Partition tool-use failures that aggregate accuracy collapses.
    Defined operationally for this paper; related to prior diagnostic work but packaged as this taxonomy.
  • Clean Tool-Use Rate (CTUR) and companion rates (TSR/RIR/OFR/UTR/CTRL-Acc) no independent evidence
    purpose: Primary leaderboard and profile metrics over Required/Called/Control sets.
    Metric definitions in Table 1; CTUR is the headline ranking quantity.
  • Always-Call failure profile (Llama-3.1 cluster) no independent evidence
    purpose: Name the high-UTR / low CTRL-Acc regime observed for Llama-3.1-8B/70B.
    Empirical cluster label from Figure 2 / Table 3, not a pre-existing standard entity.

pith-pipeline@v1.1.0-grok45 · 24500 in / 3670 out tokens · 38932 ms · 2026-07-11T15:11:58.317940+00:00 · methodology

0 comments
read the original abstract

Tool calling is central to modern language model agents, but aggregate benchmark scores often hide where tool use fails. A model that never calls a needed tool and a model that calls the tool but ignores the result can look similar under final task accuracy. We introduce ToolFailBench, a diagnostic benchmark for measuring tool-use failures across 1,000 tasks in finance, medicine, law, cybersecurity, and real estate. Tool-required tasks return values the model wouldn't guess, forcing it to trust the tool while control tasks attach the same tools but should be answered directly. We label each trace with Tool-Skip, Result-Ignore, Output-Fabrication, and Unnecessary-Tool-Use, using a rule classifier and two LLM judges aggregated by majority vote. Across 19 headline models, the best reaches 86.33% Clean Tool-Use Rate, showing that faithful tool use is not saturated. More importantly, models with similar aggregate scores fail in different ways: most stay disciplined on no-tool controls, while Llama-3.1 models show an Always-Call pattern, and at the same parameter scale Llama-3.1-70B and Qwen2.5-72B differ by 89 percentage points on control-task accuracy. Tool-use evaluation should measure not only whether agents call tools, but whether they use tool outputs correctly and avoid tools when none is needed.

Figures

Figures reproduced from arXiv: 2607.04686 by Harsh Soni.

Figure 1
Figure 1. Figure 1: ToolFailBench overview. The benchmark contains 1,000 single-turn tasks across five professional domains, with 750 tool￾required parametric traps and 250 control tasks. In each trap, the mock tool return contradicts a plausible parametric prior, so the model’s answer reveals whether it follows the tool return or its prior. Each trace is labeled by a rule classifier and two LLM judges via majority vote, prod… view at source ↗
Figure 2
Figure 2. Figure 2: CTUR–UTR failure profiles. Most models remain disci￾plined, while Llama-3.1 models form Always-Call outliers. zero, meaning they rarely call tools on control questions where no tool is needed. This cluster includes the closed￾API models, the Qwen models, Gemma, GLM-4.7-Flash, QwQ-32B, and DeepSeek-V4-Flash. Across these models, UTR stays at or below 1.61%, and CTRL-Acc stays above 95%. The Llama-3.1 models… view at source ↗
Figure 3
Figure 3. Figure 3: Domain-level rule Result-Ignore Rate across the 17 dis￾ciplined models. Finance has the highest median RIR, suggesting stronger competition from plausible prior values. Finance is the hardest domain for result faithfulness, with a median rule RIR of 12.24%, compared with Cybersecu￾rity’s 0.68%. One interpretation is that finance traps use values such as stock prices, P/E ratios, and market caps, where plau… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Execution-First Synthetic Tool-Use Trace Generation for LLM Agents

    cs.SE 2026-07 conditional novelty 5.0

    An execution-first data generator that validates tool traces before synthesizing user tasks improves Qwen agents' tool use and answer quality, with masked reasoning supervision beating full-trace supervision.

Reference graph

Works this paper leans on

29 extracted references · 5 linked inside Pith · cited by 1 Pith paper

  1. [1]

    arXiv preprint arXiv:2302.04761 , year=

    Toolformer: Language Models Can Teach Themselves to Use Tools , author=. arXiv preprint arXiv:2302.04761 , year=

    Pith/arXiv arXiv

  2. [2]

    and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E

    Patil, Shishir G. and Mao, Huanzhi and Yan, Fanjia and Ji, Charlie Cheng-Jie and Suresh, Vishnu and Stoica, Ion and Gonzalez, Joseph E. , booktitle=. The Berkeley Function Calling Leaderboard (. 2025 , publisher=

    2025

  3. [3]

    arXiv preprint arXiv:2406.12045 , year=

    -bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains , author=. arXiv preprint arXiv:2406.12045 , year=

    Pith/arXiv arXiv

  4. [4]

    and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E

    Patil, Shishir G. and Zhang, Tianjun and Wang, Xin and Gonzalez, Joseph E. , journal =. 2023 , url =

    2023

  5. [5]

    arXiv preprint arXiv:2510.25726 , year=

    The Tool Decathlon: Benchmarking Language Agents for Diverse, Realistic, and Long-Horizon Task Execution , author=. arXiv preprint arXiv:2510.25726 , year=

    arXiv

  6. [6]

    2024 , url=

    Qin, Yujia and Liang, Shihao and Ye, Yining and Zhu, Kunlun and Yan, Lan and Lu, Yaxi and Lin, Yankai and Cong, Xin and Tang, Xiangru and Qian, Bill and Zhao, Sihan and Hong, Lauren and Tian, Runchu and Xie, Ruobing and Zhou, Jie and Gerstein, Mark and Li, Dahai and Liu, Zhiyuan and Sun, Maosong , booktitle=. 2024 , url=

    2024

  7. [7]

    and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R

    Jimenez, Carlos E. and Yang, John and Wettig, Alexander and Yao, Shunyu and Pei, Kexin and Press, Ofir and Narasimhan, Karthik R. , booktitle=. 2024 , url=

    2024

  8. [8]

    Finance Agent Benchmark: Benchmarking

    Bigeard, Antoine and Nashold, Langston and Krishnan, Rayan and Wu, Shirley , journal=. Finance Agent Benchmark: Benchmarking. 2025 , url=

    2025

  9. [9]

    and Geng, Gloria and Park, Danny and Zou, James and Ng, Andrew Y

    Jiang, Yixing and Black, Kameron C. and Geng, Gloria and Park, Danny and Zou, James and Ng, Andrew Y. and Chen, Jonathan H. , journal=. 2025 , url=

    2025

  10. [10]

    2024 , url=

    Zhang, Yuxiang and Chen, Jing and Wang, Junjie and Liu, Yaxin and Yang, Cheng and Shi, Chufan and Zhu, Xinyu and Lin, Zihao and Wan, Hanwen and Yang, Yujiu and Sakai, Tetsuya and Feng, Tian and Yamana, Hayato , booktitle=. 2024 , url=

    2024

  11. [11]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle=. Judging. 2023 , url=

    2023

  12. [12]

    Grattafiori, Aaron and Dubey, Abhimanyu and Jauhri, Abhinav and Pandey, Abhinav and Kadian, Abhishek and Al-Dahle, Ahmad and Letman, Aiesha and Mathur, Akhil and Schelten, Alan and Vaughan, Alex and others , journal=. The. 2024 , url=

    2024

  13. [13]

    arXiv preprint arXiv:2412.15115 , year=

    Pith/arXiv arXiv

  14. [14]

    2024 , url=

    Guo, Zhicheng and Cheng, Sijie and Wang, Hao and Liang, Shihao and Qin, Yujia and Li, Peng and Liu, Zhiyuan and Sun, Maosong and Liu, Yang , journal=. 2024 , url=

    2024

  15. [15]

    2024 , url=

    Kokane, Shirley and Zhu, Ming and Awalgaonkar, Tulika and Zhang, Jianguo and Hoang, Thai and Prabhakar, Akshara and Liu, Zuxin and Lan, Tian and Yang, Liangwei and Tan, Juntao and Murthy, Rithesh and Yao, Weiran and Liu, Zhiwei and Niebles, Juan Carlos and Wang, Huan and Heinecke, Shelby and Xiong, Caiming and Savarese, Silvio , journal=. 2024 , url=

    2024

  16. [16]

    Educational and Psychological Measurement , volume=

    A Coefficient of Agreement for Nominal Scales , author=. Educational and Psychological Measurement , volume=

  17. [17]

    Psychological Bulletin , volume=

    Measuring Nominal Scale Agreement among Many Raters , author=. Psychological Bulletin , volume=

  18. [18]

    2025 , url=

    Yang, An and Li, Anfeng and Yang, Baosong and Zhang, Beichen and Hui, Binyuan and Zheng, Bo and Yu, Bowen and Gao, Chang and Huang, Chengen and Lv, Chenxu and others , journal=. 2025 , url=

    2025

  19. [19]

    arXiv preprint arXiv:2406.12793 , year=

    Pith/arXiv arXiv

  20. [20]

    and Zhang, Hao and Stoica, Ion , booktitle=

    Kwon, Woosuk and Li, Zhuohan and Zhuang, Siyuan and Sheng, Ying and Zheng, Lianmin and Yu, Cody Hao and Gonzalez, Joseph E. and Zhang, Hao and Stoica, Ion , booktitle=. Efficient Memory Management for Large Language Model Serving with. 2023 , doi=

    2023

  21. [21]

    2023 , url=

    Li, Minghao and Zhao, Yingxiu and Yu, Bowen and Song, Feifan and Li, Hangyu and Yu, Haiyang and Li, Zhoujun and Huang, Fei and Li, Yongbin , journal=. 2023 , url=

    2023

  22. [22]

    2023 , url=

    Chen, Zehui and Du, Weihua and Zhang, Wenwei and Liu, Kuikun and Liu, Jiangning and Zheng, Miao and Zhuo, Jingming and Zhang, Songyang and Lin, Dahua and Chen, Kai and Zhao, Feng , journal=. 2023 , url=

    2023

  23. [23]

    2024 , url=

    Lu, Jiarui and Holleis, Thomas and Zhang, Yizhe and Aumayer, Bernhard and Nan, Feng and Bai, Felix and Ma, Shuang and Ma, Shen and Li, Mengyu and Yin, Guoli and Wang, Zirui and Pang, Ruoming , journal=. 2024 , url=

    2024

  24. [24]

    2024 , url=

    Trivedi, Harsh and Khot, Tushar and Hartmann, Mareike and Manku, Ruskin and Dong, Vinty and Li, Edward and Gupta, Shashank and Sabharwal, Ashish and Balasubramanian, Niranjan , journal=. 2024 , url=

    2024

  25. [25]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging. 2023 , volume =

    2023

  26. [26]

    2026 , howpublished =

    GPU Acceleration , author =. 2026 , howpublished =

    2026

  27. [27]

    2025 , month=

    Qian, Cheng and Acikgoz, Emre Can and Wang, Hongru and Chen, Xiusi and Sil, Avirup and Hakkani-Tur, Dilek and Tur, Gokhan and Ji, Heng , booktitle=. 2025 , month=. doi:10.18653/v1/2025.findings-acl.239 , url=

    doi:10.18653/v1/2025.findings-acl.239 2025

  28. [28]

    arXiv preprint arXiv:2402.05201 , year=

    The Effect of Sampling Temperature on Problem Solving in Large Language Models , author=. arXiv preprint arXiv:2402.05201 , year=

    Pith/arXiv arXiv

  29. [29]

    The Good, The Bad, and The Greedy: Evaluation of

    Song, Yifan and Wang, Guoyin and Li, Sujian and Lin, Bill Yuchen , journal=. The Good, The Bad, and The Greedy: Evaluation of. 2024 , url=

    2024