Pith. sign in

REVIEW 4 major objections 5 minor 29 references

ESF-Bench introduces a production-style slot-filling benchmark where the strongest evaluated LLM completes all slots correctly in only a third of conversations, far behind humans.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-07-31 23:42 UTC pith:XCQWGOEX

load-bearing objection Useful benchmark with a strong taxonomy, but the headline difficulty numbers rest on lightly validated synthetic gold labels. the 4 major comments →

arxiv 2607.23326 v1 pith:XCQWGOEX submitted 2026-07-25 cs.AI

ESF-Bench: Benchmarking Challenging Slot-Filling Scenarios for Real-World Enterprise Applications

Toby Liang , Gopal Sarda , Sagar Davasam , Vikas Yadav This is my paper
classification cs.AI
keywords enterprise slot fillingLLM benchmarkmulti-turn dialogueslot-filling taxonomyfinal joint goal accuracysynthetic data evaluationreasoning modelstask-oriented dialog
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper introduces ESF-Bench, a benchmark of 810 multi-turn enterprise conversations with 6,530 slots across eight domains, built from a taxonomy of 57 hard slot-filling scenarios observed in production. The authors claim current state-of-the-art LLMs are far from reliable: the best model gets every slot right in only 33.3% of conversations, while humans do 73.3%. The goal is to expose systematic gaps—especially in reasoning, personalization, and relative-slot updates—so that enterprise AI can be made more reliable. The benchmark is designed to require not simple extraction but normalization, multi-hop inference, source-priority handling, and all-or-nothing end-state correctness.

Core claim

The central discovery is that contemporary LLMs, even with extended reasoning, frequently fail when slot values depend on multi-step inference, cross-source priority, relative dependencies, sarcasm or double negation, or persona mismatch. On ESF-Bench, high-capability models average 26.1 Final Joint Goal Accuracy and 77.0 tagged slot accuracy; reasoning-heavy models outperform low-latency ones by 16.3 points F-JGA, yet the gap to humans remains large. The paper shows that reasoning and personalization are the weakest categories for the strongest models, and that conversation understanding is comparatively strong. This indicates that the bottleneck in enterprise slot filling is not basic extr

What carries the argument

The carrying device is a 57-scenario taxonomy organized into 12 categories (selection, conditionals, reasoning, multi-source, long/multi-value, conversation understanding, personalization, schema constraints, unexpected user behavior, relative, corrections, slot reset). Each slot is tagged with its scenario, enabling category-level accuracy and fine-grained diagnosis. The paper also introduces Final Joint Goal Accuracy (F-JGA), a metric requiring the entire final dialogue state to match ground truth exactly, which is stricter than turn-level JGA and matches the all-or-nothing nature of real enterprise deployments. The synthetic data pipeline uses GPT-5 for generation and judging, with a huma

Load-bearing premise

The correctness of the 6,530 synthetic ground-truth labels rests almost entirely on GPT-5-generated labels verified by a GPT-5 judge, with only 122 slots (1.9%) human-checked; if the judge accepts wrong labels, the reported model scores and human comparison do not reflect true performance.

What would settle it

Take a random sample of ESF-Bench conversations, have independent human annotators produce ground truth from scratch, and compare LLM performance on those samples against the paper's reported F-JGA. If human-verified accuracy on the sample differs substantially from the GPT-judged accuracy, the benchmark's scores would not be trustworthy.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported numbers hold, current LLMs cannot be trusted for unattended enterprise slot filling, since partial slot correctness rarely delivers business value.
  • Reasoning-heavy models more than double F-JGA compared to low-latency variants, suggesting test-time compute is a key lever for the hardest scenarios.
  • Category-level tagging reveals where to focus: personalization and reasoning are the weakest areas even for high-capability models, while conversation understanding is comparatively strong.
  • Open-source models are competitive on selected categories such as multi-source and relative slots, but trail on overall F-JGA and unexpected user behavior.
  • The large human-model gap implies that a successful deployment would need either substantial model improvements or human-in-the-loop verification for high-stakes conversations.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The taxonomy likely generalizes beyond slot filling to other structured-extraction tasks, such as form population or API parameter extraction, where the same constraints, corrections, and unexpected user behaviors appear.
  • A testable extension is to measure whether methods like self-consistency, few-shot demonstrations, or constraint-aware decoding close the reasoning and personalization gaps identified by ESF-Bench.
  • Because ground truth was generated and verified by LLMs, scores for models in the same family might be optimistically biased; the paper reports GPT models do not score highest, which mitigates but does not remove this concern.
  • The all-or-nothing F-JGA metric suggests that partial-credit metrics common in current evaluation may overstate production readiness; deployment gates should use end-state accuracy.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces ESF-Bench, a synthetic benchmark of 810 multi-turn enterprise slot-filling dialogues with 6,530 slots across 8 domains, built from a 57-scenario taxonomy. It defines F-JGA (Appendix B, Eq. 1), an exact-match metric on the final dialogue state, and evaluates five LLM families under high-capability and low-latency configurations. The headline result is that the best model, Gemini 2.5 Flash Dynamic-Thinking, achieves 33.3 F-JGA, GPT-OSS-120b Low achieves 20.7, while a human baseline on a 122-slot subset reaches 73.33 F-JGA. The paper releases the benchmark, taxonomy, and evaluation code.

Significance. If the gold labels are trustworthy, ESF-Bench would be a useful resource: the taxonomy is detailed, the multi-source prompt design goes beyond existing TOD benchmarks, and the public release of data and code is a concrete strength. The finding that high-capability LLMs fail on a large fraction of final dialogue states is an important signal for enterprise deployment. However, the central quantitative claim rests on GPT-5-generated and GPT-5-judged labels with human validation on only 1.9% of slots, on a human baseline computed from a very small subset without confidence intervals, and on an automated LLM-based output conversion. These components need stronger validation before the specific numbers can be accepted.

major comments (4)
  1. [§3.2.1, Table 4] Gold-label validity is not established. Ground truths are generated by GPT-5 and verified by a GPT-5 judge; the only human check covers 122 of 6,530 slots (1.9%) and reports 93.83% agreement, i.e., roughly 6% disagreement even on the checked subset, with no inter-annotator agreement reported. Since F-JGA (Appendix B, Eq. 1) requires all ~8.06 slots per sample to match exactly, a 3–6% label-error rate caps achievable F-JGA well below 100% and can lower measured F-JGA by more than the observable inter-model gaps. Section 7.2 acknowledges possible synthetic-data bias but does not quantify it. Please provide a larger human-validated subset, corrected gold labels, or an explicit noise-bound analysis demonstrating that the headline gaps survive plausible label error.
  2. [§5.3, Table 4] The human baseline is computed on 122 tagged slots; with an average of 8.06 slots per sample this is roughly 15 samples. No confidence interval, number of annotators, or inter-annotator agreement is reported, so the 73.33 F-JGA point estimate and the 33-point gap to the best model are not statistically established. In addition, the same 93.83% figure is used both as evidence of label quality (§3.2.1) and as human slot accuracy (§5.3); these are different quantities and should be separated. Please report sample size, variance, and the annotation protocol.
  3. [§3.2.3] Output standardization is done by an LLM-generated Python function that converts each prompt's output into a common evaluation format. No validation or error analysis of this conversion step is reported. If the conversion mis-parses a model's prediction, it directly corrupts F-JGA and tagged slot accuracy. Please report a manual or automatic accuracy check of the conversion on a sample of outputs per model, and describe how conversion failures are handled.
  4. [§4.1, Tables 2–3] All results appear to be single-run evaluations. With temperature 0.6 used for some configurations, F-JGA values are subject to sampling noise, yet no confidence intervals or repeated runs are reported. Claims such as the +16.3% F-JGA advantage of high-capability over low-latency models (§5.1) would be more convincing with multiple seeds and a variance estimate, especially given the small human-baseline sample used for comparison.
minor comments (5)
  1. [Abstract] The abstract highlights only GPT-OSS-120b Low (20.7%); the best model result (Gemini 2.5 Flash DT, 33.3) should also be reported for balance.
  2. [Table 2] Rows are not sorted by F-JGA, and GPT-OSS-120b Low appears after GPT-5.1 Minimal despite a higher score. Sorting or grouping labels would improve readability.
  3. [§4.2] Tagged Slot Accuracy is defined only as accuracy on tagged slots, which represent 66.8% of all slots; state this explicitly in the metric definition and in the captions of Tables 2–4.
  4. [§5.3] Describe how the 122-slot subset was selected as 'representative' (random, stratified, etc.) and report the number of samples/dialogues involved, not just slots.
  5. [Throughout] Minor typos: §3.2.1 'comprises of'; §7.2 'possiblity'; §1 'importantslot' missing space. Please copyedit.

Circularity Check

0 steps flagged

No significant circularity; ESF-Bench is an empirical benchmark study with independent human anchoring.

full rationale

The paper's central claim—that current LLMs achieve low F-JGA on ESF-Bench—is an empirical measurement, not a derivation from fitted parameters. F-JGA (Appendix B, Eq. 1) is defined independently as exact matching of the predicted final dialog state against fixed ground-truth states; no parameter is fitted to the reported scores and no prediction is algebraically equivalent to an input. The taxonomy of 57 scenarios is a curation choice, not a mathematical consequence of model outputs, and the paper does not rely on a self-citation chain or a uniqueness theorem. The main validity concern is the GPT-5-generated gold labels with GPT-5 judge verification and the small 122-slot human spot-check (§3.2.1, Table 4). This is a data-quality caveat, acknowledged in §7.2, and would affect benchmark reliability rather than constitute circularity, because the measured F-JGA values are not defined in terms of the label-generation process. The human baseline of 73.33 F-JGA on the subset provides an external anchor for the benchmark's difficulty. Therefore no circular step meeting the required evidentiary threshold is present.

Axiom & Free-Parameter Ledger

0 free parameters · 4 axioms · 0 invented entities

No free parameters or invented entities; the work is empirical benchmark construction. The load-bearing assumptions are about the quality and representativeness of the synthetic data and the validity of LLM-based judging.

axioms (4)
  • domain assumption The 57-scenario taxonomy is representative of real-world enterprise slot-filling challenges.
    §3.1: scenarios selected through 'detailed review' of production prompts and defects; no external validation of completeness or frequency.
  • domain assumption LLM-generated ground-truth labels are correct, as judged by another LLM.
    §3.2.1: 'Ground truths are then generated using a separate LLM call... with a judge model verifying label correctness.' Only 122 slots human-validated.
  • domain assumption The Python conversion function generated by an LLM maps each prompt's output to the standard format without errors.
    §3.2.3: output format conversion is automated via an LLM; no manual verification reported.
  • domain assumption Decoupling slot filling from intent detection is a valid setup for evaluating slot-filling capability.
    §3: ESF-Bench decouples slot filling from intent detection; this is a design choice, not a proven representation.

pith-pipeline@v1.3.0-alltime-deepseek · 13963 in / 9974 out tokens · 90302 ms · 2026-07-31T23:42:36.398055+00:00 · methodology

0 comments
read the original abstract

The rapid rise of large language models (LLMs) has driven transformative adoption across enterprises. However, deploying these models in real-world settings presents unique challenges due to complex system constraints and unexpected user behaviors. Among these applications, slot filling is essential for converting unstructured input into structured, actionable data. In this work, we introduce ESF-Bench, a challenging Enterprise Slot Filling benchmark consisting of 810 multi-turn samples and 6530 slots over 8 unique domains. Curated using a taxonomy of the 57 most challenging slot-filling scenarios observed during real-world enterprise deployments, ESF-Bench exposes notable limitations in current state-of-the-art LLMs, with GPT-OSS-120b low successfully extracting slots for only 20.7% of benchmark samples. To support continued research in this area, we publicly release the benchmark dataset, taxonomy, and accompanying evaluation code on GitHub.

Figures

Figures reproduced from arXiv: 2607.23326 by Gopal Sarda, Sagar Davasam, Toby Liang, Vikas Yadav.

Figure 1
Figure 1. Figure 1: An example conversation and final dialog state [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Distribution of ESF-Bench Taxonomy Categories [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Performance–efficiency trade-off across SOTA models. F-JGA is plotted against median com￾pletion length on a log scale. of enterprise slot-filling scenarios into benchmark￾ing samples. The pipeline begins by stochastically sampling slot-filling scenarios from the taxonomy, which are then used to systematically generate prompt components and their corresponding meta￾data through a series of LLM calls. Groun… view at source ↗
Figure 4
Figure 4. Figure 4: Slot-Level Category Scores For High-Capability Models (Left) and Low-Latency Models (Right) [PITH_FULL_IMAGE:figures/full_fig_p008_4.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

29 extracted references · 6 linked inside Pith

  1. [1]

    Snell, Charlie and Lee, Jaehoon and Xu, Kelvin and Kumar, Aviral , booktitle =. Scaling

  2. [2]

    arXiv preprint arXiv:2601.08584 , year =

    Ministral 3 , author =. arXiv preprint arXiv:2601.08584 , year =

    Pith/arXiv arXiv

  3. [3]

    2025 , doi =

    Nature , volume =. 2025 , doi =

    2025

  4. [4]

    Long, Lin and Wang, Rui and Xiao, Ruixuan and Zhao, Junbo and Ding, Xiao and Chen, Gang and Wang, Haobo , booktitle =. On. 2024 , publisher =

    2024

  5. [5]

    and Stoica, Ion , booktitle =

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging

  6. [6]

    Robustness testing of language understanding in task-oriented dialog , author =. Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Volume 1: Long Papers) , pages =. 2021 , publisher =

    2021

  7. [7]

    Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop , pages =

    Large language models for mathematical reasoning: Progresses and challenges , author =. Proceedings of the 18th Conference of the European Chapter of the Association for Computational Linguistics: Student Research Workshop , pages =. 2024 , publisher =

    2024

  8. [8]

    2024 , eprint=

    TimeBench: A Comprehensive Evaluation of Temporal Reasoning Abilities in Large Language Models , author=. 2024 , eprint=

    2024

  9. [9]

    Feng, Yujie and Lu, Zexin and Liu, Bo and Zhan, Liming and Wu, Xiao-Ming , booktitle =. Towards. 2023 , publisher =

    2023

  10. [10]

    Frontiers of Computer Science , volume =

    Large language models for generative information extraction: A survey , author =. Frontiers of Computer Science , volume =. 2024 , publisher =

    2024

  11. [11]

    Advances in Neural Information Processing Systems , volume =

    A simple language model for task-oriented dialogue , author =. Advances in Neural Information Processing Systems , volume =

  12. [12]

    Science China Technological Sciences , volume =

    Recent advances and challenges in task-oriented dialog systems , author =. Science China Technological Sciences , volume =. 2020 , publisher =

    2020

  13. [13]

    Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =

    Budzianowski, Pawe. Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing , pages =. 2018 , publisher =

    2018

  14. [14]

    Proceedings of the AAAI Conference on Artificial Intelligence , volume =

    Towards scalable multi-domain conversational agents: The schema-guided dialogue dataset , author =. Proceedings of the AAAI Conference on Artificial Intelligence , volume =

  15. [15]

    2023 , address =

    FitzGerald, Jack and Hench, Christopher and Peris, Charith and Mackie, Scott and Rottmann, Kay and Sanchez, Ana and Nash, Aaron and Urbach, Liam and Kakarala, Vishesh and Singh, Richa and Ranganath, Swetha and Crist, Laurie and Britan, Misha and Leeuwis, Wouter and Tur, Gokhan and Natarajan, Prem , booktitle =. 2023 , address =

    2023

  16. [16]

    Lin, Zhaojiang and Madotto, Andrea and Winata, Genta Indra and Xu, Peng and Jiang, Feijun and Hu, Yuxiang and Shi, Chen and Fung, Pascale , booktitle =

  17. [17]

    Transactions of the Association for Computational Linguistics , volume =

    Hu, Songbo and Zhou, Han and Hergul, Mete and Gritta, Milan and Zhang, Guchun and Iacobacci, Ignacio and Vuli. Transactions of the Association for Computational Linguistics , volume =

  18. [18]

    , booktitle =

    Xu, Weijie and Huang, Zicheng and Hu, Wenxiang and Fang, Xi and Cherukuri, Rajesh Kumar and Nayyar, Naumaan and Malandri, Lorenzo and Sengamedu, Srinivasan H. , booktitle =. 2024 , address =

    2024

  19. [19]

    Jacqmin, L. ``. Proceedings of the 23rd Annual Meeting of the Special Interest Group on Discourse and Dialogue , pages =. 2022 , address =. doi:10.18653/v1/2022.sigdial-1.33 , url =

    doi:10.18653/v1/2022.sigdial-1.33 2022

  20. [20]

    arXiv preprint arXiv:1805.10190 , year =

    Snips voice platform: An embedded spoken language understanding system for private-by-design voice interfaces , author =. arXiv preprint arXiv:1805.10190 , year =

    Pith/arXiv arXiv

  21. [21]

    Speech model pre-training for end-to-end spoken language understanding , author =. Proc. Interspeech , pages =

  22. [22]

    SLURP : A Spoken Language Understanding Resource Package

    Bastianelli, Emanuele and Vanzo, Andrea and Swietojanski, Pawel and Rieser, Verena. SLURP : A Spoken Language Understanding Resource Package. Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP). 2020. doi:10.18653/v1/2020.emnlp-main.588

    doi:10.18653/v1/2020.emnlp-main.588 2020

  23. [23]

    Lee, Harrison and Gupta, Raghav and Rastogi, Abhinav and Cao, Yuan and Zhang, Bin and Wu, Yonghui , booktitle =

  24. [24]

    and Godfrey, John J

    Hemphill, Charles T. and Godfrey, John J. and Doddington, George R. , booktitle =. The. 1990 , publisher =

    1990

  25. [25]

    arXiv preprint arXiv:2406.08848 , year =

    An approach to build zero-shot slot-filling system for industry-grade conversational assistants , author =. arXiv preprint arXiv:2406.08848 , year =

    Pith/arXiv arXiv

  26. [26]

    Zero-shot slot filling in the age of

    Rana, Mansi and Hacioglu, Kadri and Gopalan, Sindhuja and Boothalingam, Maragathamani , booktitle =. Zero-shot slot filling in the age of. 2025 , address =

    2025

  27. [27]

    arXiv preprint arXiv:2507.06261 , year =

    Gemini 2.5: Pushing the frontier with advanced reasoning, multimodality, long context, and next generation agentic capabilities , author =. arXiv preprint arXiv:2507.06261 , year =

    Pith/arXiv arXiv

  28. [28]

    arXiv preprint arXiv:2508.10925 , year=

    gpt-oss-120b & gpt-oss-20b model card , author=. arXiv preprint arXiv:2508.10925 , year=

    Pith/arXiv arXiv

  29. [29]

    arXiv preprint arXiv:2505.09388 , year =

    Qwen3 technical report , author =. arXiv preprint arXiv:2505.09388 , year =

    Pith/arXiv arXiv