{"id":"9d8bd85e-fa39-4d60-b349-79f6cde42c98","arxiv_id":"2607.23326","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"ESF-Bench is a new 810-sample, 6,530-slot benchmark showing state-of-the-art LLMs resolve only about a third of complex enterprise slot-filling dialogues correctly, far below human performance.","lead":"This paper introduces ESF-Bench, a benchmark of 810 multi-turn enterprise conversations with 6,530 slots that tests how well large language models convert messy user requests into structured data. It reports that the best AI models fill all slots correctly in only about 33% of cases, versus 73% for humans, revealing a large gap for enterprise AI assistants.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline F-JGA numbers are built on unvalidated GPT-5-generated gold labels; the paper's own human check found only 93.83% slot agreement on 122/6530 slots, enough to materially deflate an exact-match metric.","rationale":"I read ESF-Bench as a benchmark-resource paper whose central empirical assertion is that SOTA LLMs are poor at complex enterprise slot-filling, anchored by a 20.7% F-JGA for a strong open model and a 73.33% human F-JGA. For that assertion to hold, the 6,530 synthetic gold labels must be essentially correct; otherwise both the low model scores and the human-vs-model gap are artifacts of label noise or task ambiguity. The reader's weakest-assumption analysis identifies precisely this vulnerability, and the manuscript's own data sharpen it: the human validation in §5.3 is better read as a red flag than a confirmation. A 93.83% human agreement rate on 122 tagged slots means one in sixteen checked slots is disputed, and with exact-match F-JGA the effect on sample-level accuracy is amplified by the ~8 slots per sample. The absence of inter-annotator agreement, adjudication, and confidence intervals makes the 73.33 human baseline statistically fragile. I considered whether the LLM-generated output-conversion functions (§3.2.3) are an even weaker link, but those are at least deterministic code and checkable; the gold-label problem directly determines the validity of every number in Tables 2–4. I am not recommending rejection: the benchmark has independent value as a released resource, the taxonomy is detailed, and the broad trend across many models is unlikely to reverse. But the headline numbers should be treated as provisional until a larger human adjudication is run, which is exactly the condition the reader already attached. Hence the verdict stays unchanged.","tokens_in":14255,"tokens_out":6563,"duration_ms":64400,"concrete_test":"Re-annotate a stratified random sample of at least 500 slots (drawn across all 12 taxonomy categories) with three independent human annotators and adjudicate disagreements against the original GPT-5 gold. Then recompute F-JGA for at least GPT-OSS-120b Low and Gemini 2.5 Flash DT using the corrected gold set (or a conservative 'gold-minus-disputed' variant). Report per-slot corrected accuracy, Krippendorff's alpha, and the change in F-JGA relative to Table 2. If F-JGA rises by more than 5 absolute points for either model, the original headline is materially inflated by label noise; if it moves less than 2 points, the synthetic labels are adequate.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that GPT-OSS-120b Low extracts slots for only 20.7% of samples—uses Final JGA (Appendix B, Eq. 1), which requires every slot in the final state to match the gold label exactly. The gold labels are produced by GPT-5 and verified by a GPT-5 judge (§3.2.1). The only human validation, reported in §5.3/Table 4, is a spot-check of 122 tagged slots (1.9% of 6,530) in which human predictions matched the gold on only 93.83% of slots. That is not a clean certification of label quality: it means roughly 6% of even the tagged slots are contested, and untagged slots are left unchecked. With an average of 8.06 slots per sample, a true label-error rate of even 3–6% caps achievable F-JGA well below 100% and can lower measured F-JGA by tens of points. The paper's own human baseline on the tiny subset is 73.33 F-JGA with no confidence interval and no inter-annotator agreement, so the 20.7%-vs-73.3% gap is not a calibrated yardstick. Section 7.2 admits the synthetic pipeline may be biased but does not quantify it. If gold labels are wrong on a few percent of slots, the headline 'notable limitations' may be substantially overstated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ESF-Bench, a synthetic benchmark of 810 multi-turn enterprise slot-filling dialogues with 6,530 slots across 8 domains, built from a 57-scenario taxonomy. It defines F-JGA (Appendix B, Eq. 1), an exact-match metric on the final dialogue state, and evaluates five LLM families under high-capability and low-latency configurations. The headline result is that the best model, Gemini 2.5 Flash Dynamic-Thinking, achieves 33.3 F-JGA, GPT-OSS-120b Low achieves 20.7, while a human baseline on a 122-slot subset reaches 73.33 F-JGA. The paper releases the benchmark, taxonomy, and evaluation code.","tokens_in":14590,"tokens_out":6703,"duration_ms":60473,"significance":"If the gold labels are trustworthy, ESF-Bench would be a useful resource: the taxonomy is detailed, the multi-source prompt design goes beyond existing TOD benchmarks, and the public release of data and code is a concrete strength. The finding that high-capability LLMs fail on a large fraction of final dialogue states is an important signal for enterprise deployment. However, the central quantitative claim rests on GPT-5-generated and GPT-5-judged labels with human validation on only 1.9% of slots, on a human baseline computed from a very small subset without confidence intervals, and on an automated LLM-based output conversion. These components need stronger validation before the specific numbers can be accepted.","major_comments":[{"comment":"Gold-label validity is not established. Ground truths are generated by GPT-5 and verified by a GPT-5 judge; the only human check covers 122 of 6,530 slots (1.9%) and reports 93.83% agreement, i.e., roughly 6% disagreement even on the checked subset, with no inter-annotator agreement reported. Since F-JGA (Appendix B, Eq. 1) requires all ~8.06 slots per sample to match exactly, a 3–6% label-error rate caps achievable F-JGA well below 100% and can lower measured F-JGA by more than the observable inter-model gaps. Section 7.2 acknowledges possible synthetic-data bias but does not quantify it. Please provide a larger human-validated subset, corrected gold labels, or an explicit noise-bound analysis demonstrating that the headline gaps survive plausible label error.","section":"§3.2.1, Table 4"},{"comment":"The human baseline is computed on 122 tagged slots; with an average of 8.06 slots per sample this is roughly 15 samples. No confidence interval, number of annotators, or inter-annotator agreement is reported, so the 73.33 F-JGA point estimate and the 33-point gap to the best model are not statistically established. In addition, the same 93.83% figure is used both as evidence of label quality (§3.2.1) and as human slot accuracy (§5.3); these are different quantities and should be separated. Please report sample size, variance, and the annotation protocol.","section":"§5.3, Table 4"},{"comment":"Output standardization is done by an LLM-generated Python function that converts each prompt's output into a common evaluation format. No validation or error analysis of this conversion step is reported. If the conversion mis-parses a model's prediction, it directly corrupts F-JGA and tagged slot accuracy. Please report a manual or automatic accuracy check of the conversion on a sample of outputs per model, and describe how conversion failures are handled.","section":"§3.2.3"},{"comment":"All results appear to be single-run evaluations. With temperature 0.6 used for some configurations, F-JGA values are subject to sampling noise, yet no confidence intervals or repeated runs are reported. Claims such as the +16.3% F-JGA advantage of high-capability over low-latency models (§5.1) would be more convincing with multiple seeds and a variance estimate, especially given the small human-baseline sample used for comparison.","section":"§4.1, Tables 2–3"}],"minor_comments":[{"comment":"The abstract highlights only GPT-OSS-120b Low (20.7%); the best model result (Gemini 2.5 Flash DT, 33.3) should also be reported for balance.","section":"Abstract"},{"comment":"Rows are not sorted by F-JGA, and GPT-OSS-120b Low appears after GPT-5.1 Minimal despite a higher score. Sorting or grouping labels would improve readability.","section":"Table 2"},{"comment":"Tagged Slot Accuracy is defined only as accuracy on tagged slots, which represent 66.8% of all slots; state this explicitly in the metric definition and in the captions of Tables 2–4.","section":"§4.2"},{"comment":"Describe how the 122-slot subset was selected as 'representative' (random, stratified, etc.) and report the number of samples/dialogues involved, not just slots.","section":"§5.3"},{"comment":"Minor typos: §3.2.1 'comprises of'; §7.2 'possiblity'; §1 'importantslot' missing space. Please copyedit.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The taxonomy and dataset release are valuable, and the central claim is plausible, but the paper currently overstates the reliability of its headline numbers. I would not reject: the label-validation and human-baseline concerns are addressable with additional analysis and reporting. The revision should convert the acknowledgment of possible bias in §7.2 into a quantified robustness analysis."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"ESF-Bench is a genuine contribution: 810 multi-turn samples across 8 enterprise domains, a 57-scenario taxonomy more granular than prior TOD benchmarks, and a multi-source context setup (persona, KB, forms) that is closer to production reality. The authors release code and data, and the human-baseline exercise is a good idea even if limited.\n\nThe main soft spot is label quality. Gold slots are generated by GPT-5 and verified by a GPT-5 judge; only 122 of 6530 slots were human-checked, and agreement was 93.83%. For an exact-match, all-slots metric like F-JGA, a few percent per-slot label error can deflate every model's score by tens of points. The stress-test note is fair: the paper's own numbers could be materially better than reported, and the human baseline is measured against the same imperfect gold. That said, the gap between humans and models is large enough that the qualitative conclusion—current LLMs are far from reliable on hard enterprise slot filling—probably survives. What is less certain is the precise magnitude, and the model-to-model comparisons would be more trustworthy with repeated runs and confidence intervals.\n\nThe paper's limitation section acknowledges synthetic-data bias without quantifying it, which is honest but leaves the work unfinished. For a benchmark whose whole purpose is calibrated difficulty, 1.9% human validation is thin.\n\nWho this is for: people building or evaluating enterprise conversational AI, and the task-oriented-dialog community that needs harder stress tests than MultiWOZ or SGD. I'd cite it as a resource, and I'd want to see the human validation expanded before trusting the headline numbers. It deserves peer review, not desk rejection, but the reviewers should push hard on label-quality evidence.","headline":"Useful benchmark with a strong taxonomy, but the headline difficulty numbers rest on lightly validated synthetic gold labels.","tokens_in":15079,"tokens_out":3229,"would_cite":true,"duration_ms":33923,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ESF-Bench introduces a production-style slot-filling benchmark where the strongest evaluated LLM completes all slots correctly in only a third of conversations, far behind humans.","keywords":["enterprise slot filling","LLM benchmark","multi-turn dialogue","slot-filling taxonomy","final joint goal accuracy","synthetic data evaluation","reasoning models","task-oriented dialog"],"falsifier":"Take a random sample of ESF-Bench conversations, have independent human annotators produce ground truth from scratch, and compare LLM performance on those samples against the paper's reported F-JGA. If human-verified accuracy on the sample differs substantially from the GPT-judged accuracy, the benchmark's scores would not be trustworthy.","tokens_in":14149,"feed_emoji":"🤖","tokens_out":2987,"duration_ms":30009,"temperature":0.7,"pith_summary":"This paper introduces ESF-Bench, a benchmark of 810 multi-turn enterprise conversations with 6,530 slots across eight domains, built from a taxonomy of 57 hard slot-filling scenarios observed in production. The authors claim current state-of-the-art LLMs are far from reliable: the best model gets every slot right in only 33.3% of conversations, while humans do 73.3%. The goal is to expose systematic gaps—especially in reasoning, personalization, and relative-slot updates—so that enterprise AI can be made more reliable. The benchmark is designed to require not simple extraction but normalization, multi-hop inference, source-priority handling, and all-or-nothing end-state correctness.","feed_headline":"Top LLM nails every slot in 33% of real-world enterprise chats","feed_subtitle":"A new 810-conversation benchmark shows reasoning-optimized models still trail humans by 40 points on end-state accuracy.","key_machinery":"The carrying device is a 57-scenario taxonomy organized into 12 categories (selection, conditionals, reasoning, multi-source, long/multi-value, conversation understanding, personalization, schema constraints, unexpected user behavior, relative, corrections, slot reset). Each slot is tagged with its scenario, enabling category-level accuracy and fine-grained diagnosis. The paper also introduces Final Joint Goal Accuracy (F-JGA), a metric requiring the entire final dialogue state to match ground truth exactly, which is stricter than turn-level JGA and matches the all-or-nothing nature of real enterprise deployments. The synthetic data pipeline uses GPT-5 for generation and judging, with a huma","core_discovery":"The central discovery is that contemporary LLMs, even with extended reasoning, frequently fail when slot values depend on multi-step inference, cross-source priority, relative dependencies, sarcasm or double negation, or persona mismatch. On ESF-Bench, high-capability models average 26.1 Final Joint Goal Accuracy and 77.0 tagged slot accuracy; reasoning-heavy models outperform low-latency ones by 16.3 points F-JGA, yet the gap to humans remains large. The paper shows that reasoning and personalization are the weakest categories for the strongest models, and that conversation understanding is comparatively strong. This indicates that the bottleneck in enterprise slot filling is not basic extr","pith_inferences":["The taxonomy likely generalizes beyond slot filling to other structured-extraction tasks, such as form population or API parameter extraction, where the same constraints, corrections, and unexpected user behaviors appear.","A testable extension is to measure whether methods like self-consistency, few-shot demonstrations, or constraint-aware decoding close the reasoning and personalization gaps identified by ESF-Bench.","Because ground truth was generated and verified by LLMs, scores for models in the same family might be optimistically biased; the paper reports GPT models do not score highest, which mitigates but does not remove this concern.","The all-or-nothing F-JGA metric suggests that partial-credit metrics common in current evaluation may overstate production readiness; deployment gates should use end-state accuracy."],"forward_implications":["If the reported numbers hold, current LLMs cannot be trusted for unattended enterprise slot filling, since partial slot correctness rarely delivers business value.","Reasoning-heavy models more than double F-JGA compared to low-latency variants, suggesting test-time compute is a key lever for the hardest scenarios.","Category-level tagging reveals where to focus: personalization and reasoning are the weakest areas even for high-capability models, while conversation understanding is comparatively strong.","Open-source models are competitive on selected categories such as multi-source and relative slots, but trail on overall F-JGA and unexpected user behavior.","The large human-model gap implies that a successful deployment would need either substantial model improvements or human-in-the-loop verification for high-stakes conversations."],"fun_headline_variants":["Best LLM fills 20.7% of enterprise slots in new benchmark","Enterprise slot filling: LLMs nail just 20.7% in new benchmark","Reasoning LLMs beat fast ones by 16 points on enterprise slots","ESF-Bench: 810 chats reveal LLM slot filling blind spots"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The correctness of the 6,530 synthetic ground-truth labels rests almost entirely on GPT-5-generated labels verified by a GPT-5 judge, with only 122 slots (1.9%) human-checked; if the judge accepts wrong labels, the reported model scores and human comparison do not reflect true performance.","fun_headline_variants_meta":{"raw":{"variants":["Best LLM fills 20.7% of enterprise slots in new benchmark","Enterprise slot filling: LLMs nail just 20.7% in new benchmark","Reasoning LLMs beat fast ones by 16 points on enterprise slots","ESF-Bench: 810 chats reveal LLM slot filling blind spots"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000969,"raw_usage":{"total_tokens":3928,"prompt_tokens":684,"completion_tokens":3244,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":3161}},"tokens_in":428,"tokens_out":3244,"duration_ms":22225,"temperature":1.0,"reasoning_tokens":3161,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-31T23:42:36.398055+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random sample of ESF-Bench conversations, have independent human annotators produce ground truth from scratch, and compare LLM performance on those samples against the paper's reported F-JGA. If human-verified accuracy on the sample differs substantially from the GPT-judged accuracy, the benchmark's scores would not be trustworthy.","supporting_citations":[],"review_version":1}