REVIEW 4 major objections 7 minor 7 references
Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5
T0 review · 4 major / 7 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read Exaone 3.5, an enterprise-focused LLM, answers simple SQL aggregation questions about LG Electronics sales with 93% accuracy but only 4% on arithmetic-reasoning queries and 31% on grouped-ranking queries, according to a new…
desk verdict Plausible qualitative finding about Exaone 3.5's SQL weaknesses, but the reported numbers are internally inconsistent and the H4 evaluation method doesn't measure ranked-list correctness; worth a look only after major revision. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central machinery is the Fact-Consistency Checker pipeline: a schema-constrained system prompt guides Exaone 3.5 (temperature 0.1) to generate one SQL per natural-language question; the SQL is executed against BigQuery; the first cell of the first row is extracted as the predicted business answer; and both predicted and gold-standard answers are normalized and compared for exact match. This three-stage comparison (answer extraction, normalization, exact-match labeling) is what turns raw SQL execution into a fact-consistency measurement, and it is also the component whose extraction rule carries the evaluation's validity.
What would settle it
Rerun the H4 questions with a rank- or set-aware answer comparison that checks whether the generated query returns the same ranked rows as the gold-standard query; if the generated queries frequently contain the correct rows in a different order or with a different first cell, the reported 31% accuracy is an artifact of the single-cell extraction rule. Similarly, recompute H1 accuracy after manually verifying the first cell against the full answer row.
Extended reading notes
Core claim
On its own terms, the paper's discovery is a measured performance profile: Exaone 3.5 executes 94% of simple queries (L1) successfully and answers 93% correctly, but on high-complexity arithmetic queries (H1) execution success falls to 59% and answer accuracy to 4%, with a 54% semantic error rate; on grouped-ranking queries (H4) execution success is 69%, answer accuracy 31%, and semantic errors 37%. The paper also discovers that most failures are not parse errors but semantically valid queries that compute the wrong thing—misapplied ratios, reversed numerator/denominator, missing GROUP BY keys, wrong ORDER BY or LIMIT, omitted filters—so execution-based evaluation alone would overstate reliability. These findings support the framework's premise that fact-level answer comparison is necessary for assessing LLM-generated SQL in business-critical contexts.
Load-bearing premise
The evaluation assumes every business question is correctly answered by the first cell of the first row returned by the SQL, which is questionable for grouped-ranking questions whose true answer is an ordered list of rows.
Editorial extensions
If this is right
- If the benchmark results hold, Exaone 3.5 can be trusted for routine aggregation queries in BI settings but must be paired with a verification layer for any query involving post-aggregation arithmetic or ranked output.
- The 54% semantic error rate in H1 implies that execution success is a poor proxy for correctness; BI pipelines should compare generated answers against validated ground truths rather than only checking query validity.
- The framework offers a reproducible 219-question benchmark that other LLMs can be scored on, enabling model-to-model comparison in enterprise text-to-SQL.
- Non-response behavior in H1 and H4 suggests that the model sometimes abstains when uncertain, which is preferable to returning a confident wrong answer in high-stakes business decisions.
Reading between the lines
- A likely consequence the author leaves implicit is that the single-cell extraction rule inflates measured difficulty for list- and ranking-style questions; a rank-aware comparison would likely change the H4 numbers and give a more accurate picture of the model's true fact-consistency.
- Because the benchmark is skewed toward L1 questions (170 of 219), overall accuracy is dominated by trivial cases; readers should rely on per-category numbers when deciding where Exaone 3.5 can be deployed.
- A natural extension would be to run the same 219-question benchmark on other bilingual or instruction-tuned LLMs; if the large gap between simple and arithmetic queries persists across models, the bottleneck is architectural rather than specific to Exaone 3.5.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a fact-consistency evaluation framework for text-to-SQL generation in business intelligence. The authors build a domain benchmark from LG Electronics sales data in BigQuery, organize it into five complexity levels (L1 simple aggregation; H1 arithmetic reasoning; H2 compound filter logic; H3 conditional aggregation; H4 grouped ranking), generate SQL for each question with Exaone 3.5 at temperature 0.1, execute the generated queries, extract the first cell of the first row as the predicted answer, and compare it against normalized gold answers. They report per-category execution success, answer accuracy, semantic error, and non-response rates, with headline results of 93% answer accuracy on L1, 4% on H1, and 31% on H4, followed by qualitative error analysis identifying arithmetic, grouping/ranking, and filter-omission failures. The paper concludes that Exaone 3.5 is reliable for simple aggregation but degrades substantially on multi-step reasoning, motivating fact-consistency validation layers and hybrid approaches.
Significance. If the reported numbers were trustworthy, this would be a useful, domain-grounded data point: a realistic enterprise dataset with a sensible complexity taxonomy, an evaluation protocol that moves beyond string matching and raw execution accuracy, and an honest account of failure modes, including a thoughtful interpretation of non-response as potentially appropriate abstention in high-stakes settings. The protocol contains no fitted parameters, so there is no circularity concern; the near-complementarity of accuracy and semantic error is definitional under the paper's binary Correct/Incorrect checker. The concrete per-category performance profile is in principle falsifiable. The main weakness is that the quantitative claims are compromised by multiple internal inconsistencies: the benchmark size (219) contradicts the category counts (328) and the wording of Section 1, the Overall row of Table 1 cannot be reproduced from the category rows, and the metric definitions do not match the table. The claimed reproducibility is also not verifiable because no dataset, code, or example questions are provided.
major comments (4)
- [Abstract; §1; §3.1; Table 1] The benchmark size is reported inconsistently: the abstract and §3.1/§6 state 219 questions, §1 says “over 300”, and the per-category counts in §3.1 (170 L1 + 46 H1 + 38 H2 + 39 H3 + 35 H4) sum to 328. These cannot all be correct. The discrepancy is load-bearing because the Overall row of Table 1 (82% execution success, 61% accuracy, 21% semantic error, 18% non-response) cannot be reproduced from the category rows under any weighting consistent with the stated counts: count-weighted with the 328 total gives approximately 85.5% execution success, and equal weighting gives approximately 80.4%; neither equals 82%. Please reconcile the total count, report the true per-category N, and recompute all overall figures from the raw data.
- [§3.3; §3.4 with §3.1 (H4) and Table 1] The answer-extraction rule (“typically the first cell of the first row”) is not a valid extraction for the H4 category, which §3.1 defines as GROUP BY with ORDER BY and LIMIT, i.e., questions whose gold answer is a ranked set of rows. Comparing only the first cell of the first row of the generated result against the gold first row measures at most whether the top-ranked value matches; it does not measure whether the ranked answer is fact-consistent. Consequently, the reported 31% H4 answer accuracy does not establish the paper's grouped-ranking degradation claim, and the §4.1 grouping/ranking error analysis (missing grouping keys, incorrect sorting, LIMIT misapplication) is qualitative rather than supported by the stated metric. The authors must specify the gold-answer format for each category and re-score H4 with a rank-aware or set-aware matching criterion, or explicitly restrict their claims to top-1 correctness.
- [§3.5; Table 1] The metric definitions are inconsistent with Table 1. §3.5 defines answer accuracy as the percentage among successfully executed queries and semantic error rate as the percentage among executed queries; given §3.4's binary Correct/Incorrect labeling, these two rates must sum to 100% within each category. Table 1 instead reports 93+1=94 for L1 and 31+37=68 for H4 (against listed success rates of 94 and 69), so at least one of the definitions or the numbers is wrong. The rows of Table 1 would be coherent if accuracy and semantic error were percentages of all questions (they then sum to 100 with non-response, e.g., L1: 93+1+6=100), but that contradicts the text. Please restate the denominator of every metric explicitly, recompute the table accordingly, and clarify whether Overall is a weighted or unweighted average.
- [§1; §6] The paper claims to contribute a “reproducible benchmark” and evaluation methodology, but no dataset, question list, gold SQL, code, prompts, or per-question results are included or linked. Without these artifacts, none of the reported percentages can be independently checked or reproduced, and the claimed contribution is not verifiable from the manuscript. At minimum, provide an artifact or availability appendix with the question set, category labels, gold answers, and model outputs needed to reconstruct Table 1.
minor comments (7)
- [Table 1, H1 and H4 rows] In the H1 and H4 rows, accuracy + semantic error + non-response equals 99 rather than 100 (4+54+41 and 31+37+31), presumably due to rounding; reporting the raw counts would remove this ambiguity.
- [§4.1; Table 1] The H1 accuracy of 4% on the stated 46 questions corresponds to roughly one or two correct answers; given this small sample, the headline 4% figure should be reported with its raw counts and an appropriate caveat about fragility.
- [§3.1] The “validated ground-truth answer” is asserted but no validation protocol (annotator count, agreement, or adjudication) is described; please state how the gold answers were validated.
- [§3.3] The handling of empty query results (generated SQL returning zero rows) in the Fact-Consistency Checker is unspecified; please state the labeling rule for this case.
- [§3.2] Results are reported from a single generation run at temperature 0.1; please state whether multiple runs were averaged and whether a fixed seed was used.
- [References] References [5] and [6] contain garbled author strings (e.g., “Sharan Narang Wang” in [6]); the bibliography should be corrected.
- [§3.1] Including one representative natural language question and its gold SQL per category would clarify the intended answer formats and help readers assess the L1–H4 taxonomy.
Circularity Check
No circularity: the core results are direct benchmark measurements against manually constructed gold SQL and ground-truth answers; the flagged issues are construct-validity or data-consistency concerns, not reductions to the paper's own inputs.
full rationale
The empirical chain is a direct benchmark rather than a derivation from an assumed premise. The benchmark questions, gold-standard SQL, and ground-truth answers are manually constructed (Section 3.1); Exaone only generates candidate SQL (Section 3.2); and the final judgment compares the executed result with the gold answer after normalization (Sections 3.3-3.4). The reported rates are counts, not outputs of a fitted model, and no parameter is tuned on a subset and then used to derive a close prediction. The only citation tied to the same enterprise is the LG AI Research model card for Exaone 3.5, used descriptively (Section 2.3) and not load-bearing for any evaluation outcome. Two manuscript issues are validity risks rather than circular reductions: (i) Section 3.3's 'typically the first cell of the first row' means H4 grouped-ranking accuracy tests only the top returned cell, not a full ranked list, so the 31% figure must be interpreted as top-element accuracy; (ii) Table 1's semantic error rate is numerically close to 100 percent minus answer accuracy minus non-response, so it is a complement of the same exact-match signal rather than an independent semantic label. The paper also reports inconsistent benchmark sizes (219 vs. over 300 vs. 328 by category sum), a data-quality problem. None of these makes the central accuracy result equivalent to its inputs by construction, so the paper is not circular.
Assumptions & free parameters
assumptions (4)
- domain assumption The hand-written gold-standard SQL and ground-truth answers in the benchmark are correct.
- domain assumption Exaone 3.5 output at temperature 0.1 is deterministic enough for a single evaluation pass.
- ad hoc to paper Exact string match after normalization equals factual consistency.
- ad hoc to paper The first cell of the first row of the query result is the correct answer for every question.
Cite this review
Pith. "Pith review of Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5." pith.science (2026). https://pith.science/paper/ZN2327EB
@misc{pith2026250500060,
author = {Pith},
title = {Pith review of: Fact-Consistency Evaluation of Text-to-SQL Generation for Business Intelligence Using Exaone 3.5},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZN2327EB}},
note = {Machine review of arXiv:2505.00060}
}
read the original abstract
Large Language Models (LLMs) have shown promise in enabling natural language interfaces for structured data querying through text-to-SQL generation. However, their application in real-world Business Intelligence (BI) contexts remains limited due to semantic hallucinations, structural errors, and a lack of domain-specific evaluation frameworks. In this study, we propose a Fact-Consistency Evaluation Framework for assessing the semantic accuracy of LLM-generated SQL outputs using Exaone 3.5--an instruction-tuned, bilingual LLM optimized for enterprise tasks. We construct a domain-specific benchmark comprising 219 natural language business questions across five SQL complexity levels, derived from actual sales data in LG Electronics' internal BigQuery environment. Each question is paired with a gold-standard SQL query and a validated ground-truth answer. We evaluate model performance using answer accuracy, execution success rate, semantic error rate, and non-response rate. Experimental results show that while Exaone 3.5 performs well on simple aggregation tasks (93% accuracy in L1), it exhibits substantial degradation in arithmetic reasoning (4% accuracy in H1) and grouped ranking tasks (31% in H4), with semantic errors and non-responses concentrated in complex cases. Qualitative error analysis further identifies common failure types such as misapplied arithmetic logic, incomplete filtering, and incorrect grouping operations. Our findings highlight the current limitations of LLMs in business-critical environments and underscore the need for fact-consistency validation layers and hybrid reasoning approaches. This work contributes a reproducible benchmark and evaluation methodology for advancing reliable natural language interfaces to structured enterprise data systems.
Reference graph
Works this paper leans on
-
[1]
Seq2sql: Gener ating structured queries from natural language using reinforcement learning
Victor Zhong, Caiming Xiong, and Richard Socher. Seq2sql: Gener ating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103 , 2017
arXiv 2017
-
[2]
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, and Percy Liang. Spider: A large-scale human-labeled dataset for complex and cross -domain semantic parsing and text-to-sql task. arXiv preprint arXiv:1809.08887 , 2018
arXiv 2018
-
[3]
Syntaxsqlnet: Syn- tax tree networks for complex and cross-domain text-to-sql ta sk
Tao Yu, Bin Xue, Rui Zhang, Shangqing Zha, Yuwen Wang, and Xi V ictoria Li. Syntaxsqlnet: Syn- tax tree networks for complex and cross-domain text-to-sql ta sk. arXiv preprint arXiv:1810.05237 , 2018
arXiv 2018
-
[4]
Bi-fidelity Variational Auto-encoder for Uncertainty Quantification
Yusen Liu and Dragomir Radev. Evaluating the factual consisten cy of text-to-sql generation. arXiv preprint arXiv:2305.16530, 2023
work page Pith review arXiv 2023
-
[5]
Teql: Text- to-sql evaluation using question logic
Liunian Harold Chen, Yuxiang Li, Caiming Xiong, Kai-Wei Chang, and Y izhou Wang. Teql: Text- to-sql evaluation using question logic. arXiv preprint arXiv:2305.14430 , 2023
arXiv 2023
-
[6]
Scaling instruction-finetuned language models
Hyung Won Chung, Le Hou, Shayne Longpre, Barret Zoph, Yi Ta y, William Fedus, Sharan Narang Wang, et al. Scaling instruction-finetuned language models. arXiv preprint arXiv:2210.11416, 2022
arXiv 2022
-
[7]
Exaone 3.5: Series of large language models for r eal-world use cases
LG AI Research. Exaone 3.5: Series of large language models for r eal-world use cases. arXiv preprint arXiv:2412.04862, 2024. 6
arXiv 2024
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.