{"id":"ea66ce32-4192-4895-8765-24316b940e70","arxiv_id":"2505.00060","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Exaone 3.5 answers simple SQL aggregation questions on LG sales data with 93% accuracy but drops to 4% on arithmetic-reasoning queries.","lead":"The paper tests how well the Exaone 3.5 language model turns plain-language business questions into SQL queries on LG Electronics sales data. Simple aggregation questions are answered accurately, but arithmetic and ranking questions often fail, which matters for companies considering LLM-based business intelligence.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"H4 accuracy is not measuring ranked-list correctness because Section 3.3 extracts only the first cell of the first row; re-scoring with rank-aware matching is needed before the 31% figure can be interpreted.","rationale":"The reader's verdict is CONDITIONAL, and my stress-test agrees. The most load-bearing assumption in the paper is the answer-extraction rule in Section 3.3: using the first cell of the first row as the answer. This is a reasonable simplification for scalar-answer questions, and it is probably fine for L1/H1/H2/H3 where the business answer is a single number or label. But H4 is explicitly a grouped-ranking category (GROUP BY + ORDER BY + LIMIT), so the correct answer is a ranked set of rows. The paper never explains how a ranked set is reduced to one cell, and no sample instances are provided. If the gold answers are stored as lists, then the reported 31% H4 accuracy is not a measure of fact consistency for the actual question; it is a measure of first-cell agreement, which conflates partial, top-item-only matches with full correct answers. This directly weakens one of the two headline degradation claims in the abstract. The count inconsistency (219 vs 328 vs 'over 300') is a separate reproducibility concern, but it is less directly tied to the validity of the H4 measurement. The proposed concrete test--inspect the H4 gold rows and re-score with rank-aware matching--would settle the extraction issue. If list-match accuracy is close to 31%, the concern is resolved; if it differs materially, the paper's central H4 claim needs to be revised. I therefore keep the CONDITIONAL verdict: the qualitative L1/H1 pattern is plausible, but the grouped-ranking result is not yet measured in a valid way, and the benchmark numbers need clarification and release.","tokens_in":5270,"tokens_out":4236,"duration_ms":39928,"concrete_test":"Take the 35 H4 questions and retrieve the full result sets of the gold-standard SQL and of the Exaone-generated SQL for each. Manually classify each H4 gold answer as scalar (one cell) or multi-row ranked list. Then re-score H4 under an order-sensitive list match (e.g., exact top-N row match, with normalization as in Section 3.4) and compare that accuracy with the published first-cell 31%. If the list-match accuracy differs by more than, say, 5 percentage points, the published H4 figure is an artifact of the Section 3.3 extraction rule; if it is within 5 points, the extraction rule is probably harmless for this benchmark. Also re-derive Table 1 denominators from the released question list to confirm whether 219 or 328 is the true sample size.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim depends on the reported per-category answer-accuracy numbers, and the most load-bearing threat is the answer-extraction rule in Section 3.3. The paper says the 'primary answer' is 'typically the first cell of the first row,' and Section 3.4 compares only that extracted cell to a normalized ground truth. For L1/H1/H2/H3 the desired answer may indeed be a scalar, so first-cell extraction is plausible. But H4 is defined as GROUP BY with ORDER BY and LIMIT (Section 3.1), i.e., a ranked set of rows. If the gold answer for an H4 question is a ranked list, comparing only the first cell of the first row measures at most whether the top-ranked item is correct, not whether the full ranked answer is fact-consistent. The reported 31% H4 accuracy therefore does not yet establish the paper's grouped-ranking degradation claim; it could be higher or lower under a proper set/list match. A second, independent red flag is that the benchmark size is given as 219 in the abstract and Sections 3.1 and 6, 'over 300' in Section 1, and 328 when summing the per-category counts (170+46+38+39+35), so the denominators behind Table 1 are also uncertain. Neither issue by itself proves the qualitative pattern wrong, but both must be resolved before the headline numbers can be accepted.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a fact-consistency evaluation framework for text-to-SQL generation in business intelligence. The authors build a domain benchmark from LG Electronics sales data in BigQuery, organize it into five complexity levels (L1 simple aggregation; H1 arithmetic reasoning; H2 compound filter logic; H3 conditional aggregation; H4 grouped ranking), generate SQL for each question with Exaone 3.5 at temperature 0.1, execute the generated queries, extract the first cell of the first row as the predicted answer, and compare it against normalized gold answers. They report per-category execution success, answer accuracy, semantic error, and non-response rates, with headline results of 93% answer accuracy on L1, 4% on H1, and 31% on H4, followed by qualitative error analysis identifying arithmetic, grouping/ranking, and filter-omission failures. The paper concludes that Exaone 3.5 is reliable for simple aggregation but degrades substantially on multi-step reasoning, motivating fact-consistency validation layers and hybrid approaches.","tokens_in":5492,"tokens_out":20230,"duration_ms":191348,"significance":"If the reported numbers were trustworthy, this would be a useful, domain-grounded data point: a realistic enterprise dataset with a sensible complexity taxonomy, an evaluation protocol that moves beyond string matching and raw execution accuracy, and an honest account of failure modes, including a thoughtful interpretation of non-response as potentially appropriate abstention in high-stakes settings. The protocol contains no fitted parameters, so there is no circularity concern; the near-complementarity of accuracy and semantic error is definitional under the paper's binary Correct/Incorrect checker. The concrete per-category performance profile is in principle falsifiable. The main weakness is that the quantitative claims are compromised by multiple internal inconsistencies: the benchmark size (219) contradicts the category counts (328) and the wording of Section 1, the Overall row of Table 1 cannot be reproduced from the category rows, and the metric definitions do not match the table. The claimed reproducibility is also not verifiable because no dataset, code, or example questions are provided.","major_comments":[{"comment":"The benchmark size is reported inconsistently: the abstract and §3.1/§6 state 219 questions, §1 says “over 300”, and the per-category counts in §3.1 (170 L1 + 46 H1 + 38 H2 + 39 H3 + 35 H4) sum to 328. These cannot all be correct. The discrepancy is load-bearing because the Overall row of Table 1 (82% execution success, 61% accuracy, 21% semantic error, 18% non-response) cannot be reproduced from the category rows under any weighting consistent with the stated counts: count-weighted with the 328 total gives approximately 85.5% execution success, and equal weighting gives approximately 80.4%; neither equals 82%. Please reconcile the total count, report the true per-category N, and recompute all overall figures from the raw data.","section":"Abstract; §1; §3.1; Table 1"},{"comment":"The answer-extraction rule (“typically the first cell of the first row”) is not a valid extraction for the H4 category, which §3.1 defines as GROUP BY with ORDER BY and LIMIT, i.e., questions whose gold answer is a ranked set of rows. Comparing only the first cell of the first row of the generated result against the gold first row measures at most whether the top-ranked value matches; it does not measure whether the ranked answer is fact-consistent. Consequently, the reported 31% H4 answer accuracy does not establish the paper's grouped-ranking degradation claim, and the §4.1 grouping/ranking error analysis (missing grouping keys, incorrect sorting, LIMIT misapplication) is qualitative rather than supported by the stated metric. The authors must specify the gold-answer format for each category and re-score H4 with a rank-aware or set-aware matching criterion, or explicitly restrict their claims to top-1 correctness.","section":"§3.3; §3.4 with §3.1 (H4) and Table 1"},{"comment":"The metric definitions are inconsistent with Table 1. §3.5 defines answer accuracy as the percentage among successfully executed queries and semantic error rate as the percentage among executed queries; given §3.4's binary Correct/Incorrect labeling, these two rates must sum to 100% within each category. Table 1 instead reports 93+1=94 for L1 and 31+37=68 for H4 (against listed success rates of 94 and 69), so at least one of the definitions or the numbers is wrong. The rows of Table 1 would be coherent if accuracy and semantic error were percentages of all questions (they then sum to 100 with non-response, e.g., L1: 93+1+6=100), but that contradicts the text. Please restate the denominator of every metric explicitly, recompute the table accordingly, and clarify whether Overall is a weighted or unweighted average.","section":"§3.5; Table 1"},{"comment":"The paper claims to contribute a “reproducible benchmark” and evaluation methodology, but no dataset, question list, gold SQL, code, prompts, or per-question results are included or linked. Without these artifacts, none of the reported percentages can be independently checked or reproduced, and the claimed contribution is not verifiable from the manuscript. At minimum, provide an artifact or availability appendix with the question set, category labels, gold answers, and model outputs needed to reconstruct Table 1.","section":"§1; §6"}],"minor_comments":[{"comment":"In the H1 and H4 rows, accuracy + semantic error + non-response equals 99 rather than 100 (4+54+41 and 31+37+31), presumably due to rounding; reporting the raw counts would remove this ambiguity.","section":"Table 1, H1 and H4 rows"},{"comment":"The H1 accuracy of 4% on the stated 46 questions corresponds to roughly one or two correct answers; given this small sample, the headline 4% figure should be reported with its raw counts and an appropriate caveat about fragility.","section":"§4.1; Table 1"},{"comment":"The “validated ground-truth answer” is asserted but no validation protocol (annotator count, agreement, or adjudication) is described; please state how the gold answers were validated.","section":"§3.1"},{"comment":"The handling of empty query results (generated SQL returning zero rows) in the Fact-Consistency Checker is unspecified; please state the labeling rule for this case.","section":"§3.3"},{"comment":"Results are reported from a single generation run at temperature 0.1; please state whether multiple runs were averaged and whether a fixed seed was used.","section":"§3.2"},{"comment":"References [5] and [6] contain garbled author strings (e.g., “Sharan Narang Wang” in [6]); the bibliography should be corrected.","section":"References"},{"comment":"Including one representative natural language question and its gold SQL per category would clarify the intended answer formats and help readers assess the L1–H4 taxonomy.","section":"§3.1"}],"recommendation":"major_revision","confidential_remarks":"To the editor: the paper addresses a genuinely useful problem, and the qualitative pattern (strong on simple aggregation, weak on arithmetic and grouped ranking) is plausible. However, the quantitative reporting is not in a publishable state: there are at least three independent internal inconsistencies (benchmark size 219 vs 328 vs “over 300”; the Overall row of Table 1 not derivable from the category rows; metric definitions contradicting Table 1), plus the H4 answer-extraction validity problem. These are all correctable within the paper's scope because the authors hold the underlying data, so I recommend major revision rather than rejection. I would ask the editor to require that the authors supply the full question manifest and per-question results for review, and to insist on an artifact or availability statement given the paper's explicit “reproducible benchmark” claim. I see no evidence of anything beyond reporting carelessness, but the number of independent inconsistencies is high enough that the raw data should be examined before the headline numbers are accepted."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's the punchline: the paper's qualitative claim is plausible — Exaone 3.5 handles simple aggregation well but struggles with arithmetic and grouped-ranking SQL — but the numbers behind it don't add up, and the evaluation rule for ranking queries is mismatched to the task. I'd send it to peer review only with the expectation that the authors fix the arithmetic and clarify the method.\n\nWhat's genuinely new: the benchmark is grounded in LG Electronics' real BigQuery sales data, with manually verified gold SQL and answers. That's a useful departure from Spider/WikiSQL. The error taxonomy (arithmetic failures, grouping/ranking errors, condition omissions) is sensible and matches the examples. Choosing Exaone 3.5, a real enterprise-focused model, also gives the study practical relevance.\n\nThe soft spots, in rough order of severity:\n\n1. The dataset size is contradictory. The abstract and Section 3.1 say 219 questions, but the five category counts (170, 46, 38, 39, 35) sum to 328. The introduction says 'over 300.' The overall row in Table 1 doesn't match any weighted combination of the per-category rows using either denominator. This is load-bearing: every headline percentage is suspect.\n\n2. H4 accuracy is not measuring ranked-list correctness. Section 3.3 says the 'primary answer' is 'typically the first cell of the first row.' But H4 is defined as GROUP BY with ORDER BY and LIMIT — the correct answer is a ranked set, not a scalar. Comparing only the top cell tells us at most whether the top-ranked item is right, not whether the full ranking matches. The 31% figure can't be interpreted as ranked-list accuracy.\n\n3. No baselines and no released data/code. A single model with no comparison makes the performance numbers uncontextualized, and the abstract's claim of a 'reproducible benchmark' is false without release of the questions and gold SQL.\n\nThese issues are not fatal to the qualitative direction, and the paper does not appear to be dishonest — but the quantitative backbone is not reliable as written. The authors need to fix the count, re-evaluate H4 with a proper list match, add at least one baseline model, and release a sample of the data. The topic is relevant, and the underlying idea is sound enough to deserve a referee's time after substantial revision.","headline":"Plausible qualitative finding about Exaone 3.5's SQL weaknesses, but the reported numbers are internally inconsistent and the H4 evaluation method doesn't measure ranked-list correctness; worth a look only after major revision.","tokens_in":6034,"tokens_out":5455,"would_cite":false,"duration_ms":54296,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Exaone 3.5, an enterprise-focused LLM, answers simple SQL aggregation questions about LG Electronics sales with 93% accuracy but only 4% on arithmetic-reasoning queries and 31% on grouped-ranking queries, according to a new…","keywords":["text-to-SQL","fact-consistency evaluation","business intelligence","Exaone 3.5","semantic error analysis","enterprise benchmark","SQL complexity levels","LLM evaluation"],"falsifier":"Rerun the H4 questions with a rank- or set-aware answer comparison that checks whether the generated query returns the same ranked rows as the gold-standard query; if the generated queries frequently contain the correct rows in a different order or with a different first cell, the reported 31% accuracy is an artifact of the single-cell extraction rule. Similarly, recompute H1 accuracy after manually verifying the first cell against the full answer row.","tokens_in":4989,"feed_emoji":"📊","tokens_out":6016,"duration_ms":52725,"temperature":0.7,"pith_summary":"The paper tries to establish that a fact-consistency evaluation framework—comparing executed answers from generated SQL against validated business ground truths—can measure how trustworthy an LLM-based text-to-SQL system is in real enterprise settings. Applying it to Exaone 3.5 on 219 questions built from LG Electronics' internal sales data, the paper finds near-production-grade reliability for simple aggregations (93% L1 accuracy) but severe breakdowns when questions require arithmetic reasoning (4% H1 accuracy) or grouped ranking with LIMIT (31% H4 accuracy). The point of the study is that enterprise BI needs semantic validation beyond execution success or string matching, because most failures are syntactically valid SQL that returns the wrong business answer.","feed_headline":"Text-to-SQL model hits 93% on simple queries, 4% on arithmetic","feed_subtitle":"New business benchmark shows the gap between routine aggregation SQL and multi-step reasoning.","key_machinery":"The central machinery is the Fact-Consistency Checker pipeline: a schema-constrained system prompt guides Exaone 3.5 (temperature 0.1) to generate one SQL per natural-language question; the SQL is executed against BigQuery; the first cell of the first row is extracted as the predicted business answer; and both predicted and gold-standard answers are normalized and compared for exact match. This three-stage comparison (answer extraction, normalization, exact-match labeling) is what turns raw SQL execution into a fact-consistency measurement, and it is also the component whose extraction rule carries the evaluation's validity.","core_discovery":"On its own terms, the paper's discovery is a measured performance profile: Exaone 3.5 executes 94% of simple queries (L1) successfully and answers 93% correctly, but on high-complexity arithmetic queries (H1) execution success falls to 59% and answer accuracy to 4%, with a 54% semantic error rate; on grouped-ranking queries (H4) execution success is 69%, answer accuracy 31%, and semantic errors 37%. The paper also discovers that most failures are not parse errors but semantically valid queries that compute the wrong thing—misapplied ratios, reversed numerator/denominator, missing GROUP BY keys, wrong ORDER BY or LIMIT, omitted filters—so execution-based evaluation alone would overstate reliability. These findings support the framework's premise that fact-level answer comparison is necessary for assessing LLM-generated SQL in business-critical contexts.","pith_inferences":["A likely consequence the author leaves implicit is that the single-cell extraction rule inflates measured difficulty for list- and ranking-style questions; a rank-aware comparison would likely change the H4 numbers and give a more accurate picture of the model's true fact-consistency.","Because the benchmark is skewed toward L1 questions (170 of 219), overall accuracy is dominated by trivial cases; readers should rely on per-category numbers when deciding where Exaone 3.5 can be deployed.","A natural extension would be to run the same 219-question benchmark on other bilingual or instruction-tuned LLMs; if the large gap between simple and arithmetic queries persists across models, the bottleneck is architectural rather than specific to Exaone 3.5."],"forward_implications":["If the benchmark results hold, Exaone 3.5 can be trusted for routine aggregation queries in BI settings but must be paired with a verification layer for any query involving post-aggregation arithmetic or ranked output.","The 54% semantic error rate in H1 implies that execution success is a poor proxy for correctness; BI pipelines should compare generated answers against validated ground truths rather than only checking query validity.","The framework offers a reproducible 219-question benchmark that other LLMs can be scored on, enabling model-to-model comparison in enterprise text-to-SQL.","Non-response behavior in H1 and H4 suggests that the model sometimes abstains when uncertain, which is preferable to returning a confident wrong answer in high-stakes business decisions."],"supporting_citations":[{"why":"Identifies the model under test—Exaone 3.5, its instruction-tuning, bilingual capability, and context length—as the system whose SQL generation the benchmark evaluates.","marker":"[7]"},{"why":"Supplies the fact-consistency evaluation idea that the framework's answer-comparison approach builds on, directly motivating the semantic-level metric.","marker":"[4]"},{"why":"Provides WikiSQL, one of the standard execution-accuracy benchmark datasets that the paper argues are insufficient for real-world BI evaluation.","marker":"[1]"},{"why":"Provides Spider, the cross-domain text-to-SQL benchmark against which the paper positions its domain-specific enterprise benchmark.","marker":"[2]"}],"fun_headline_variants":["Text-to-SQL: 93% on easy, 4% on arithmetic reasoning","LLM SQL accuracy drops from 93% to 4% on complex tasks","Semantic errors, not syntax, sink text-to-SQL on hard queries","Exaone 3.5: simple SQL 93%, arithmetic 4% — semantic gap"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes every business question is correctly answered by the first cell of the first row returned by the SQL, which is questionable for grouped-ranking questions whose true answer is an ordered list of rows.","fun_headline_variants_meta":{"raw":{"variants":["Text-to-SQL: 93% on easy, 4% on arithmetic reasoning","LLM SQL accuracy drops from 93% to 4% on complex tasks","Semantic errors, not syntax, sink text-to-SQL on hard queries","Exaone 3.5: simple SQL 93%, arithmetic 4% — semantic gap"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000371,"raw_usage":{"total_tokens":2022,"prompt_tokens":1018,"completion_tokens":1004,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":634,"completion_tokens_details":{"reasoning_tokens":913}},"tokens_in":634,"tokens_out":1004,"duration_ms":9699,"temperature":1.0,"reasoning_tokens":913,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:55:14.851294+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rerun the H4 questions with a rank- or set-aware answer comparison that checks whether the generated query returns the same ranked rows as the gold-standard query; if the generated queries frequently contain the correct rows in a different order or with a different first cell, the reported 31% accuracy is an artifact of the single-cell extraction rule. Similarly, recompute H1 accuracy after manually verifying the first cell against the full answer row.","supporting_citations":[{"cited_title":"Bi-fidelity Variational Auto-encoder for Uncertainty Quantification","cited_arxiv_id":"2305.16530","evidence_quote":"Supplies the fact-consistency evaluation idea that the framework's answer-comparison approach builds on, directly motivating the semantic-level metric."}],"review_version":1}