{"id":"ca390608-03d2-4df1-94b7-dffc59da7c38","arxiv_id":"2412.05208","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":0.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A high-level review of benchmarks, models, applications, and challenges in LLM-based text-to-SQL, with no new experiments or methods.","lead":"This paper surveys text-to-SQL systems, which use large language models to turn plain-English questions into SQL database queries. It lists benchmarks, models, applications, and open challenges, but it does not run any new experiments.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table I contains internally inconsistent benchmark numbers (SQLova 95% vs X-SQL 91.8% despite X-SQL being called state-of-the-art), undermining the central claim of a comprehensive, reliable overview.","rationale":"The Pith Reader's weakest assumption was that the paper's secondhand descriptions are faithful and its benchmark/model selection is representative. My concern sharpens that assumption into a specific, checkable failure: Table I contains an internally contradictory pair of accuracy numbers, and the MedT5SQL architecture description contradicts its cited source. Because the paper labels itself 'comprehensive' and uses Table I as its main comparative evidence, these errors directly undercut the strongest claim rather than merely affecting an auxiliary detail. The paper could still become a useful reference after correcting the table, adding quantitative entries, and justifying the selection, which is exactly the CONDITIONAL verdict the Reader gave. I therefore keep the verdict unchanged while providing a concrete audit step that would confirm or refute the concern. No issue is raised with author conduct; the problem is with the accuracy and reproducibility of the compiled data. I also note that Figure 1 claims a methodology but the text does not specify inclusion criteria, which compounds the difficulty of auditing the 'comprehensive' claim, though the Table I errors are the more load-bearing issue.","tokens_in":7847,"tokens_out":5271,"duration_ms":55322,"concrete_test":"Verify Table I against the original papers: pull the WikiSQL accuracies reported in ref [17] (SQLova) and ref [19] (X-SQL). If SQLova's published accuracy is ~86% rather than 95%, or if X-SQL's is not lower than SQLova's, the table is wrong. Also check ref [16] to confirm whether MedT5SQL is based on T5 rather than BERT+LSTM. This single audit of Table I settles whether the survey's central factual apparatus is reliable.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's advertised value is a 'comprehensive overview' of text-to-SQL models, and Table I is the core evidence of model progress. That table is internally inconsistent: it lists SQLova at 95% on WikiSQL while listing X-SQL at 91.8% and simultaneously claims X-SQL achieves state-of-the-art on WikiSQL (Section IV-B). Both cannot be correct under the same metric. The original SQLova paper (ref [17]) reports around 86% logical-form accuracy on WikiSQL, and X-SQL (ref [19]) reports around 91.8%, so the 95% entry is either mis-sourced or uses an unstated, non-comparable metric. The table also describes MedT5SQL as 'BERT-based encoder with LSTM decoder,' whereas the cited source [16] is a T5-family transformer model. Several entries are non-quantitative ('High', 'Optimized'), and the selection omits widely used LLM methods such as DAIL-SQL and DIN-SQL. Because the survey's central claim is to be a complete, accurate reference, these concrete factual problems are more damaging than the absence of a formal search protocol.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript is a narrative survey of text-to-SQL systems, covering benchmarks and datasets (Spider, WikiSQL, BIRD, CoSQL, and others), a table of representative models, application domains (healthcare, education, finance, business intelligence), and a set of challenges and future directions centered on generalization, ambiguity, external knowledge, efficiency, interpretability, and NoSQL support. The abstract and introduction position the paper as a comprehensive overview of LLM-based text-to-SQL, and the paper's main deliverable is the reliability of its secondhand descriptions of prior work and the completeness of its coverage.","tokens_in":8050,"tokens_out":4077,"duration_ms":40369,"significance":"If corrected, the survey could serve as a concise entry point for readers new to text-to-SQL, particularly for its organization of industry applications and its discussion of the NoSQL dataset gap. The paper makes no new technical contribution, and its value rests entirely on the accuracy and completeness of the surveyed material. The claims of comprehensiveness are therefore load-bearing: errors in Table I, mischaracterizations of cited models, and omissions of influential LLM-based methods directly undercut the advertised contribution. The paper does not report a formal search protocol or inclusion criteria, which further limits the reproducibility of its coverage claims.","major_comments":[{"comment":"Table I lists SQLova at 95% on WikiSQL while listing X-SQL at 91.8%, and Section IV-B states that X-SQL achieves state-of-the-art performance on WikiSQL; these entries are mutually inconsistent under the same metric, and the cited source for SQLova [17] reports roughly 86% logical-form accuracy. Please report the exact metric (logical form vs. execution accuracy), cite the specific result, and ensure that the table is internally consistent.","section":"Table I / Section IV-B"},{"comment":"The MedT5SQL row and bullet describe the model as a \"BERT-based encoder with LSTM decoder,\" but the cited source [16] is a T5-family transformer model adapted for healthcare text-to-SQL. This mischaracterizes the architecture of a surveyed model and is a factual error in the central comparison table.","section":"Table I / Section IV-B"},{"comment":"Despite the title's focus on large language models, the model section omits several influential LLM-based text-to-SQL systems from the 2023-2024 period, including DIN-SQL and DAIL-SQL, and does not report the standard GPT-4 or Codex baselines on Spider or BIRD. Given the survey's advertised comprehensiveness, this omission is not merely cosmetic; it weakens the claimed coverage of LLM-based generative approaches.","section":"Section IV-B / Table I"},{"comment":"Table I mixes non-comparable metrics and qualitative labels (\"High\", \"Optimized\", \"37.4% IEX\", \"N/A\") without a metric column, so the rows cannot be compared as a progression of model performance. In addition, EHRSQL is listed as a model name even though Section IV-B describes it as a benchmark; the table should distinguish models from benchmarks and include a metric/version column.","section":"Table I"},{"comment":"Table III categorizes \"EDU-T5\" as a domain-specific SQL dataset, but Section IV-B presents EDU-T5 as a model; it also lists CoSQL under both \"Relational SQL\" and \"Dialogue-Based SQL.\" This inconsistent categorization makes the NoSQL-gap argument difficult to assess and should be corrected with dataset names and a clear taxonomy.","section":"Table III / Section IV-B"}],"minor_comments":[{"comment":"Figure 1 contains unfinished placeholder text (e.g., \"♂searchKeyword\", \"/brainExploration\", \"/da◎abaseDataset Analysis\", \"/char◎-pieEvaluation\") and should be redrawn as a proper methodology diagram.","section":"Figure 1"},{"comment":"The MedT5SQL bullet writes \"MedTS generates SQL queries\" instead of \"MedT5SQL\"; the model name should be used consistently.","section":"Section IV-B"},{"comment":"EDU-T5 is described as optimized for educational data but is cited to [15], which is the general T5 paper and does not describe an EDU-T5 model; the citation should be corrected or the EDU-T5 entry should be removed.","section":"Section IV-B / Reference [15]"},{"comment":"BIRD is described as containing \"12,751 question-SQL pairs across 95 databases,\" but no version or date is given for these statistics; please add the dataset version and a direct citation for each benchmark statistic.","section":"Section IV-A"},{"comment":"The survey does not state a search protocol, inclusion/exclusion criteria, or the date up to which literature was covered; adding a short methodology paragraph would improve the reproducibility of the coverage claims.","section":"Section I/II"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the scope of a survey venue but is currently below the bar for acceptance because its central comparative table contains factual errors and its coverage omits influential LLM-based methods. The issues appear correctable: every row of Table I should be verified against the cited paper, a metric column should be added, the model/dataset distinction should be enforced, and the LLM-era methods should be incorporated. There is no evidence of intentional misrepresentation, and the paper's application-level discussion has some value. A major revision with these corrections would make the survey serviceable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a passable orientation to text-to-SQL for people new to the area, but it is not a trustworthy reference in its current form. The survey's advertised value is comprehensiveness, and the core comparison table undercuts that claim.\n\nWhat's good: the paper lays out the main benchmark datasets (Spider, WikiSQL, BIRD, CoSQL, Spider 2.0, CSpider, UNITE) and sketches the model line from Seq2SQL through SQLova/RAT-SQL to PICARD. The applications section on healthcare, education, finance, and business intelligence is concrete, and the NoSQL gap discussion makes a legitimate point. The writing is clear and the structure is easy to follow. For a reader who wants a broad map before diving into primary sources, this could serve.\n\nWhere it falls short: Table I is not reliable. It gives SQLova 95% on WikiSQL while listing X-SQL at 91.8% and, in the text, calls X-SQL state-of-the-art; both cannot be true under the same metric, and the original SQLova paper reports around 86% logical-form accuracy. It describes MedT5SQL as a BERT encoder with LSTM decoder, but the cited paper is a T5-family transformer model. It lists EHRSQL, which is a benchmark, as a model. Several rows have only qualitative labels like \"High\" or \"Optimized.\" On coverage, DAIL-SQL and DIN-SQL, two widely used LLM-based methods, are missing. There is no systematic search protocol, so the selection looks ad hoc rather than comprehensive.\n\nBecause the advertised value of a survey is accurate synthesis, these are not cosmetic defects. A newcomer who trusts Table I will carry away wrong numbers and misattributions. That said, the paper is not incoherent; the errors are fixable, and the narrative around datasets, applications, and challenges is sensible.\n\nWho it's for: beginners and practitioners wanting a quick scan of the area. Not for people building on benchmark numbers. If the authors fix Table I, add the missing methods, and soften the comprehensiveness claim, it is publishable as a survey. I'd send it to peer review with major revisions, not desk-reject it, because the field can use a corrected entry-level survey.","headline":"A readable entry-level survey of text-to-SQL that is not yet trustworthy as a reference – the central comparison table contains concrete errors and the coverage has notable omissions.","tokens_in":8545,"tokens_out":3346,"would_cite":false,"duration_ms":33640,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Survey maps AI text-to-SQL landscape and its gaps","keywords":["text-to-SQL","large language models","natural language interfaces","database benchmarks","Spider","CoSQL","NoSQL databases","generative AI"],"falsifier":"A reader could check Table I against the original papers and the cited datasets; if several entries do not match the primary sources—for example the listed accuracy for SQLova or the architecture description for MedT5SQL—the survey's promise of a reliable overview is undercut.","tokens_in":7661,"feed_emoji":"🗄️","tokens_out":6390,"duration_ms":55550,"temperature":0.7,"pith_summary":"This survey sets out to give a broad, current picture of LLM-based text-to-SQL systems, which translate natural-language questions into SQL queries over databases. The authors assemble the field's foundational pipeline—natural language understanding, schema linking, semantic parsing, SQL generation—and the benchmark datasets such as Spider, WikiSQL, CoSQL, BIRD, and Spider 2.0 that measure progress. They catalogue representative models from Seq2SQL through T5-3B, PICARD, and domain-specific systems like MedT5SQL and EHRSQL. Their central message is that dedicated text-to-SQL systems outperform general-purpose AI for database interaction, and that the remaining obstacles are domain generalization, ambiguity handling, query efficiency, and the lack of NoSQL and dynamic multi-turn datasets. If the survey is right, a reader gets a reliable map of where the field stands and a concrete agenda for the next wave of research.","feed_headline":"Survey maps AI text-to-SQL landscape and its gaps","feed_subtitle":"Benchmarks and models from Spider to NoSQL, with the open problems that keep natural-language databases out of reach.","key_machinery":"The organizing structure is the four-stage text-to-SQL pipeline—natural language understanding, schema linking, semantic parsing, and SQL generation—used as a lens for classifying models and benchmarks. The survey also leans on its comparison table of models, datasets, training methods, and accuracy values to support the narrative of progress and to expose gaps such as the empty row for NoSQL dataset availability.","core_discovery":"The paper tries to establish that text-to-SQL has evolved from rule-based and sequence-to-sequence parsing into an LLM-driven capability whose progress is benchmarked largely by Spider, WikiSQL, and CoSQL, and whose next advances will come from specialized datasets and models rather than from general-purpose LLMs alone. It argues that the field now faces a set of named bottlenecks—cross-domain generalization, ambiguous natural language, integration of external domain knowledge, SQL efficiency, interpretability, and the absence of NoSQL benchmarks—and it maps each bottleneck to a future research direction. The survey positions itself as the reference that ties together foundational components, models, evaluation metrics, and application domains.","pith_inferences":["Because the survey's own table shows large accuracy gaps across benchmarks, a reader could infer that current text-to-SQL is reliable for narrow, well-schematized domains but far from dependable for open-domain or enterprise-scale workflows; the paper itself does not rank or recommend specific systems.","The emphasis on NoSQL suggests a testable extension: a benchmark that converts relational Spider or BIRD questions into MongoDB aggregation pipelines would directly measure whether LLM text-to-SQL skills transfer to unstructured data.","If the field adopts the paper's agenda, evaluation may shift from single-query accuracy to workflow-level metrics that include debugging effort and clarification efficiency."],"forward_implications":["Dedicated text-to-SQL systems remain preferable to general-purpose AI chatbots for precise database queries, especially in healthcare, finance, and education.","Progress is measured almost entirely on relational benchmarks, so the lack of NoSQL datasets is a real barrier to enterprise adoption.","Future systems will need multi-turn interaction, external knowledge integration, and human-in-the-loop validation to be useful in real deployments.","New datasets tailored to dynamic schemas and conversational refinement would likely yield faster gains than further scaling of existing relational benchmarks."],"supporting_citations":[{"why":"Spider is the central cross-domain dataset that defines the generalization benchmark the survey's narrative builds on.","marker":"[1]"},{"why":"Seq2SQL is the early reinforcement-learning model and the source of the WikiSQL benchmark used throughout the model comparisons.","marker":"[7]"},{"why":"Spider 2.0 supplies the real-world enterprise workflow benchmark that motivates the survey's emphasis on scalability and multi-query SQL.","marker":"[6]"},{"why":"BIRD provides the large-scale multi-domain dataset that the survey cites for challenges involving extensive database contents and external knowledge.","marker":"[8]"},{"why":"CoSQL is the dialogue-based benchmark that anchors the survey's discussion of multi-turn text-to-SQL interaction.","marker":"[11]"},{"why":"PICARD is the constrained decoding method that the survey credits for improving T5-based models on dialogue and multi-turn SQL generation.","marker":"[22]"},{"why":"MedT5SQL is the domain-specific healthcare model that supports the survey's application-section claim about medical text-to-SQL.","marker":"[16]"},{"why":"EHRSQL is the electronic health record benchmark used to illustrate domain-specific challenges in healthcare text-to-SQL.","marker":"[20]"},{"why":"SQLova is the WikiSQL model whose reported accuracy appears in the central comparison table and grounds the paper's discussion of high-precision general-purpose SQL generation.","marker":"[17]"},{"why":"RAT-SQL is the relation-aware transformer that grounds the survey's account of advanced models for complex multi-table queries.","marker":"[18]"}],"fun_headline_variants":["LLM text-to-SQL: benchmarks, apps, and the open gaps","Text-to-SQL survey: from Spider to NoSQL, challenges ahead","AI text-to-SQL: LLM progress, applications, and missing pieces","Survey: How LLMs are reshaping text-to-SQL and its limits","Text-to-SQL with LLMs: benchmarks, use cases, and roadblocks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The survey's usefulness rests on the assumption that its secondhand descriptions of models and benchmarks are faithful to the cited sources and that the selected benchmarks and models are representative of the field.","fun_headline_variants_meta":{"raw":{"variants":["LLM text-to-SQL: benchmarks, apps, and the open gaps","Text-to-SQL survey: from Spider to NoSQL, challenges ahead","AI text-to-SQL: LLM progress, applications, and missing pieces","Survey: How LLMs are reshaping text-to-SQL and its limits","Text-to-SQL with LLMs: benchmarks, use cases, and roadblocks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00023,"raw_usage":{"total_tokens":1455,"prompt_tokens":889,"completion_tokens":566,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":505,"completion_tokens_details":{"reasoning_tokens":466}},"tokens_in":505,"tokens_out":566,"duration_ms":6122,"temperature":1.0,"reasoning_tokens":466,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:48:51.284748+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A reader could check Table I against the original papers and the cited datasets; if several entries do not match the primary sources—for example the listed accuracy for SQLova or the architecture description for MedT5SQL—the survey's promise of a reliable overview is undercut.","supporting_citations":[{"cited_title":"Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-sql task,","cited_arxiv_id":null,"evidence_quote":"Spider is the central cross-domain dataset that defines the generalization benchmark the survey's narrative builds on."},{"cited_title":"Medt5sql: a transformers-based large language model for text-to-sql conversion in the healthcare domain,","cited_arxiv_id":null,"evidence_quote":"MedT5SQL is the domain-specific healthcare model that supports the survey's application-section claim about medical text-to-SQL."},{"cited_title":"Rat-sql: Relation-aware schema encoding and linking for text-to-sql parsers,","cited_arxiv_id":null,"evidence_quote":"RAT-SQL is the relation-aware transformer that grounds the survey's account of advanced models for complex multi-table queries."}],"review_version":1}