{"id":"820356a4-2aaa-405e-8234-11785b0f6b57","arxiv_id":"2506.03598","paper_version":3,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"AP-SQL combines schema filtering, retrieval-augmented generation, and reasoning prompts, reporting small accuracy gains on the Spider text-to-SQL benchmark.","lead":"This paper describes AP-SQL, a pipeline that helps language models turn natural-language questions into SQL queries more accurately. It combines several known techniques: filtering database schemas, retrieving similar example queries, and using reasoning prompts such as Chain-of-Thought.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's claimed gains (0.5–3.2 EX points) rest on an undocumented baseline protocol and unmatched conditions; the reported differences are indistinguishable from evaluation noise as presented.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: the baseline comparison is not demonstrably matched, and the gains are small enough that evaluation variance could erase them. I agree with the REJECT verdict, so no verdict adjustment is needed. My stress-test adds a sharper technical detail: the provenance of the baseline rows is not merely undisclosed but internally contradictory in the reference mapping, which makes it impossible to verify whether the numbers come from original papers, reimplementations, or a different evaluation setup. The concrete check I propose is the minimal one that would settle the concern: a paired, seeded, same-condition rerun of all baselines with bootstrap confidence intervals on the differences. If the check succeeds, the paper still needs code release and detailed hyperparameters, but the central empirical claim would at least be testable. I do not see a separate internal inconsistency in the method itself; the schema-filtering, RAG, and prompt-engineering components are plausible, and the weakness is squarely in the evidence supporting the claimed superiority.","tokens_in":8334,"tokens_out":2711,"duration_ms":30510,"concrete_test":"Obtain or re-implement the published E-SQL, DIN-SQL, ACT-SQL, and C3-SQL pipelines and run AP-SQL plus all baselines on the same Spider dev split with identical K=3 retrieval, identical prompt budget, identical sampling temperature, and at least three seeds. For each backbone, compute per-example execution accuracy and a paired bootstrap 95% confidence interval on the AP-SQL-vs-best-baseline difference. If any of the four differences has a confidence interval containing zero, or if the baseline numbers cannot be reproduced from their cited sources, Table 1's 'consistently outperforms' claim is not established.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AP-SQL consistently beats E-SQL, DIN-SQL, ACT-SQL, and C3-SQL on Spider dev across four backbones. The load-bearing premise is that the baseline rows in Table 1 were produced under matched conditions (same backbone, same retrieval setup, same prompt budget, same evaluation harness) and that the 0.5–3.2 EX differences are real rather than evaluation noise. The paper provides no protocol for baselines: no code, no sampling temperature, no repeated runs, no variance information, and the cited provenance is incoherent—§3.2 says 'E-SQL[3], ACT-SQL[1], C3-SQL[2]' while reference [1] is C3, [2] is E-SQL, [3] is Codes, and DIN-SQL/ACT-SQL do not appear in the reference list at all. Thus neither the source of the baseline numbers nor the comparability of conditions can be checked. Because the reported advantages are comparable to Spider's known run-to-run and selection variance (e.g., GPT-4o-mini: AP-SQL 83.2 vs E-SQL 82.6), the headline superiority claim is not supported by the evidence as presented. This is not a criticism of the architectural idea; it is a failure of the empirical demonstration.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AP-SQL (Auto Prompt SQL), a modular text-to-SQL architecture for resource-constrained environments. The system first filters database schemas using a fine-tuned Qwen-3B model, then performs schema linking via a scored, two-stage table/column selection process, retrieves top-K question-SQL pairs through a RAG module, and finally generates SQL using either chain-of-thought or graph-of-thought prompting depending on query complexity. The method is evaluated on the Spider dataset across four LLM backbones (Qwen-7B, Llama-8B, GPT-4o-mini, GPT-4o), and Table 1 reports that AP-SQL achieves higher Execution Accuracy and Test Suite Accuracy than several baseline methods on the Spider development set. The central claim is that this modular pipeline consistently improves text-to-SQL performance across diverse backbones while remaining resource-efficient.","tokens_in":8563,"tokens_out":4783,"duration_ms":47926,"significance":"If the reported results were reproducible and the comparisons valid, the paper would offer a practical, modular recipe for improving small-model text-to-SQL accuracy through schema filtering, RAG, and prompt-design choices. However, as presented, the empirical evidence is not sufficient to support the superiority claim: the baseline results in Table 1 lack a documented protocol, no variance or statistical tests are reported, and no ablations isolate the contribution of the proposed modules. The individual components (schema filtering, RAG, and CoT prompting) are known techniques, and the paper does not provide enough implementation detail to assess novelty or reproducibility. Consequently, the significance is currently low; the ideas may have merit, but the paper does not demonstrate it.","major_comments":[{"comment":"The baseline comparison is not verifiable. The text states that AP-SQL outperforms E-SQL[3], ACT-SQL[1], and C3-SQL[2], but the reference list assigns [1] to C3, [2] to E-SQL, and [3] to Codes; DIN-SQL and ACT-SQL do not appear in the reference list at all. Furthermore, the paper does not state whether the baseline numbers were produced by re-running the baselines in this paper's setup (same backbone, same retrieval, same prompt budget, same evaluation harness) or were copied from other publications. Because the claimed improvements are small (0.5–3.2 EX points), even modest evaluation variance or differences in prompting could erase them. The paper must provide a detailed baseline protocol, run the baselines under matched conditions, and report variance or statistical significance; otherwise the central claim of consistent superiority is unsupported.","section":""},{"comment":"Core components are underspecified. The RAG module is described only as retrieving 'the Top-K most relevant Text-SQL pairs' but the embedding model, similarity metric, example-library construction, and K selection criterion are not given. The schema-linking scoring function in Eq. (3) is not fully defined: the content of the prompt p, the exact form of V(p, q), and how the scalar score v(d) is computed or calibrated are omitted. The 'voting' mechanism for column selection is mentioned but never explained, and the 'difficulty grading mechanism' introduced in Section 2 is not described at all. The CoT and GoT prompt templates are said to be designed but no examples are shown. Without these details, the method cannot be reproduced, and the reader cannot determine whether the reported gains are due to the proposed architecture or to undocumented choices.","section":""},{"comment":"There is no ablation study. The paper claims that schema filtering, RAG, schema linking, and CoT/GoT prompting each contribute to the final performance, but no experiment isolates any of these modules. Consequently, the observed differences in Table 1 could be driven entirely by, for example, the fine-tuned schema filter or the specific prompt templates, rather than by the integrated AP-SQL design. An ablation separating each component is necessary to support the paper's central claim that 'AP-SQL's modular pipeline improves Execution Accuracy and Test Suite Accuracy.'","section":""},{"comment":"The resource-efficiency claim is not quantified. The title and abstract emphasize 'resource-efficient' and 'constrained environments,' but the paper reports no measurements of training/inference cost, GPU memory, latency, or API expense. The system actually uses GPT-4o and GPT-4o-mini as generation backbones, which are not resource-constrained in the usual sense, and the only local computation is a fine-tuned Qwen-3B filter. If the contribution is about reducing the prompt length rather than the overall compute budget, this should be stated explicitly and supported with measurements.","section":""},{"comment":"The evaluation is narrow and lacks robustness evidence. Only the Spider development set is used; there is no test-set evaluation, no cross-benchmark validation, and no repeated runs with different seeds or input orders. Given that the reported improvements are within a few points, the authors should demonstrate that the results are stable across evaluation conditions. The phrase 'comprehensive evaluations' in the abstract is not supported by the experimental content.","section":""}],"minor_comments":[{"comment":"The model name is used inconsistently: 'Auto Prompt SQL' in the title, 'Auto-Prompt' in the text, and 'AP-SQL' in the table and abstract. Please standardize the terminology.","section":"Throughout"},{"comment":"The reference list contains duplicates: [19] and [24] are the same paper, and [4] and [66] both describe XiYan-SQL with slightly different titles. Several references (e.g., [7]-[41] and [44]-[56]) are on latent factor models, robot control, and recommender systems and appear to have little direct connection to text-to-SQL or prompt engineering; these should be either removed or properly integrated into the related-work discussion.","section":"References"},{"comment":"The sentence introducing 'Auto-prompt' as a 'plug-and-play open-source language model' is misleading; what is described is an architecture, not a language model. This wording should be corrected.","section":"§1.1"},{"comment":"Equation (3) has garbled notation: 'V(p, q)' and 'v(d)' are not clearly defined, and the formula appears to be missing punctuation and set notation. Please rewrite it in a clear mathematical form.","section":"§2.3, Eq. (3)"},{"comment":"The schema filter is fine-tuned on approximately 80,000 question-schema pairs, but no information is given about the source or composition of this dataset, or about the evaluation of the filter itself (e.g., precision/recall of table/column selection). Reporting such metrics would help assess the filter's contribution.","section":"§2.2"},{"comment":"The distinction between CoT and GoT prompting is asserted but not illustrated. Provide at least one example of each template, or a precise description of how a 'graph of interconnected reasoning nodes' is represented in the prompt.","section":"§2.4"}],"recommendation":"reject","confidential_remarks":"The manuscript's reference list is dominated by self-citations unrelated to the topic, and the baseline citations in Section 3.2 are incoherent (wrong references, missing entries). This pattern, combined with the absence of any protocol for the baseline numbers, makes it likely that the authors did not run the baselines themselves and simply transcribed numbers from other sources. That is a serious validity threat that cannot be addressed without a full re-evaluation. The paper is also far below the typical standard of a serious journal in terms of method detail and experimental rigor. I recommend reject; if the authors substantially revise the empirical work and methodology, a new submission might be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a plausible integration of known text-to-SQL components, but the empirical claim does not survive contact with the paper. The reported gains over baselines are 0.5-3.2 EX points on Spider dev, and the paper gives no variance, no significance tests, no ablations, no code, and no description of how the baselines were run. The baseline provenance is incoherent: the text says E-SQL[3], ACT-SQL[1], C3-SQL[2], but reference [1] is C3, [2] is E-SQL, [3] is Codes, and DIN-SQL and ACT-SQL do not appear in the reference list at all. That is a load-bearing failure for a table whose headline differences are within run-to-run noise.\n\nWhat is actually new: the specific pipeline - fine-tuned Qwen-3B schema filter + RAG with top-K examples + schema linking prompts + CoT/GoT split by difficulty - is new as a combination, and the authors do report both EX and TS on the standard Spider split. The schema filter fine-tuning on 80k pairs is a concrete detail, and the resource-efficiency motivation is sensible. If the results were real, this would be a modest, useful increment.\n\nThe soft spots beyond the baseline problem: core mechanisms are under-specified. Retrieval, voting, difficulty grading, and the prompt templates are described at a level that prevents reproduction. There are no ablations, so we do not know which component contributes what. The reference list contains many unrelated papers (recommender systems, robotics, robot calibration) that look like padding; that is a red flag for citation diligence.\n\nMy verdict: the idea is not wrong, but the paper currently fails to support its central claim. The reader's REJECT is right, and the stress-test note correctly identifies the matched-conditions premise as the load-bearing assumption. I would desk reject with an invitation to resubmit with code, matched baselines, variance information, and a clean citation list. A serious referee cannot do anything with this as-is.\n\nRecommendation: do not send to peer review; send back for major revision at best.","headline":"Plausible integration of known components, but the 0.5-3.2 point gains rest on an undocumented baseline protocol and incoherent citations; desk reject in current form.","tokens_in":9130,"tokens_out":2771,"would_cite":false,"duration_ms":29454,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AP-SQL, a modular text-to-SQL pipeline, reports higher Execution Accuracy and Test Suite Accuracy than compared baselines on the Spider dev set across four language-model backbones.","keywords":["Text-to-SQL","prompt engineering","chain-of-thought","graph-of-thought","schema linking","retrieval-augmented generation","schema filtering","Spider benchmark"],"falsifier":"Re-run AP-SQL and the four baselines on the Spider dev set with identical schema filters, retrieval pools, prompt budgets, and multiple independent evaluation runs; if the Execution Accuracy gaps in Table 1 shrink to within run-to-run variance, the central claim is refuted.","tokens_in":8106,"feed_emoji":"🧩","tokens_out":9162,"duration_ms":83135,"temperature":0.7,"pith_summary":"AP-SQL is a modular text-to-SQL architecture aimed at environments where only small open-source models can be deployed. It claims that decomposing the task into schema filtering, retrieval of similar question-SQL pairs, schema linking, and prompt-driven generation lets small models approach the SQL accuracy of large closed models. On the Spider development set, AP-SQL reports higher Execution Accuracy and Test Suite Accuracy than the compared baselines for each of four backbones: Qwen-7B, Llama-8B, GPT-4o-mini, and GPT-4o. The practical point is that resource-constrained settings can get closer to state-of-the-art text-to-SQL without running large proprietary models.","feed_headline":"Prompt pipeline tops SQL baselines on four language models","feed_subtitle":"Schema filtering plus retrieval-augmented reasoning prompts beats SQL baselines on Spider.","key_machinery":"The load-bearing mechanism is AP-SQL's modular pipeline. A supervised fine-tuned Qwen-3B schema filter takes the natural-language question and keeps the top three tables and top three columns per table, compressing the prompt. A retrieval-augmented generation module supplies the K=3 most relevant question-SQL pairs from the training set. A schema-linking prompt then scores each candidate table on a 1-10 scale, keeps tables above threshold 6, and compares columns by voting. Finally, SQL generation routes simple single-table queries through chain-of-thought prompts and complex multi-table queries through graph-of-thought prompts. Each module is designed to reduce prompt length or strengthen reasoning, and together they let a small or large backbone produce the final query.","core_discovery":"The paper's central claim is that the combination of a fine-tuned schema filter, a retrieval-augmented example selector, and complexity-based prompt routing improves text-to-SQL accuracy on the Spider benchmark for every language model tested. Table 1 reports AP-SQL at 68.3 EX / 60.8 TS with Qwen-7B, 72.4 / 64.1 with Llama-8B, 83.2 / 75.8 with GPT-4o-mini, and 89.7 / 82.6 with GPT-4o, in each case above the E-SQL, DIN-SQL, ACT-SQL, and C3-SQL baselines. The method works by selecting the top three relevant tables and top three columns per table with a fine-tuned Qwen-3B, retrieving K=3 similar examples, scoring tables during schema linking, and then generating SQL with chain-of-thought templates for simple single-table queries and graph-of-thought templates for complex multi-table queries.","pith_inferences":["If the gains survive a matched-protocol re-run, the design suggests that routing queries by complexity to different reasoning templates is a transferable choice for other structured-output generation tasks.","The fine-tuned schema filter is itself a reusable artifact: any text-to-SQL system could adopt the three-table, three-column pruning step without the rest of AP-SQL.","The reported improvements are small, so the practical value may depend on whether the comparison was run under identical conditions; a variance-aware replication would tell whether the gain is real or noise."],"forward_implications":["AP-SQL outperforms the compared baselines on both Execution Accuracy and Test Suite Accuracy for Qwen-7B, Llama-8B, GPT-4o-mini, and GPT-4o on the Spider dev set.","The pipeline is backbone-agnostic: the same modular prompts improve both small open models and large closed models.","Schema filtering keeps only three tables and three columns per table, which shortens prompts and lowers inference cost in constrained environments.","Because the components are open-source and plug-and-play, the architecture can be adopted without re-training the final generation model."],"supporting_citations":[{"why":"Supplies the C3-SQL baseline that AP-SQL compares against for zero-shot ChatGPT-based text-to-SQL.","marker":"[1]"},{"why":"Supplies the E-SQL baseline and the schema-linking-by-question-enrichment approach that AP-SQL extends.","marker":"[2]"},{"why":"Establishes the decoupling of schema linking and skeleton parsing that AP-SQL's modular design builds on.","marker":"[5]"},{"why":"Provides the Graph-of-Thought prompting method used for complex multi-table SQL generation.","marker":"[42]"},{"why":"Provides the Chain-of-Thought prompting method used for simple single-table SQL generation.","marker":"[43]"},{"why":"Describes open-source language models for text-to-SQL, the resource-efficient setting AP-SQL targets.","marker":"[3]"}],"fun_headline_variants":["Lightweight SQL model tops Spider baselines on four LLMs","AP-SQL: tiny model beats big SQL baselines on Spider","Resource-lean prompt pipeline outdoes SQL heavyweights on Spider","Efficient text-to-SQL: prompt routing beats four baselines","Small model, big SQL wins: AP-SQL on Spider"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper's comparison assumes the baseline systems were run with the same backbone, example retrieval, and prompt budget as AP-SQL, and that the small differences in Table 1 are not due to evaluation variance.","fun_headline_variants_meta":{"raw":{"variants":["Lightweight SQL model tops Spider baselines on four LLMs","AP-SQL: tiny model beats big SQL baselines on Spider","Resource-lean prompt pipeline outdoes SQL heavyweights on Spider","Efficient text-to-SQL: prompt routing beats four baselines","Small model, big SQL wins: AP-SQL on Spider"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000373,"raw_usage":{"total_tokens":1969,"prompt_tokens":897,"completion_tokens":1072,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":513,"completion_tokens_details":{"reasoning_tokens":984}},"tokens_in":513,"tokens_out":1072,"duration_ms":8319,"temperature":1.0,"reasoning_tokens":984,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T10:59:02.099183+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run AP-SQL and the four baselines on the Spider dev set with identical schema filters, retrieval pools, prompt budgets, and multiple independent evaluation runs; if the Execution Accuracy gaps in Table 1 shrink to within run-to-run variance, the central claim is refuted.","supporting_citations":[],"review_version":1}