{"id":"4faea6ae-f9e5-4c3f-94eb-d9a4ab5e5af8","arxiv_id":"2509.05899","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"X-SQL, an expert schema-linking and schema-understanding framework, achieves 84.9% and 82.5% execution accuracy on Spider-Dev and Spider-Test using open-source LLMs.","lead":"X-SQL improves Text-to-SQL accuracy by adding a fine-tuned schema-linking step and a schema-explanation step before SQL generation. It reports state-of-the-art open-source results on the Spider benchmark, though the best scores on the two splits use different model configurations.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline 84.9/82.5 pairs results from two different LLM configurations: the best Dev row (CQ+DC) scores 81.7 on Test, while the best Test row (QW+LS) scores 83.4 on Dev.","rationale":"The reader's weakest_assumption targeted the undocumented curation of the X-Linking SFT dataset, which is indeed a real reproducibility gap. However, the most load-bearing concern for the paper's central claim, 'X-SQL achieves 84.9 on Spider-Dev and 82.5 on Spider-Test as the leading open-source Text-to-SQL framework,' is that these two numbers come from two different model configurations. Table 6 shows no row with both values; the Dev-best row gives 81.7 on Test, and the Test-best row gives 83.4 on Dev. The abstract therefore reports a system that does not exist as a single configuration, and the selection was made after inspecting test-set results, so the test number is not an unbiased estimate. This is an internal consistency issue visible in the authors' own tables, not a matter of external consensus or reimplementation risk. It directly undermines the SOTA claim and the Multi-LLM improvement claim, which compares a single fixed configuration against a per-split best-of-16 selection. I still view the underlying components as plausible: the X-Linking SFT results in Table 1 and the controlled ablation in Table 5 give some independent support to the core idea, and the paper does report full combinations in Appendix C rather than hiding them. But the headline should be corrected to a fixed configuration, or the selection rule should be made explicit and validation-based. Because the reader already assigned CONDITIONAL, my concern reinforces that verdict rather than changing it; the paper should not be accepted as-is, but the evidence is not so broken as to demand rejection of the entire framework.","tokens_in":13400,"tokens_out":4831,"duration_ms":43621,"concrete_test":"Fix one configuration from Table 6, e.g., CQ+DC (CodeQwen1.5-7B-Chat for X-Admin and deepseek-coder-7b-instruct-v1.5 for Debugging), and evaluate the identical X-SQL pipeline on both Spider-Dev and Spider-Test, reporting both numbers for that single row. If the Test result is 81.7 rather than 82.5, the abstract's 84.9/82.5 pair is unattainable by one system. A complementary check: pre-register the selection rule on Spider-Dev only (choose the best Dev configuration), then run exactly that configuration on Spider-Test and compare with the current 82.5; if it drops, the claimed test accuracy is an artifact of test-set selection.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central SOTA claim is internally undermined by per-split model selection. In 'Multi-LLMs Ablation Studies' (Table 6), the row with the best Spider-Dev score, CodeQwen1.5-7B-Chat + deepseek-coder-7b-instruct-v1.5 (CQ+DC), achieves 84.9 on Dev but only 81.7 on Test; the row with the best Spider-Test score, Qwen2-7B-Instruct + llama-3-sqlcoder-8B (QW+LS), achieves 83.4 on Dev but 82.5 on Test. The abstract reports 84.9 on Dev and 82.5 on Test together, but no single configuration in Table 6 or Table 7 attains both numbers. Figure 1's caption states the LLM setup is 'based on the best Spider-Test result,' and the 'best' configurations were chosen only after running all 16 combinations in Appendix C. Thus the headline numbers are not an evaluation of a fixed X-SQL system; they are the per-split maxima from a test-set-driven selection. This makes the 'leading open-source Text-to-SQL framework' claim unverifiable from the reported experiments and inflates the apparent gain of the Multi-LLM component (1.3%/2.2%), because the comparison system is one fixed row while the proposed system is allowed to pick the best row per benchmark. This issue is separate from, and more direct than, the undocumented X-Linking training-data curation: even if the SFT data is perfectly clean, the reported pair of scores does not correspond to any single reproducible system.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes X-SQL, a modular text-to-SQL system for the Spider benchmark built on open-source LLMs. Its first component, X-Linking, is an SFT-trained schema linker (Q-LoRA on LLMs) that takes a question, candidate schemas, and foreign keys and outputs relevant table names; at inference it unions outputs over five shuffled orderings. The second component, X-Admin, prompts an LLM to explain linked columns in natural language. A third stage generates and debug-fixes SQL. The authors also assign different LLMs to the X-Admin and Debugging components. They report execution accuracies of 84.9% on Spider-Dev and 82.5% on Spider-Test, claiming state-of-the-art performance among open-source frameworks and attributing large gains to schema learning.","tokens_in":13818,"tokens_out":4841,"duration_ms":40850,"significance":"The X-Linking contribution is potentially valuable: Table 4 shows a controlled plug-and-play comparison in which adding X-Linking improves CodeQwen1.5-7B-Chat from 74.3 to 81.6 on Spider-Dev, and Table 5 identifies X-Linking as the largest component, with a 7.3% drop when it is removed. Unlike prior schema-linking modules, the method uses dedicated SFT and is evaluated externally on Spider with recall metrics, so the core X-Linking result is not circular. If the training-data construction is disclosed and the per-split model-selection issue is resolved, the framework and ablations would be a useful addition to the open-source text-to-SQL literature. The X-Admin idea is also clearly motivated and shows a modest but positive ablation. The paper's strengths are its controlled component ablations and its use of an external benchmark rather than a self-constructed evaluation.","major_comments":[{"comment":"The reported headline pair 84.9/82.5 is not achieved by any single system configuration. In Table 6, the row CQ+DC scores 84.9 on Spider-Dev but 81.7 on Spider-Test, while the row QW+LS scores 83.4 on Spider-Dev and 82.5 on Spider-Test; no row in Table 7 attains both values. The abstract and Table 2 present these numbers together as X-SQL's performance, and Figure 1's caption states that the LLM setup is based on the best Spider-Test result, which was selected after running all 16 combinations in Appendix C. This is per-split, test-set-driven model selection, so the claim that X-SQL is the leading open-source framework and the Multi-LLM gains of 1.3%/2.2% are not supported as stated. Please either report a single pre-specified configuration for both splits or explicitly present the results as an upper envelope, and re-evaluate the Multi-LLM comparison against a fixed baseline rather than per-split maxima.","section":"Abstract; Table 2; Table 6; Table 7; Figure 1 caption"},{"comment":"The manuscript states 'We carefully curate the dataset for SFT Schema Linking training' but never describes how the candidate table schemas S, foreign keys K, and ground-truth table sets T are constructed from Spider. Appendix B gives only Q-LoRA hyperparameters and hardware details. This omission is load-bearing because the central 7.3% ablation gain depends on the training distribution: if candidate tables are sampled from the same databases as the evaluation questions, or if the negative candidates are unrepresentative, the reported improvements may not transfer. Please specify the data construction procedure, including the source of negative examples, the number of training instances, and whether Spider-train databases overlap with the dev and test databases.","section":"X-Linking (Schema Linking); Appendix B"},{"comment":"No significance tests or measures of variation are reported, even though the paper states that all experiments are conducted three times and average results are reported. Key comparisons in Tables 5 and 6 involve differences of 0.8-1.7 percentage points, which could plausibly be within run-to-run noise for stochastic LLM inference with self-consistency and debugging. Please report standard deviations or confidence intervals, or run paired significance tests, for at least the headline comparisons and the Multi-LLM claims.","section":"Experiments; Tables 5-6"}],"minor_comments":[{"comment":"There is a typo in the section heading: 'Schmea Understanding' should be 'Schema Understanding'.","section":"X-Admin (Schema Understanding)"},{"comment":"The backbone model name is inconsistent: Tables 1 and 3 refer to CodeQwen1.5-7B-Instruct for X-Linking, while the text in 'X-Linking Ablation Studies' and Table 6 refer to CodeQwen1.5-7B-Chat; Appendix C defines CQ as CodeQwen1.5-7B-Instruct. Please standardize the model naming throughout.","section":"Tables 1, 3, 6, and Appendix C"},{"comment":"The sentence 'We use the its best result for analysis' contains a grammatical error; it should read 'We use its best result for analysis.'","section":"End-to-end Text-to-SQL Results"},{"comment":"The checklist states that source code will be made publicly available, but no repository link or release plan is given in the paper; please add a concrete availability statement, along with the seed handling and software versions noted as partial in the checklist.","section":"Reproducibility Checklist"}],"recommendation":"major_revision","confidential_remarks":"The main concern is not the novelty of the components but the reporting of the headline numbers: combining per-split best configurations and calling the result a single system is a credibility issue for a benchmark-focused paper. I would encourage the editor to require the authors to either report one fixed configuration or clearly relabel the results as an upper envelope, and to disclose the X-Linking training-data construction before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core idea here is simple and credible: train a dedicated Q-LoRA SFT model to output relevant table names given a question, schema, and foreign keys, then use that to filter the schema before SQL generation. The X-Admin component that translates abstract schema into natural-language descriptions is a nice, low-cost addition. The X-Linking ablation is the strongest part of the paper: Table 4 shows a consistent 1.4–7.3 point gain over no schema linking across four models, and performance lands close to the Oracle-schema upper bound. That is a real, externally benchmarked effect, and the comparison against DIN-SQL, PET-SQL, and MAC-SQL schema linking modules is informative.\n\nThe main soft spot is exactly what the stress-test note says: the headline 84.9/82.5 does not correspond to any fixed system. In Table 6, the best Dev row (CQ+DC) scores 81.7 on Test, and the best Test row (QW+LS) scores 83.4 on Dev. The abstract presents both numbers together, but no single configuration in the paper attains both. The authors searched all 16 combinations and selected the best per split, then compared against one fixed single-LLM baseline. That makes the Multi-LLM advantage claim unverifiable from the reported experiments and inflates the apparent gain. This is a reporting flaw, not a fatal one, but it needs to be fixed before the SOTA claim is taken seriously: report a single fixed configuration, or pre-specify a validation-based selection rule.\n\nA second soft spot is the X-Linking SFT training data. The paper says the dataset was \"carefully curated\" but gives no details about how candidate schemas, foreign keys, and ground-truth table sets were sampled from Spider. The external test split mitigates circularity, but the exact construction matters for understanding whether the method transfers to new databases. Minor issues: Table 1 labels the X-Linking model as CodeQwen1.5-7B-Instruct while the text says the best result used CodeQwen1.5-7B-Chat; Table 3 shows Re=0.891 for Instruct while Table 8 shows 0.922 for SFT+EI, and the model name discrepancy is never explained. No significance tests, no seed details, and code/data are not yet released; the checklist acknowledges most of this.\n\nWho this is for: anyone working on open-source Text-to-SQL or schema linking. The X-Linking recipe is worth scrutinizing and possibly adopting, but only after the authors release code and report a single reproducible configuration. I would send it to peer review with a strong request for those revisions, rather than desk-rejecting.","headline":"Worth reading for the dedicated SFT schema linker, but the headline 84.9/82.5 is a per-split best-of-16 selection, not a single system.","tokens_in":14306,"tokens_out":2126,"would_cite":false,"duration_ms":19630,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dedicated schema linker and a plain-language schema explainer lift open-source text-to-SQL to 84.9% execution accuracy on Spider-Dev.","keywords":["text-to-sql","schema linking","schema understanding","supervised fine-tuning","multi-LLM systems","execution accuracy","Spider benchmark"],"falsifier":"Reproduce X-Linking with a fully specified sampling procedure for candidate schemas and evaluate table recall and end-to-end accuracy on Spider-Test and on held-out databases not in Spider; if the gain over no linking shrinks materially or table recall drops, the reported 7.3-point improvement is not a general schema-linking capability.","tokens_in":13206,"feed_emoji":"🗄️","tokens_out":7907,"duration_ms":63728,"temperature":0.7,"pith_summary":"X-SQL argues that the bottleneck in text-to-SQL is not SQL generation itself but the database schema: LLMs cannot reliably pick the right tables from a large schema, and even when shown the right tables, they misread abstract column names. The paper proposes two schema experts to fix this. X-Linking is a small model fine-tuned specifically to output the needed table names from candidate schemas, foreign keys, and the user's question, with a shuffle-and-union inference step. X-Admin is a prompt-based component that rewrites the linked schema in natural language, glossing each column and hinting at joins. Together these components add roughly nine points of execution accuracy, and assigning different LLMs to different pipeline stages adds another point or two, yielding reported accuracies of 84.9% on Spider-Dev and 82.5% on Spider-Test using only open-source models.","feed_headline":"Schema experts push open-source text-to-SQL to 84.9%","feed_subtitle":"A fine-tuned table selector plus plain-language schema explanations add nine accuracy points and beat prior open-source systems.","key_machinery":"The load-bearing object is X-Linking, a schema linker built by supervised fine-tuning: its training objective maximizes the probability of the correct table-name sequence T given a prompt that concatenates candidate schemas S, foreign keys K, and the user question Q. Inference adds a self-consistency step in which the candidate order is shuffled five times and the union of predicted table names becomes the final linked schema. The second mechanism is X-Admin, a role-prompted schema-understanding stage that turns abstract column names and sample values into plain-language definitions and join hints. The argument also depends on a multi-LLM assignment: different backbone models are used for linking, generation, and debugging, with a debugger different from the generator consistently helping, which the paper attributes to LLMs favoring their own outputs.","core_discovery":"The central claim is that schema linking and schema understanding are distinct, learnable bottlenecks and that a text-to-SQL system should treat them as separate components rather than as incidental steps in prompting. X-Linking is trained with supervised fine-tuning to output a sequence of table names T given candidate schemas S, foreign keys K, and question Q; at inference the inputs are shuffled five times and the predicted table sets are unioned. X-Admin then takes the linked tables and produces natural-language descriptions of each column and join relationship, so the generator receives a compact, human-readable view instead of raw DDL. The paper reports that removing X-Linking costs 7.3 points of execution accuracy, removing X-Admin costs 1.7 points, and the complete multi-LLM system reaches 84.9% on Spider-Dev and 82.5% on Spider-Test, the highest open-source-model results it compares against.","pith_inferences":["Editorial inference: because the SFT training data for X-Linking is described as 'carefully curated' but not specified, the 7.3-point gain could partly reflect the sampling of candidate schemas rather than a general linking skill; a reproducible curation procedure is needed to test this.","Editorial inference: X-Admin's mechanism of explaining schema elements in natural language likely transfers to other structured-code tasks such as API selection or dataframe transformation, though the paper does not test those settings.","Editorial inference: the near-oracle gap in the paper's own table suggests that once schemas are linked well, remaining errors come from generation and debugging, so the next gains should come from those stages rather than from better linking.","Editorial inference: using a different, possibly weaker, model as the debugger might be as effective and cheaper; the paper reports the benefit of different-backbone debugging but does not explore the cheapest such configuration."],"forward_implications":["X-Linking can be dropped into existing text-to-SQL pipelines as a pre-step; the paper shows it improves SQL generation for four different generator models relative to no linking and approaches oracle-schema accuracy.","X-Admin needs no fine-tuning and can be added to any framework, giving a gain comparable to the widely used debugging step.","Longer context windows will increase, not decrease, the value of a dedicated linker, because more irrelevant tables will otherwise crowd the prompt.","Assigning different LLMs to generation and debugging is a cheap way to improve open-source systems, consistent with the paper's self-preference explanation."],"supporting_citations":[{"why":"Supplies the Spider benchmark, its splits, and the execution-accuracy ground truth used in all results.","marker":"(Yu et al. 2018)"},{"why":"Provides the PET-SQL baseline, the SQL-Tailored prompt template that X-SQL refines, and the R_e/R_s schema-linking recall metrics.","marker":"(Li et al. 2024b)"},{"why":"Defines the DIN-SQL baseline whose schema-linking module and decomposed prompt structure are reproduced and compared against.","marker":"(Pourreza and Rafiei 2024)"},{"why":"Provides the MAC-SQL multi-agent baseline and the only prior training-based schema-linking approach that X-Linking outperforms.","marker":"(Wang et al. 2023)"},{"why":"Supplies the parameter-efficient fine-tuning method used to train the X-Linking schema linker.","marker":"(Dettmers et al. 2024)"},{"why":"Motivates the shuffle-and-union self-consistency inference strategy applied to X-Linking.","marker":"(Wang et al. 2022)"}],"fun_headline_variants":["Schema linking and understanding as separate experts hit 84.9%","Multi-LLM schema duo pushes text-to-SQL to 84.9%","Separate schema experts boost open-source text-to-SQL to 84.9%","X-SQL: two schema experts, one 84.9% Spider score"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the training pairs for X-Linking—candidate schemas, foreign keys, and ground-truth table sets sampled from Spider—represent the schema-linking task broadly; the paper does not describe how those pairs were curated, so the large reported gain could be an artifact of that unstated sampling.","fun_headline_variants_meta":{"raw":{"variants":["Schema linking and understanding as separate experts hit 84.9%","Multi-LLM schema duo pushes text-to-SQL to 84.9%","Separate schema experts boost open-source text-to-SQL to 84.9%","X-SQL: two schema experts, one 84.9% Spider score"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001031,"raw_usage":{"total_tokens":4359,"prompt_tokens":976,"completion_tokens":3383,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":592,"completion_tokens_details":{"reasoning_tokens":3297}},"tokens_in":592,"tokens_out":3383,"duration_ms":20381,"temperature":1.0,"reasoning_tokens":3297,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T16:20:01.759940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reproduce X-Linking with a fully specified sampling procedure for candidate schemas and evaluate table recall and end-to-end accuracy on Spider-Test and on held-out databases not in Spider; if the gain over no linking shrinks materially or table recall drops, the reported 7.3-point improvement is not a general schema-linking capability.","supporting_citations":[],"review_version":2}