{"id":"58e71601-0bb8-4b74-9fb9-2132e58212c4","arxiv_id":"2502.05237","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"PSM-SQL iteratively filters tables and columns for text-to-SQL, but its best reported numbers come from a two-module ablation, not from the full three-level multi-granularity method described in the paper.","lead":"A text-to-SQL system from Tencent and Guangdong University of Technology prunes database schemas step by step before asking an LLM to write SQL. The paper's claimed gains rest on dev-set numbers, and the reported system actually corresponds to an ablation that omits the paper's signature column-level module.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Paper's own Table 5 contradicts the central claim: the full MSL configuration is worse than the ablated variant actually reported as PSM-SQL.","rationale":"I agree with the reader's REJECT verdict and do not recommend moving it. The reader identified the chain-loop risk of discarding gold schemas as the weakest assumption; that is real and conceded in §7. However, the more load-bearing problem is attribution: the main PSM-SQL numbers match the Cross_encoder+Gen_LLM ablation, and the full MSL variant performs worse on the paper's own metrics. This is an internal contradiction, not a disagreement with external consensus, so it directly undermines the abstract's claim that multi-granularity MSL yields 1–3 point gains. Missing test-set results and absent code make the empirical support incomplete. The proposed check—running the full configuration and comparing—settles whether the central claim can be salvaged; on current evidence it cannot.","tokens_in":12438,"tokens_out":6908,"duration_ms":66804,"concrete_test":"Re-run the full MSL (All) pipeline of §4.2 on Bird dev with Cycle_2 (BGE triplet column scorer + RoBERTa cross-encoder + Llama3-8B database-level filter) and compare with Table 5. If MSL (All) reproduces EX 58.80 / VES 53.28 while reported PSM-SQL numbers require disabling Emb_LLM, then the headline improvement is not produced by the proposed method. Additionally, run MSL (All) on Spider dev and report EX/EM; the current paper only compares the two variants on Bird, leaving open whether the full configuration ever wins.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported PSM-SQL results in Tables 1–2 omit test-set numbers, and the Bird-dev numbers are identical to the Cycle_2 'Cross_encoder+Gen_LLM' ablation (EX 59.71, VES 65.15; table MA 56.65, IA 97.91, RE 25.43), not to 'MSL (All)' in Tables 4–5. In Cycle_2, MSL (All) is worse on SQL generation (EX 58.80 vs 59.71; VES 53.28 vs 65.15) and in schema linking (table RE 43.07 vs 25.43; table MA 32.79 vs 56.65). Thus the column-level triplet-loss component (Emb_LLM) appears to hurt the very metric the paper claims to improve. Since the central claim requires the full multi-granularity MSL configuration to be responsible for the reported gains, the paper's own ablation falsifies that attribution. The chain-loop concern in §7 is valid but secondary; the primary problem is that the model named PSM-SQL in the headline experiments is not the full proposed method.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"PSM-SQL proposes a progressive schema linking framework with a multi-granularity schema linking (MSL) module operating at column, table, and database levels, combined with a chain loop that iteratively prunes redundant schemas before SQL generation. The paper evaluates on Spider and Bird, reporting schema-linking metrics (MA, IA, RE) and SQL-generation metrics (EX, EM, VES). The central claim is that the multi-granularity schema linking and the chain loop improve execution accuracy by 1–3 percentage points over existing methods.","tokens_in":12718,"tokens_out":5852,"duration_ms":55955,"significance":"If the claimed gains were supported, the idea of iteratively pruning schemas at multiple granularities would be a useful contribution to the text-to-SQL literature. The paper has some strengths: it includes a dedicated ablation study (Tables 4–5), reports schema-linking metrics in addition to end-to-end EX/VES, and explicitly acknowledges the chain-loop trade-off in Section 7. However, as detailed below, the paper's own ablation contradicts the attribution of the reported gains to the full MSL configuration, so the central claim is not currently supported.","major_comments":[{"comment":"The reported PSM-SQL results are identical to the Cycle_2 Cross_encoder+Gen_LLM ablation, not to the full MSL (All) configuration. Table 5 reports Cycle_2 Cross_encoder+Gen_LLM as EX 59.71 and VES 65.15 on the Bird dev set, and Table 1 reports PSM-SQL+Llama3-8B as EX 59.71 and VES 65.15. In contrast, the full MSL (All) configuration in Cycle_2 gives lower EX (58.80) and much lower VES (53.28). Thus the model named PSM-SQL in the headline experiments is not the full method described in Section 4, and the headline results do not demonstrate any benefit from the multi-granularity column-level component (Emb_LLM). This is a load-bearing inconsistency because the paper's central claim attributes the gains to the multi-granularity module.","section":"§5.5, §5.6, Tables 1 and 5"},{"comment":"The ablation also shows that MSL (All) is worse than Cross_encoder+Gen_LLM on the schema-linking metrics themselves: in Cycle_2, table-level MA is 32.79 versus 56.65 and table-level RE is 43.07 versus 25.43. The text in §5.6 states that 'MSL achieves better performance than Cross_encoder, Emb_LLM, and Gen_LLM', but the table does not support this for the combined configuration; only IA is higher. This further undermines the mechanism claimed for the reported improvements.","section":"§5.6, Table 4"},{"comment":"No test-set results are reported for PSM-SQL; the Test columns are '-' for all PSM-SQL rows on both Bird and Spider. Since the abstract and introduction claim 1–3 percentage point improvements over existing methods, and the tables include test-set numbers for baseline methods such as CHESS and SFT CODES, the claimed superiority is not substantiated on held-out test sets. The paper should state explicitly why test results are omitted, or the claims should be limited to dev-set performance.","section":"Tables 1–2"},{"comment":"The chain loop can discard correct schemas, and the paper acknowledges this in Section 7 but does not quantify or mitigate it. Because each cycle's filter is trained using ground-truth schema labels derived from gold SQL, an incorrect prune in an early cycle cannot be recovered by the later SQL generator. The paper should report per-cycle recall/IA and, if possible, an oracle experiment that re-injects gold schemas, to establish that iterative pruning does not systematically lose information needed for SQL generation.","section":"§4.2 and §7"}],"minor_comments":[{"comment":"The threshold of 0.5 and the Top-8/Top-4 column selection counts are stated without a sensitivity analysis; since these hyperparameters interact with the chain loop, a brief sensitivity study would strengthen the empirical claims.","section":"§5.4"},{"comment":"The ground-truth schema labels d_gt are used before being formally defined; please state explicitly how d_gt is derived from gold SQL and whether it includes both tables and columns at each granularity level.","section":"§4.2"},{"comment":"The VES formula appears to compute an average of time ratios multiplied by EX, but the text describes it as measuring 'execution efficiency of accurately generated SQL queries'; please clarify whether VES is computed only on correctly executed queries or over all predictions.","section":"§5.2, Eq. (22)"},{"comment":"The row labels 'PSM-SQL+DeepSeek-7B' and 'PSM-SQL+Llama3-8B' are confusing because Section 4 describes Llama3-8B as the database-level reasoner and SQL generator; please clarify which backbone is used for which component in each reported row.","section":"Tables 1–2 and §5.5"},{"comment":"There are several typos and minor wording issues, including 'Trubo' for 'Turbo', 'chemas' for 'schemas', and 'employes' for 'employs'; these should be corrected in a revision.","section":"Throughout"}],"recommendation":"reject","confidential_remarks":"The internal contradiction between Tables 1 and 5 is decisive for me: the paper's own ablation shows that the full multi-granularity MSL configuration is worse than the ablated Cross_encoder+Gen_LLM configuration that actually produces the reported PSM-SQL numbers. This is not a presentation issue; it invalidates the central attribution claim. The authors might be able to salvage a smaller paper about iterative pruning with a cross-encoder-plus-generation filter, but that would be a different paper with a different claim."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is my take on arXiv:2502.05237. The central claim — that the multi-granularity MSL module drives the reported gains — is not supported by the paper's own data. In Table 5, Cycle 2, the full 'MSL (All)' configuration gets EX 58.80 and VES 53.28 on Bird dev, while the ablated 'Cross_encoder+Gen_LLM' gets EX 59.71 and VES 65.15. The numbers listed for PSM-SQL in Tables 1-2 match the ablated variant, not the full method. So the column-level triplet component appears to hurt SQL generation, and the paper's abstract overstates the benefit of the multi-granularity design.\n\nThat said, there is something here. The idea of chaining schema linking to iteratively prune redundant schemas is a reasonable direction, and the schema-linking metrics in Table 3 show large improvements in MA and RE at both table and column levels compared to the MAC-SQL baselines. The combination of BGE fine-tuning, a RoBERTa cross-encoder with two scoring heads, and a LoRA-tuned LLM is an engineering contribution worth examining, even if none of the components are individually new. The authors are also honest enough to show the full ablation and to admit in Section 5.6 that Cross_encoder+Gen_LLM outperforms MSL on SQL generation.\n\nThe soft spots are significant. First, the name 'PSM-SQL' is used for a system that is not the full proposed method; that is a reporting flaw that undermines the main claim. Second, there are no test-set results for PSM-SQL, only dev. Third, the comparison to CHESS is a hair: 65.06 vs 65.00 on Bird dev, so the claimed 1-3 point gain is not visible there. Fourth, no code or data are released, making the experimental results non-reproducible. The chain-loop limitation in Section 7 is real but secondary; the primary issue is the configuration mismatch.\n\nMy recommendation: this deserves a serious referee, but not acceptance in its current form. The referee should ask the authors to clarify which variant they actually report, provide test-set numbers, release the code, and either justify the column-level module or remove it. The paper is useful as a cautionary example of how ablation results can contradict a paper's central narrative, and the schema-linking idea is worth a second look if the reporting is fixed.","headline":"The paper's own ablation contradicts its central claim: the full MSL method performs worse than the ablated variant that is actually reported as PSM-SQL.","tokens_in":13243,"tokens_out":3573,"would_cite":false,"duration_ms":34338,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PSM-SQL treats schema linking as a progressive, multi-granularity filtering task: by pruning redundant tables and columns in a chain loop before SQL generation, it raises execution accuracy on Spider and Bird by 1 to 3 percentage points…","keywords":["text-to-SQL","schema linking","multi-granularity semantics","chain loop strategy","schema pruning","LLM fine-tuning","execution accuracy"],"falsifier":"Take the Bird development set and, after each PSM-SQL cycle, check whether every table and column required by the gold SQL is still present in the filtered schema set. If any query whose required schema was pruned in an earlier cycle yields a wrong execution result—while the same query with that schema restored yields the correct result—then the chain loop's irreversible pruning is the cause. A direct test: run Cycle_2 with the gold schemas forcibly added back after pruning; if execution accuracy does not increase over standard Cycle_2, then the pruning loss is not the binding constraint.","tokens_in":12282,"feed_emoji":"🗄️","tokens_out":5760,"duration_ms":47390,"temperature":0.7,"pith_summary":"This paper attempts to establish that text-to-SQL accuracy improves when the database schema given to the SQL generator is progressively pruned of redundant tables and columns, rather than passed in full. It claims that learning schema semantics at three granularities—columns, tables, and whole databases—and applying this schema linker iteratively in a chain loop keeps the schemas the gold SQL needs while discarding the rest. If true, this matters because redundant schemas interfere with both fine-tuned and prompted large language models, so a reliable pruning method could make text-to-SQL systems cheaper and more accurate. The paper reports execution-accuracy gains of 1 to 3 percentage points over existing methods on the Spider and Bird benchmarks.","feed_headline":"Progressive schema pruning gains 1-3 points for text-to-SQL","feed_subtitle":"A three-level schema linker with a chain loop trims redundant tables and columns, improving Spider and Bird accuracy.","key_machinery":"The load-bearing component is the multi-granularity schema linking (MSL) module, which operates at three levels: column level, using a fine-tuned embedding model (BGE) trained with triplet loss to rank column relevance; table level, using a RoBERTa-LSTM cross-encoder with a disentangled question embedding, multi-head attention over columns, and both cosine-similarity and classifier scores to select schemas; and database level, using a LoRA-fine-tuned LLM (Llama3-8B) that takes an instruction and the current schema set and outputs relevant schemas. The MSL outputs are unioned into a filtered schema set, which is fed back into the next round of the chain loop. The chain loop itself is the second load-bearing idea: it trades a lower theoretical upper bound on accuracy for a reduced task difficulty by shrinking the schema set each cycle.","core_discovery":"PSM-SQL's central claim is that schema linking for text-to-SQL should be treated as a progressive, multi-granularity filtering task rather than a one-shot selection. The method trains a column-level embedding model with triplet loss to rank candidate columns, a table-level cross-encoder with a classifier and cosine-similarity scores to select tables and columns, and a database-level fine-tuned LLM to reason about which schemas are relevant. These three filters are applied in a chain loop: each cycle's filtered schema set becomes the next cycle's input, so the task difficulty decreases as redundant schemas are removed. The paper argues that this continuous reduction in schema count, combined with the complementary semantic signals from the three granularities, is what yields higher matching accuracy, lower redundancy, and better execution accuracy than methods that link schemas only at the table level and only once.","pith_inferences":["If the approach is correct, the schema-linking module could be trained once and reused across different SQL generators (fine-tuned or prompted), because it is decoupled from the parse step; this would make schema pruning a reusable preprocessing service for text-to-SQL.","The chain loop's irreversible pruning suggests a testable extension: add a fallback that, when final SQL generation fails, re-expands the schema set from an earlier cycle; if this recovers accuracy, it would show the loop's trade-off can be softened.","The ground-truth schema labels are derived from gold SQL; a natural extension would be to generate pseudo-labels from a trained SQL model's attention or from execution feedback, which could remove the dependency on annotated schema labels.","The multi-granularity idea could transfer to other retrieval-augmented tasks with hierarchical structure, such as API selection or function-signature matching, not just database schemas."],"forward_implications":["If the central claim holds, replacing one-shot schema linking with iterative multi-granularity pruning reduces the number of tokens and irrelevant columns in prompts, lowering inference cost and helping LLMs focus attention on the relevant schema.","The reported gains on both Bird and Spider suggest the method transfers across different dataset distributions, from smaller manually annotated schemas in Spider to large, noisy, evidence-augmented databases in Bird.","Because the paper shows that plugging PSM-SQL's predicted schemas into an existing method (MAC-SQL) improves that method's accuracy, the schema linker could be used as a drop-in component in other text-to-SQL pipelines.","The chain loop explanation implies that schema-pruning quality, not just SQL-generation capability, is a limiting factor for current text-to-SQL systems.","Larger LLMs (Llama3-70B) benefit more from the pruned schemas than smaller ones, suggesting the method's value grows as base models get stronger."],"supporting_citations":[{"why":"Provides the Spider dataset used for evaluating schema linking and SQL generation in this paper.","marker":"(Yu et al., 2018)"},{"why":"Provides the Bird dataset with large, evidence-augmented databases used as the second evaluation benchmark.","marker":"(Li et al., 2023c)"},{"why":"Supplies the BGE embedding model that is fine-tuned at the column level of the MSL module.","marker":"(Xiao et al., 2024)"},{"why":"Supplies the RoBERTa model used as the backbone of the table-level cross-encoder.","marker":"(Liu et al., 2019)"},{"why":"Supplies the LoRA fine-tuning method used to adapt LLMs for database-level schema reasoning and SQL generation.","marker":"(Hu et al., 2022)"},{"why":"Provides the CHESS baseline, a strong hierarchical schema-pruning competitor that PSM-SQL compares against on Bird.","marker":"(Talaei et al., 2024)"}],"fun_headline_variants":["Progressive multi-level schema linking wins 1-3 points for text-to-SQL","Chain-loop schema pruning boosts text-to-SQL by up to 3 points","Column-to-database schema filtering trims noise, lifts SQL accuracy","Multi-granularity schema linker outdoes one-shot selection for SQL","Progressive schema reduction cuts redundancy, gains 1-3 points on Spider"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The weakest load-bearing premise is that the schema filter in each cycle, trained on ground-truth schema labels, never prunes away a table or column that the gold SQL actually needs, and that those ground-truth labels are complete and correct; if either fails, the correct SQL can become unreachable.","fun_headline_variants_meta":{"raw":{"variants":["Progressive multi-level schema linking wins 1-3 points for text-to-SQL","Chain-loop schema pruning boosts text-to-SQL by up to 3 points","Column-to-database schema filtering trims noise, lifts SQL accuracy","Multi-granularity schema linker outdoes one-shot selection for SQL","Progressive schema reduction cuts redundancy, gains 1-3 points on Spider"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000276,"raw_usage":{"total_tokens":1657,"prompt_tokens":965,"completion_tokens":692,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":592}},"tokens_in":581,"tokens_out":692,"duration_ms":6205,"temperature":1.0,"reasoning_tokens":592,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-08T21:37:19.367969+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the Bird development set and, after each PSM-SQL cycle, check whether every table and column required by the gold SQL is still present in the filtered schema set. If any query whose required schema was pruned in an earlier cycle yields a wrong execution result—while the same query with that schema restored yields the correct result—then the chain loop's irreversible pruning is the cause. A direct test: run Cycle_2 with the gold schemas forcibly added back after pruning; if execution accuracy does not increase over standard Cycle_2, then the pruning loss is not the binding constraint.","supporting_citations":[],"review_version":1}