{"id":"e11a8bb6-fbba-4b1e-b34c-54b01755e649","arxiv_id":"2607.22624","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"CHS-SQL combines beam search with model-internal-confidence filtering in schema linking to improve small-model text-to-SQL accuracy on Spider and BIRD.","lead":"This paper proposes a two-stage text-to-SQL pipeline for small language models that filters database schema information using beam search and a model-confidence score, then generates SQL from the filtered schema. It reports large gains on the Spider and BIRD benchmarks, though several of the paper's claims and settings do not line up internally.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Trace Confidence is asserted as a correctness signal but never calibrated; the main mechanism may be dev-set threshold tuning.","rationale":"The reader's weakest assumption and my concern coincide: the load-bearing component is the equation of Trace Confidence with correctness. This is the component whose failure would invalidate the claimed precision–recall optimization, because without a valid confidence signal the filtering step has no principled basis. The paper's Table 3 ablation does give some direction (removing the filter costs 1 EX point and the full system is best), but it is a single dev-set comparison with a threshold tuned on that same set, so it cannot distinguish a real confidence signal from a tuned cutoff. The larger gains come from beam search / all-columns recall, not necessarily from TC. I therefore recommend keeping the CONDITIONAL verdict: the method is plausible and testable, but the central mechanism should be validated by a calibration study before the SOTA claim is accepted. I do not see a reason to reject outright; the code is claimed public, making the test feasible. I note further reporting inconsistencies (e.g., 'sub-billion' for 4B/7B models, BIRD EM mentioned but not tabled, ensemble size differing between Table 1 and Table 3) that reinforce the need for an independent check, but they are secondary to the confidence-calibration issue.","tokens_in":15466,"tokens_out":10966,"duration_ms":108610,"concrete_test":"Using the released code, run CodeLlama-7B schema linking on Spider Dev with beam width K=10. For every candidate trace h, compute TC(h) and a correctness label from Parse(h) against the gold tables/columns (e.g., schema F1). Compute the Spearman correlation between TC and correctness, and compare the average correctness of the retained set {h: TC≤θ} with the unfixed top-beam candidate, using a θ fixed a priori from the training split. If the correlation is not significantly positive and the retained set is not more accurate, Eq. 10 is unsupported. (Optionally repeat for SQL candidates and Eq. 19 using execution accuracy.)","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central innovation (Sec. 4.2.3–4.2.4, Eqs. 5–10) is to filter schema-linking beam traces by Trace Confidence, defined as the standard deviation of per-token negative average log-probabilities, and to use the same signal to select SQL in Eq. 19. The paper states that low TC means high model certainty, but this is not definitionally true: a uniformly uncertain trace has nearly constant T_j and thus TC≈0, and would be treated as maximally confident. No calibration is reported that links lower TC to higher correctness of the parsed schema or to higher execution accuracy. The only evidence is the Table 3 ablation (w/o TC filtration drops EX by 1 point on Spider Dev) and a citation to prior work. Since the threshold θ is picked on Spider Dev by the underspecified 'Max-Gap Midpoint Sampling', the reported ≈2–7 point gains over DTS-SQL could be dominated by a dev-set-tuned cutoff rather than a real confidence signal. If TC is not a valid validity signal, the 'optimal precision-recall trade-off' claim and the ensemble selection collapse to arbitrary filtering.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CHS-SQL, a two-stage text-to-SQL pipeline for small language models. In the schema-linking stage, a LoRA-fine-tuned SLM generates candidate table/column sequences via beam search; the candidates are then filtered using a proposed 'Trace Confidence' metric (the standard deviation of per-token negative log-probabilities) with a threshold θ, and the surviving tables' columns are all retained. In the SQL generation stage, multiple fine-tuned SLMs generate candidate SQL queries via beam search, and the query with the lowest Trace Confidence is selected as the final answer. The authors report gains over DTS-SQL on Spider Dev (e.g., EX 0.766 vs 0.674 for Qwen3-4B), Spider Test, and BIRD Dev, and claim that CHS-SQL achieves an optimal precision–recall trade-off in schema linking and state-of-the-art results for SLMs.","tokens_in":15785,"tokens_out":4701,"duration_ms":44103,"significance":"If the reported results are robust, the paper would make a useful empirical contribution by showing that beam-search-based schema linking with confidence filtering can improve text-to-SQL accuracy on small models. The manuscript also provides code, detailed LoRA hyperparameters, and a clear two-stage architecture, which supports reproducibility. However, the central claims rest on two load-bearing assumptions: (1) that the Trace Confidence metric actually measures model certainty, and (2) that the configuration selected on the dev sets represents an 'optimal' precision–recall trade-off. The current evidence for both is weak, and Table 2 appears to contradict the precision–recall claim. The theoretical flaw in the confidence metric is not a mere presentation issue; it affects the core mechanism, so the significance of the contribution depends on whether the authors can provide calibration or reformulate the selection criterion.","major_comments":[{"comment":"Trace Confidence is defined as the standard deviation of token-level confidence values T_j. A trace in which the model is uniformly uncertain at every position will have nearly constant T_j, hence TC≈0, and would be treated as maximally confident by Eq. (19). Conversely, a trace with mixed confident and uncertain tokens gets a high TC. Thus TC measures variability, not certainty, and the paper's statement that 'low trace confidence corresponds to greater model certainty' is not supported by the definition. Since this metric is used both to filter schema-linking traces (Eq. 10) and to select the final SQL (Eq. 19), the entire confidence mechanism is built on an unvalidated assumption. No calibration analysis is provided linking low TC to higher accuracy. Please either correct the definition (e.g., use T_avg or entropy) or provide empirical evidence that low TC correlates with correctness","section":"§4.2.3, Eqs. (5)–(7); §4.3, Eq. (19)"},{"comment":"The central claim of an 'optimal precision–recall trade-off' is not supported by the reported data. In Table 2, CHS-SQL has the lowest table precision (T-P=0.661) and by far the lowest column precision (C-P=0.192) among the compared methods, and its table/column F1 (0.793/0.322) are also below DTS-SQL (0.954/0.415). The method essentially achieves high recall at the expense of precision, with a small EX gain (0.736 vs 0.731) and a lower EM (0.697 vs 0.720) relative to DTS-SQL on the same CodeLlama-7B backbone. Calling this configuration 'optimal' is not justified by the table. The later statement in §5.3 that 'the combination of Beam Search and Trace Confidence filtration helps CHS-SQL find the perfect precision-recall balance' is directly contradicted by the numbers. Please reframe the claim to 'high-recall, lower-precision configuration' and provide evidence that this trade-off is inde","section":"Table 2, §5.3"},{"comment":"The hyperparameters (k*, θ*) are selected to maximize the sum of table/column precision and recall (Eq. 12), and the final EX/EM are then reported on the same Spider Dev and BIRD Dev sets used for this selection. This constitutes test-set tuning for the main reported numbers. The 'gains' over DTS-SQL may therefore partly reflect overfitting to the dev set rather than a generalizable improvement. The paper reports only one held-out test split (Spider Test, CodeLlama-7B), so the claim of SOTA is not established. Please report results on held-out test sets for all models and datasets, or use a validation split separate from the final evaluation, and show sensitivity of EX/EM to θ and k.","section":"Eq. (12), §5.1.3, Tables 1–3"},{"comment":"The paper repeatedly states that the experiments use 'SLMs with fewer than 1 billion parameters' and 'sub-billion-parameter LLMs', but the actual models are CodeLlama-7B, Qwen3-4B, and DeepSeek-Coder-6.7B-Instruct—all above 1B parameters (Qwen3-4B is 4B). This is inconsistent with the abstract's emphasis on small models and with the claim that the method works for 'SLMs'. Either correct the model sizes or revise the claim to 'models in the 4B–7B range'. This is not merely a typo; it changes the scope of the contribution.","section":"§5.1.2 and Abstract"}],"minor_comments":[{"comment":"The 'Max-Gap Midpoint Sampling' method for choosing θ is mentioned but never described. Please provide the algorithm or a reference, as it is essential for reproducibility.","section":"§4.2.4"},{"comment":"The figure caption uses abbreviations M1–M7, but the legend in the text (M1: FT-SLM, etc.) does not exactly match the order listed in the caption. Please align them.","section":"§5.3, Figure 2"},{"comment":"In the ablation text, 'standard CHS-SQL (the voting version)' is contrasted with 'single model version CHS-SQL', but the table shows CHS-SQL (only with Codellama-7b) as the single-model version. The wording is confusing; clarify which version is used in each ablation row.","section":"§5.4"},{"comment":"The text in the final paragraph of §5.2 says 'our scores reached 75.4% (EX) and 75.2% (EM)', but Table 1 lists the ensemble as 0.773 EX and 0.739 EM. These numbers do not match; please correct.","section":"§5.2"},{"comment":"The notation P_M∫ uses an integral sign that appears to be a typo (likely P_M_s). The same issue appears in Eq. (3) with L_M∗. Please use consistent subscript formatting.","section":"§3.2, Eq. (2)"},{"comment":"Reference [17] is cited for Beam Search but points to 'Efficient Estimation of Word Representations in Vector Space' (Mikolov et al.). Please cite a proper beam search reference.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The manuscript has several load-bearing issues—most notably the definition of Trace Confidence, the selection of hyperparameters on the evaluation sets, and the contradiction between the 'optimal trade-off' claim and Table 2. These are fixable in principle, but the paper as written does not support its central abstract claims. If the authors can re-define or calibrate the confidence metric, report results on proper held-out test sets, and soften the trade-off claim to match the data, the contribution could become a solid empirical study. I would not recommend rejection outright, but a thorough revision is needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort version: this is DTS-SQL plus beam search and a trace-confidence filter on schema linking, plus the same confidence idea for ensemble SQL selection. That is a legitimate extension, and the ablation suite is a plus. Qwen3-4B at 76.6 EX on Spider Dev is a nice datapoint. But the performance claims are not yet trustworthy. Check two things before citing the numbers.\n\nFirst, the trace-confidence definition. Eq. 7 defines TC as the standard deviation of per-token average negative log-probs. The paper equates low TC with high model certainty. That is only true if the mean T_j is also low. A uniformly uncertain trace has nearly constant T_j, so TC≈0 and the paper would treat it as maximally confident. No calibration links TC to schema-linking correctness or SQL execution accuracy. The ablation shows only a 1-point EX drop without TC filtration, which suggests most of the gain comes from beam search and the all-columns expansion, not from TC. If TC is not a valid correctness signal, the 'optimal precision-recall trade-off' claim collapses to arbitrary thresholding.\n\nSecond, the selection of θ and k. Eq. 12 picks the pair that maximizes recall plus precision on Spider Dev and BIRD Dev — the same sets where final EX/EM are reported. The 'Max-Gap Midpoint Sampling' for θ is underspecified. Without a held-out split or at least a sensitivity analysis, the 2–7 point gains over DTS-SQL could be dev-set overfitting. Also, Table 2 uses a stronger DTS-SQL baseline (0.731 vs 0.715 in Table 1) because it adds beam search; the paper never reconciles this with the headline comparisons.\n\nMinor but annoying: the 'sub-billion parameter' description of 4–7B models is wrong. There are no error bars, and the ensemble description changes between Table 1 (two models) and Table 3 (three models).\n\nBottom line: the core idea is not fatal, but the main mechanism is under-supported. A serious referee should ask for calibration analysis of TC, a proper validation split for θ and k, and consistent baselines. It deserves a round of revision, not desk rejection.","headline":"A plausible DTS-SQL extension, but the confidence signal is uncalibrated and the main gains may be dev-set threshold tuning.","tokens_in":16243,"tokens_out":2569,"would_cite":false,"duration_ms":24512,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By treating schema linking as a tunable precision-recall problem and using model internal confidence to prune beam-search candidates, CHS-SQL lifts sub-7B language models to state-of-the-art text-to-SQL accuracy on Spider and BIRD.","keywords":["text-to-SQL","schema linking","precision-recall trade-off","beam search","trace confidence","small language models","LoRA fine-tuning","execution accuracy"],"falsifier":"A calibration check: on a held-out set of schema-linking predictions, compute TC for correct vs incorrect traces. If the distributions overlap substantially or higher TC does not correspond to higher error rates, then thresholding on TC cannot be what drives the reported precision improvements, and the gains would be attributable to beam-search recall alone. Repeating the comparison on BIRD with a model whose probabilities are known to be miscalibrated (e.g., a quantized model under distribution shift) would test the assumption directly.","tokens_in":15365,"feed_emoji":"🔍","tokens_out":3588,"duration_ms":33682,"temperature":0.7,"pith_summary":"CHS-SQL claims that the precision-recall trade-off in schema linking is the decisive factor for text-to-SQL accuracy in small language models, and that it can be controlled explicitly. The paper introduces a two-stage framework: a schema linking module that uses beam search to increase recall and a trace-confidence filter to increase precision, followed by a SQL generation module that applies the same confidence-guided selection and an ensemble of models. Evaluated on Spider and BIRD with sub-7B models, it reports state-of-the-art execution accuracy, improving over DTS-SQL by 2 to 9 points. The core message is that balancing, not maximising, schema recall and precision is what drives downstream SQL correctness.","feed_headline":"Small-model text-to-SQL hits 0.773 with confidence-guided links","feed_subtitle":"Controlling precision-recall in schema linking lifts sub-7B models past much larger baselines on Spider and BIRD.","key_machinery":"Trace Confidence (TC): the standard deviation of per-token negative log-probabilities over a generated trace, interpreted as a model-certainty score. Low TC indicates high certainty and is used both to discard low-quality schema-linking traces (with a threshold θ) and to pick the final SQL among ensemble candidates (lowest TC). Beam width k and θ are the two hyperparameters that quantitatively set the precision-recall operating point; the paper also uses the 'All Columns Chosen' expansion to recover column recall once tables are selected.","core_discovery":"The central claim is that high recall of ground-truth tables and columns is a necessary precondition for accurate SQL generation, but that indiscriminate recall introduces noise that hurts performance; the optimal operating point is a controlled trade-off. CHS-SQL implements this with beam search over schema-linking traces (which raises recall) and a trace-confidence filter (which raises precision by discarding high-variance, low-certainty traces). The filtered high-recall table set is then expanded to include all columns of the chosen tables, further increasing column recall. During SQL generation, the same confidence heuristic selects the most stable query from beam-searched candidates pro","pith_inferences":["If trace confidence is a valid correctness signal, it generalises beyond SQL: any structured generation task with beam search could use per-trace variance to prune candidates, e.g., code generation or information extraction.","The paper's reported optimum favours recall over precision (table F1 0.793 vs 0.954 for baselines) — a reader might infer that for schema linking, missing a relevant table is costlier than including an irrelevant one, which could inform dataset design and evaluation weighting.","The limitation section implies an open problem: achieve 100% recall while raising precision. Combining CHS-SQL with hard constraints (e.g., string match on values/dates, as the 'TEXT/NUM/DATE match' variant does) is a concrete direction to push precision higher without sacrificing recall.","The ensemble result (single-model 0.736 vs ensemble 0.773 EX) suggests that diversity across models contributes beyond the schema linking gain; ablating model diversity separately would quantify that residual."],"forward_implications":["Schema linking should be treated as an explicit precision-recall trade-off problem, not just a filtering step, and evaluated with both table and column metrics.","Sub-7B models fine-tuned with LoRA can reach execution accuracies on Spider and BIRD that rival much larger models, making private, single-GPU text-to-SQL deployment feasible.","The same trace-confidence criterion can be reused at both pipeline stages — schema linking and SQL generation — suggesting a general method for uncertainty-aware generation with SLMs.","Adjusting beam width and confidence threshold provides a practical two-knob control for adapting text-to-SQL pipelines to different accuracy requirements or noisy schemas."],"fun_headline_variants":["Confidence-guided schema linking lifts small text-to-SQL to SOTA","Balance precision-recall in schema linking for better small-model SQL","CHS-SQL: small models beat big SQL baselines via confidence search","For small text-to-SQL, recall needs a confidence filter","Small-model SQL: control schema-linking noise to top leaderboards"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"Trace Confidence — the spread of token-level probabilities — is a reliable signal of whether a generated schema link or SQL query is actually correct; if it is not calibrated for a given model or dataset, the filtering mechanism that provides the precision gain breaks down.","fun_headline_variants_meta":{"raw":{"variants":["Confidence-guided schema linking lifts small text-to-SQL to SOTA","Balance precision-recall in schema linking for better small-model SQL","CHS-SQL: small models beat big SQL baselines via confidence search","For small text-to-SQL, recall needs a confidence filter","Small-model SQL: control schema-linking noise to top leaderboards"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000201,"raw_usage":{"total_tokens":1226,"prompt_tokens":765,"completion_tokens":461,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":509,"completion_tokens_details":{"reasoning_tokens":369}},"tokens_in":509,"tokens_out":461,"duration_ms":4789,"temperature":1.0,"reasoning_tokens":369,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T10:55:14.863497+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A calibration check: on a held-out set of schema-linking predictions, compute TC for correct vs incorrect traces. If the distributions overlap substantially or higher TC does not correspond to higher error rates, then thresholding on TC cannot be what drives the reported precision improvements, and the gains would be attributable to beam-search recall alone. Repeating the comparison on BIRD with a model whose probabilities are known to be miscalibrated (e.g., a quantized model under distribution shift) would test the assumption directly.","supporting_citations":[],"review_version":1}