{"id":"b77d0b4d-5b15-4419-a591-9d16c7b6f70a","arxiv_id":"2506.11986","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":8,"one_line_summary":"Schema-R1 combines cold-start SFT on 200 CoT samples with GRPO rule-based RL, reporting table and column filter accuracy gains of 10 percentage points or more over a fine-tuned baseline on Spider-dev.","lead":"This paper trains small open language models to reason about which database tables and columns a text-to-SQL question needs, using a short supervised warm-up followed by reinforcement learning. On the Spider benchmark, the method reports large gains in schema-linking accuracy over standard fine-tuning, which could make on-premise text-to-SQL assistants more reliable.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (7) defines the table reward as len(t*_i - t'_i), which counts missed tables, not correctly predicted tables; an empty prediction would maximize this reward, so the written reward cannot explain the reported gains unless Eq. (7) is a typo.","rationale":"The reader's weakest assumption identifies the same load-bearing concern I find: the set-difference expression in Eq. (7) is not just a minor notation slip but would be catastrophic if implemented, because it rewards omissions rather than hits. The paper's own description confirms the authors intended the first term to represent correctly predicted tables, so a typo is the most plausible explanation. This concern is empirically testable by checking the supplementary code, and it does not by itself invalidate the central claim—it makes the method description unreliable pending that check. Since the reader already issued a CONDITIONAL verdict based on this exact issue, my read does not change the verdict. I agree with the reader's emphasis and would keep the conditional status until the code is verified. I did not find another concern that would be more load-bearing: the comparison to DTS-SQL is within a consistent single-dataset setup, the training/test split is described, and the reported improvements are large enough that a single seed is not the primary threat. The missing hyperparameters for the length reward and the absence of error bars are secondary reproducibility issues, but they do not make the claimed mechanism impossible.","tokens_in":10317,"tokens_out":3506,"duration_ms":42182,"concrete_test":"Inspect the released code at https://github.com/hongWin/Schema-R1/ and locate the schema linking reward computation. Determine whether the positive term uses len(set(t_star) & set(t_pred)) (intersection) or len(set(t_star) - set(t_pred)) (set difference). If the code uses intersection, Eq. (7) is a notation error and the reported gains are internally consistent; if the code uses the literal set difference, retrain Qwen2.5-1.5B on the 8329 GRPO samples with that reward and check whether FilteredAcc approaches 89.94. Under the literal reward, the model should collapse toward empty predictions, falsifying the central claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central empirical claim is that Schema-R1's GRPO training improves table and column FilteredAcc by at least 10% over DTS-SQL (Table 1: 89.94 vs 75.0; Table 2: 68.82 vs 42.43 for Qwen2.5-1.5B). This result depends on the RL reward providing positive signal for correct predictions. In Eq. (7), the 'correctly predicted table set' is written as t*_i - t'_i, which in set notation is the set of ground-truth tables missed by the prediction, not the intersection t*_i ∩ t'_i. The first reward term therefore rewards missed tables, and because it is normalized by len(t*_i), predicting no tables at all yields the maximum possible Rst (since t*_i - ∅ = t*_i, and the penalty term is zero). The second term penalizes false positives via t'_i - t*_i, but this does not compensate for rewarding omissions. If the implementation matched the written equation, the training reward would favor degenerate empty outputs, contradicting the reported reward curves in Fig. 2 and the FilteredAcc improvements. The paper's own text states that '(t*_i - t'_i)' is the correctly predicted set, which is mathematically wrong, and also contains a typo ('Ptmax' for the reward scale). Thus the described reward function is internally inconsistent with the reported results. For the central claim to hold, the implemented reward must use an overlap-based term (e.g., len(t*_i ∩ t'_i)). This is the most load-bearing concern because it directly affects whether the proposed method, as specified, can produce the claimed outcomes. Secondary reproducibility gaps (missing Lower/Upper Length values, no seed/error bars) are less critical because they do not make the result impossible as written.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes Schema-R1, a three-stage training approach for schema linking in Text-to-SQL: (1) constructing 200 high-quality chain-of-thought samples via commercial LLMs (DeepSeek-R1), (2) supervised fine-tuning for cold-start initialization, and (3) GRPO-based reinforcement learning with rule-based rewards for table/column prediction. Experiments on Spider-dev with Qwen2.5-0.5B and Qwen2.5-1.5B report substantial improvements in table and column FilteredAcc over a DTS-SQL fine-tuned baseline, e.g., 89.94 vs. 75.0 for 1.5B table prediction. The paper claims at least a 10% improvement in filter accuracy and provides a GitHub repository.","tokens_in":10747,"tokens_out":4751,"duration_ms":62352,"significance":"If the results are reproducible, the work is a useful demonstration that reasoning-focused RL can improve a downstream NLP task like schema linking, even starting from a small cold-start SFT set. The approach is simple, uses open models, and targets a privacy-relevant application. The authors provide code, and the three-stage pipeline (prompt-based CoT generation, cold-start SFT, rule-based GRPO) is clearly a reasonable template for similar tasks. However, the manuscript as written contains a load-bearing notation error in the reward definition, omits key training hyperparameters, and reports single-run results without error bars, so the empirical claims are not yet fully substantiated.","major_comments":[{"comment":"The authors must clarify that the set difference in Eq. (7) is a typographical error and provide the correct reward formula. Without this correction, the method as described cannot work.","section":"Sec. 4.2, Eq. (7)-(8)"},{"comment":"The paper must include the missing reward and R L hyperparameters, as the reported gains are the core contribution.","section":"Sec. 5.1, training settings"},{"comment":"The authors need to add error bars or multiple runs to support the quantitative claims.","section":"Tables 1 & 2 and Sec. 5.1"},{"comment":"A precise, formal definition of FilteredAcc is needed; the current one-sentence description is insufficient.","section":"Sec. 5.1, evaluation metrics"}],"minor_comments":[{"comment":"The acronym 'GPRO' appears in the introduction ('We employ GPRO with the SFTed model as reference'); it should be 'GRPO'.","section":"Sec. 1"},{"comment":"The caption contains a typo: 'Rlue-Base' should be 'Rule-Based'.","section":"Fig. 1 caption"},{"comment":"In the text following Eq. (7), 'the average reward for each correctly predicted item is calculated as Ptmax/len(t*_i)' should read 'Rtmax/len(t*_i)'. Using 'Ptmax' for the reward scale is confusing.","section":"Sec. 4.2"},{"comment":"The inference engine is written as 'VLLM'; the correct name is 'vLLM'.","section":"Sec. 5.1"},{"comment":"The caption says 'Performance of table prediction for different method in Columns prediction Task'; it should say 'column prediction'.","section":"Table 2 caption"},{"comment":"In the prompt template, the XML-like tags are inconsistent: '<database>...<database>' should be closed as '</database>'.","section":"Appendix A"},{"comment":"The GRPO objective as typeset has mis-matched brackets and the KL penalty is written inside the expectation; a cleaner, complete formulation (including the definition of the advantage A-hat) would help reproducibility.","section":"Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The core idea is sound and relevant, but the manuscript is not yet ready for publication. The Eq. (7) set-difference error is a serious flaw in the method description, even though it is likely a notation slip; it must be fixed and the implementation clarified. The missing hyperparameters and lack of error bars will require a full experimental revision. I see no evidence of fabrication, but the empirical claims are under-supported as written."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: Schema-R1 is a straightforward but legitimate extension of the Table-R1/Reasoning-Table line: cold-start SFT on a few hundred CoT samples, then GRPO with rule-based rewards, applied to schema linking with both table and column prediction. On Spider-dev it reports double-digit FilteredAcc gains over DTS-SQL for Qwen2.5-1.5B. The gains are plausible and the recipe is clearly described. The paper also ships code, which is good.\n\nThe genuinely new piece is the two-level reward structure and the demonstration that a 1.5B model can beat a stronger SFT baseline by reasoning. That is worth knowing for anyone building privacy-sensitive text-to-SQL pipelines. The paper is honest about not testing larger models.\n\nThe soft spots: Eq. (7) defines the table reward using t*_i - t'_i and calls it the correctly predicted set. That is the set of missed tables. As written, an empty prediction would maximize Rst, which contradicts the reported training curves and results. I suspect it's a notation error and the implementation uses t*_i ∩ t'_i, but the paper must say so. This is not a minor typo; it is the load-bearing reward definition. Also missing are the reward scale hyperparameters (Rtmax etc.), the Lower/Upper length bounds, and any seed variance. Single-run results with no error bars are acceptable as a short paper, but they limit how strongly we can read the 10% claim.\n\nThe citation pattern looks fine; [42,43] are properly credited as the direct predecessors. The related work is adequate for an arXiv preprint.\n\nBottom line: I'd send this to review. The core idea is sound and the empirical claim, if the reward implementation is corrected, is useful. The referee should ask for Eq. (7) to be fixed, the missing hyperparameters reported, and ideally a second seed for the main table. With those, it would be a solid short paper.","headline":"A useful and plausible empirical extension of reasoning RL to schema linking, but Eq. (7) as written describes a reward that would reward missing tables, and that has to be fixed before the central claim is fully supported.","tokens_in":11264,"tokens_out":1668,"would_cite":true,"duration_ms":124923,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Schema-R1 claims that a small language model trained for schema linking with 200 chain-of-thought samples plus GRPO reinforcement learning beats a fully supervised fine-tuning baseline on Spider-dev.","keywords":["Text-to-SQL","schema linking","reasoning model","reinforcement learning","GRPO","chain-of-thought","supervised fine-tuning","Spider benchmark"],"falsifier":"Look at the released code for the schema-linking reward: if it computes the reward exactly as written, using the set difference of ground truth minus prediction, the reported improvement should vanish; a reproduction run with that code would decide whether the equation or an overlap-based implementation produced the numbers.","tokens_in":10126,"feed_emoji":"🧠","tokens_out":7583,"duration_ms":88722,"temperature":0.7,"pith_summary":"Schema-R1 claims that a schema-linking model for Text-to-SQL can be trained to reason instead of memorize by combining a tiny supervised cold start with reinforcement learning. The authors take 200 examples from Spider, prompt a large language model to write chain-of-thought explanations for the known correct answers, fine-tune a small open model on those examples, then train with GRPO using rule-based rewards for format, reasoning length, and schema-link accuracy. On Spider-dev the 1.5B model reaches 89.94 table FilteredAcc and 68.82 column FilteredAcc, beating the DTS-SQL fine-tuning baseline by about 15 and 26 percentage points respectively. This matters because schema linking is the step that tells a Text-to-SQL system which tables and columns to use, and the recipe needs only 200 labeled reasoning samples rather than a large supervised set.","feed_headline":"Schema linking: 200 reasoning samples plus RL beat full SFT","feed_subtitle":"A 1.5B model jumps from 75.0 to 89.94 filtered table accuracy on Spider-dev; columns gain about 26 points.","key_machinery":"The load-bearing mechanism is a three-stage training pipeline: prompt-based chain-of-thought generation, cold-start supervised fine-tuning, and GRPO reinforcement learning. In the first stage, prompt templates containing the question, database schema, and ground-truth table/column sets are sent to DeepSeek-R1 to reconstruct the missing reasoning steps, yielding 200 chain-of-thought samples. Stage two is cold-start SFT on those samples, teaching the model to emit response sections for reasoning and for the final schema link. Stage three is GRPO, a group-relative policy optimization that draws multiple responses per query and computes advantages from three rule-based rewards: a format reward, a reasoning-length reward, and a schema-linking reward based on filter accuracy. The schema-linking reward is what ties the reinforcement signal directly to the task.","core_discovery":"The paper's central claim is that decoupling schema linking from SQL generation and training a dedicated small language model with reasoning-oriented reinforcement learning outperforms conventional supervised fine-tuning. The evidence is the comparison on Spider-dev: cold-start SFT alone underperforms full SFT, but adding the GRPO stage lifts the 1.5B model to 73.21 exact match and 89.94 FilteredAcc for tables and 38.24 exact match and 68.82 FilteredAcc for columns, while the DTS-SQL baseline reaches 64.84 and 75.0 for tables and 31.17 and 42.43 for columns. The paper interprets the jump as a shift from rote learning to reflective reasoning: the model learns to generate thinking traces that lead to correct answer table and column sets.","pith_inferences":["The cold-start-plus-RL recipe separates the reasoning trace from the final answer, which should make schema-linking decisions easier to audit and correct in deployed text-to-SQL pipelines.","The same recipe may transfer to other structured-output prediction tasks where the answer is a set of schema elements, such as entity linking or feature selection.","Because the reward uses filter accuracy, the model optimizes coverage of correct schema elements; pairing the reward with an end-to-end SQL correctness signal could close the gap between schema linking and final query accuracy.","A direct next test would be running the same 200-sample cold start plus GRPO on other benchmarks such as BIRD, or on larger open models, to see whether the gains persist and grow with scale."],"forward_implications":["Cold-start SFT on 200 samples plus GRPO reasoning training exceeds full-data supervised fine-tuning on both table and column prediction, so small curated reasoning sets can substitute for large supervised sets.","Models trained this way produce explicit reasoning traces, making their schema-link decisions inspectable and eligible for rule-based rewards.","The 1.5B model benefits more from the reasoning stage than the 0.5B model, suggesting the approach scales with model size.","The same reward design covers both table prediction and column prediction, allowing a single model to output the full schema link."],"supporting_citations":[{"why":"DTS-SQL supplies the supervised fine-tuning baseline, the schema-linking task formulation, and the filter-accuracy evaluation that Schema-R1 is measured against.","marker":"[11]"},{"why":"DeepSeek-R1 is the model prompted to generate the 200 chain-of-thought training samples and the reference for RL reasoning.","marker":"[18]"},{"why":"This work introduces GRPO, the group-relative policy optimization algorithm used in Schema-R1's RL stage.","marker":"[19]"},{"why":"Chain-of-thought prompting is the reasoning-elicitation technique whose high-quality samples the cold-start stage aims to provide.","marker":"[17]"},{"why":"SQL-R1 demonstrates reinforcement-learning training for natural-language-to-SQL reasoning, the adjacent setting Schema-R1 extends.","marker":"[2]"},{"why":"Reasoning-Table applies RL to table prediction in Text-to-SQL, the predecessor that Schema-R1 extends to column prediction.","marker":"[43]"},{"why":"Table-R1 is the other prior RL table-reasoning approach that Schema-R1 compares itself against.","marker":"[42]"}],"fun_headline_variants":["RL with just 200 reasoning samples boosts schema linking accuracy","Reasoning RL lifts schema linking to 89.9% filtered table accuracy","1.5B model with RL reasoning outperforms SFT on schema linking","Schema linking via RL: 200 samples, 10% better than baselines","Cold-start SFT + GRPO gives 89.9 table accuracy for schema linking"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The central claim rests on the reward in Eq. (7) and Eq. (8) being implemented as an overlap between predicted and ground-truth table and column sets, not as the set difference the printed equations show, since a set difference would fail to reward correct predictions and the reported gains would not be possible.","fun_headline_variants_meta":{"raw":{"variants":["RL with just 200 reasoning samples boosts schema linking accuracy","Reasoning RL lifts schema linking to 89.9% filtered table accuracy","1.5B model with RL reasoning outperforms SFT on schema linking","Schema linking via RL: 200 samples, 10% better than baselines","Cold-start SFT + GRPO gives 89.9 table accuracy for schema linking"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000261,"raw_usage":{"total_tokens":1561,"prompt_tokens":881,"completion_tokens":680,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":580}},"tokens_in":497,"tokens_out":680,"duration_ms":7714,"temperature":1.0,"reasoning_tokens":580,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:59:39.766106+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Look at the released code for the schema-linking reward: if it computes the reward exactly as written, using the set difference of ground truth minus prediction, the reported improvement should vanish; a reproduction run with that code would decide whether the equation or an overlap-based implementation produced the numbers.","supporting_citations":[{"cited_title":"DTS-SQL: Decomposed text-to-SQL with small large language models","cited_arxiv_id":null,"evidence_quote":"DTS-SQL supplies the supervised fine-tuning baseline, the schema-linking task formulation, and the filter-accuracy evaluation that Schema-R1 is measured against."},{"cited_title":"Chain-of-thought prompting elicits reasoning in large language models.Advances in neural information processing systems, 35:24824–24837, 2022","cited_arxiv_id":null,"evidence_quote":"Chain-of-thought prompting is the reasoning-elicitation technique whose high-quality samples the cold-start stage aims to provide."}],"review_version":1}