{"id":"5efe01c6-022c-4d56-a2af-c3a9c0c0c039","arxiv_id":"1908.11052","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"An auxiliary word-to-column mapping task, trained jointly with a text-to-SQL generator, improves accuracy on WikiSQL, especially on questions about tables never seen in training.","lead":"This paper shows that text-to-SQL models make more mistakes on questions about tables they never saw during training, and proposes an extra training goal that maps question words to column names. Adding this goal lifts accuracy by about 3 points overall and 5 points on unseen tables in the WikiSQL benchmark, while also introducing a new zero-shot evaluation split.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Zero-shot claim rests on table-disjoint split with 72% seen column names; the 4.9% gain may reflect composition over familiar columns, not mapping of never-before-seen names.","rationale":"The reader's weakest_assumption identifies exactly the same load-bearing concern: W-0 is table-disjoint but not schema-disjoint, so 72% of column names in the zero-shot test have been seen in training. This is the central issue because the paper's contribution is framed as improved zero-shot generalization. The paper itself provides the crucial statistic (28% unseen column names) and even breaks down Gen-model versus Full-model by seen/unseen columns, but it never reports the headline baseline comparison on truly unseen columns. A targeted re-computation on a column-disjoint subset would settle whether the auxiliary mapping task actually confers an advantage on never-before-seen names or merely helps with novel combinations of familiar names. The concern is addressable and does not by itself invalidate the empirical finding that Full-model helps on W-0; it redefines the strength and scope of the claim. Therefore the existing CONDITIONAL verdict remains appropriate, pending the proposed re-analysis. I agree with the reader that this is the weakest and most load-bearing assumption.","tokens_in":9934,"tokens_out":4213,"duration_ms":40394,"concrete_test":"Construct a subset W-0-unseen of W-0 consisting of test questions whose gold WHERE condition column name does not occur in any training table schema, and the complementary subset W-0-seen. Recompute ACCqm and ACCex for C2F, Gen-model, and Full-model on both subsets. If Full-model's advantage over C2F on W-0-unseen is not at least as large as on W-0-seen, or if the advantage disappears entirely, then the headline zero-shot gain is an artifact of schema overlap rather than evidence of mapping never-before-seen names. Report bootstrap confidence intervals to determine whether the differences are significant.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The headline generalization claim depends on W-0 being a zero-shot test. Section 4.1 defines W-0 as test tables with 0 occurrences in training, but Section 5.2 explicitly states: 'On unseen test table schema, 28% column names never appear in training set.' Thus 72% of columns in W-0 tables have been seen individually during training. The auxiliary mapping task is trained to associate question words with column names from the training schema, so on W-0 most condition-column decisions can be made by selecting from familiar column names in a new table context. That is compositional generalization, not the 'true zero-shot' claimed in Section 2.2. The paper's own Figure 5(b) shows Full-model improves over Gen-model by 4.2% on unseen column names but only 2.1% on seen column names; however, the headline comparison in Figure 5(a) is against the C2F baseline and is not broken down by seen/unseen column names. Without that breakdown, the claimed 4.9% ACCqm / 4.4% ACCex advantage on W-0 cannot be attributed to handling never-before-seen names. Because the abstract and conclusion describe the result as zero-shot generalization to unseen data, this is a load-bearing gap rather than a minor presentation issue.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper addresses text-to-SQL generalization on WikiSQL. It introduces an auxiliary mapping task that tags condition-value spans in the question and maps those spans to condition column names, trained jointly with the generation model via a multi-task loss. The generation model itself uses a coarse-to-fine decoder enhanced with bi-attention and attentive pooling. On the full WikiSQL test set, the Full-model reaches 75.0% query-match accuracy and 81.7% execution accuracy, about 3% absolute above the coarse-to-fine baseline. On a new subset W-0 consisting of test tables with zero occurrences in training, the model improves over the baseline by 4.9% in query-match accuracy and 4.4% in execution accuracy. The paper includes ablations (removing attentive pooling, removing the mapping task) and a case study attributing the zero-shot gains to improved condition-column prediction.","tokens_in":10181,"tokens_out":5161,"duration_ms":46276,"significance":"The contribution is a simple and plausible auxiliary objective for improving cross-table generalization, together with a new table-frequency split (W-0) that exposes a generalization gap in existing WikiSQL models. The ablations and the case study provide useful diagnostic evidence. The main caveat is that W-0 is table-disjoint but not schema-disjoint: Section 5.2 reports that only 28% of column names in W-0 are unseen at training time, so the headline 'zero-shot' claim is partially a claim about compositional generalization over familiar column names. If the authors add column-level zero-shot analysis and temper the terminology, the paper would be a solid incremental contribution to text-to-SQL and semantic parsing generalization.","major_comments":[{"comment":"The abstract and Section 2.2 describe W-0 as a 'zero-shot' testbed, but W-0 is defined by table frequency, not by schema novelty. Section 5.2 states that only 28% of column names on unseen test tables never appear in the training set, meaning 72% of column names in W-0 have been seen individually. The auxiliary mapping task is trained on those seen column names, so W-0 mostly measures robustness to new combinations of familiar column names. The paper should either (a) explicitly qualify the claim as table-level zero-shot, or (b) add a column-level zero-shot subset (e.g., test conditions whose column name never occurs in training) and report the headline C2F-vs-Full comparison on that subset. Without such a restriction, the conclusion that the model generalizes to 'unseen data' is not supported at the schema level.","section":"§4.1, §5.2, Abstract"},{"comment":"The 4.9% ACCqm and 4.4% ACCex improvement on W-0 in Figure 5(a) is reported against the C2F baseline, which lacks both the bi-attention layer and the attentive pooling of Gen-model. The paper does not break this headline comparison down by seen versus unseen column names. Figure 5(b) provides a seen/unseen breakdown only for Gen-model versus Full-model, showing a 4.2% gain on unseen column names and 2.1% on seen column names. To support the claim that the auxiliary mapping task is responsible for the zero-shot improvement, the authors should report the same seen/unseen column-name breakdown for C2F and Gen-model as well, or otherwise isolate the contribution of the mapping task from the architectural changes.","section":"§5.2, Figure 5"},{"comment":"The auxiliary mapping task is the main novelty, but it is never evaluated directly (e.g., precision and recall of value-to-column mapping). The case study in Section 5.3 uses 100 samples per category with overlapping categories, and while it suggests that the mapping task corrects wrong condition-column predictions, it does not quantify how often the mapping task is responsible for the improvement on W-0. The authors should report direct mapping accuracy on W-0 or an analysis of the mapping task's predictions, to make the mechanism claim testable.","section":"§3.2, §3.3, §5.3"},{"comment":"No variance or significance estimates are reported. The absolute gains are 1.5–1.6% on the full set and 4.9% on W-0; given typical run-to-run variance in neural text-to-SQL models, the reader cannot determine whether the overall gains are statistically reliable. The authors should report results over multiple random seeds with standard deviations, or at least provide a bootstrap significance test for the main comparisons (Full-model vs. baseline and Full-model vs. Gen-model) in Tables 1 and Figure 5.","section":"§5, Tables 1 and 5"}],"minor_comments":[{"comment":"The sentence 'Experimentally, We evaluate our models...' has an inconsistent capitalization of 'We'; please fix the capitalization and punctuation throughout.","section":"Abstract"},{"comment":"The phrase 'true zero-shot sub test-set' is ambiguous and arguably inaccurate given the column-name overlap discussed in Section 5.2; please define zero-shot precisely at the schema level or avoid the word 'true'.","section":"§2.2"},{"comment":"The word 'meassure' is a typo for 'measure'; also, in Eq. (3), the bias term b is written without a subscript, while the preceding text uses b with attention parameters; please make the notation consistent.","section":"§3.1"},{"comment":"'300-dim Glove' should be '300-dim GloVe'.","section":"§4.3"},{"comment":"The sentence 'We believe this is due to the fact that table itself is the other fact affecting models’ performance' contains a redundant and confusing use of 'fact'; please rephrase, e.g., 'table identity is another factor affecting performance.'","section":"§5.1"},{"comment":"The definitions of Case-Correct and Case-Wrong are easy to misread; please state explicitly that Case-Correct contains examples where Full-model is correct and Gen-model is wrong, and Case-Wrong the reverse. Also clarify that an example can belong to more than one error category.","section":"§5.3, Table 3"},{"comment":"The IncSQL reference is garbled: 'Tianze Shi, Kedar Tatwawadi, Kaushik Chakrabarti, Yi Zhang, and Chen Weizhu Polozov, Oleksandr...' should be corrected to list all authors accurately.","section":"References"},{"comment":"The footnote says 'Our code will be released after paper is reviewed.' If this manuscript is under journal review, please provide a statement of code availability consistent with the journal's policy, or include an anonymized release mechanism.","section":"Footnote 1"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an arXiv preprint (v1) and appears to be submitted without a clear venue; the zero-shot terminology is likely to attract criticism from reviewers, so the editor may wish to require the authors to qualify the claim or add a column-level zero-shot experiment. The reference list also contains several formatting errors that should be corrected before final acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the paper is worth refereeing. It contributes a clean diagnostic—a table-disjoint split of WikiSQL (W-0)—and a simple auxiliary mapping task (detect condition values, point them to column names) that consistently improves a strong baseline. The headline numbers are real in the sense that they are on the standard WikiSQL test plus the new split; the main caveat is that \"zero-shot\" is doing more work than it should.\n\nThe W-0 split is the most useful piece. The authors show that the original WikiSQL train/test split has 70% schema overlap, so the standard evaluation overstates generalization. Splitting by table occurrences gives a harder subset, and their model keeps most of its gain there. That alone is a contribution future work can adopt.\n\nThe method is straightforward: a two-stage mapper that first BIO-tags condition values in the question, then uses a pointer to select the column. Training it jointly with the generator adds a small but consistent boost (1.5–1.6% overall; 4.9% query-match, 4.4% execution on W-0). The case study attributing the gain to fixing wrong condition columns is credible, and the ablation showing a larger gain on never-seen column names (4.2% vs 2.1%) supports the proposed mechanism.\n\nSoft spots, in order: no significance tests or variance numbers, and no released code. Those are addressable. More substantively, the W-0 split is zero-shot only at the table level: 72% of column names in unseen tables appear in training. The stress-test note is right that the headline 4.9% versus the baseline is not broken down by seen/unseen columns, so we cannot attribute it entirely to handling novel surfaces. But the paper explicitly acknowledges the 28% number and provides the seen/unseen breakdown for Gen-model vs Full-model, so the concern is partially mitigated. A revision should add the same breakdown for the baseline comparison and soften \"zero-shot\" to \"unseen tables\" or at least define it clearly.\n\nBottom line: a solid empirical contribution for the text-to-SQL and semantic parsing community. A good referee could help sharpen the claims and fill the reproducibility gaps. I would send it out.","headline":"A genuinely useful zero-shot split for WikiSQL plus a simple auxiliary mapping task that helps; the 'zero-shot' label oversells it slightly because 72% of column names appear in training.","tokens_in":10702,"tokens_out":2732,"would_cite":false,"duration_ms":25576,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"An auxiliary task that maps question words to column names improves text-to-SQL accuracy on tables never seen in training.","keywords":["text-to-SQL","zero-shot learning","auxiliary task","WikiSQL","semantic parsing","column-name mapping","multi-task learning","generalization"],"falsifier":"Take the W-0 subset and further restrict it to questions whose tables contain no column name that appeared in any training table; if the Full-model's advantage over the baseline on that restricted subset is close to zero or vanishes, the claim that the auxiliary mapping task teaches generalization to novel column names is refuted.","tokens_in":9755,"feed_emoji":"🗄️","tokens_out":7926,"duration_ms":66010,"temperature":0.7,"pith_summary":"Existing text-to-SQL models are usually evaluated on a test split where most tables share schemas with training data, which masks how they behave on tables they have never seen. This paper constructs a zero-shot subset of WikiSQL, W-0, where the test table has zero occurrences in training, and shows that accuracy on the WHERE clause drops sharply for such tables. The paper's central proposal is an auxiliary mapping task that explicitly learns to pair words in the question with column names, trained jointly with the SQL generation model as a regularizer. On the full WikiSQL test set the resulting Full-model reaches 81.7% execution accuracy, about three absolute points above the coarse-to-fine baseline; on W-0 the gain grows to 4.4-4.9 absolute points. The main source of the improvement is better prediction of condition columns in WHERE clauses, which is exactly what the mapping task targets.","feed_headline":"Auxiliary mapping lifts zero-shot text-to-SQL by 5 points","feed_subtitle":"A word-to-column mapping task raises WikiSQL execution accuracy to 81.7% and powers gains on unseen tables.","key_machinery":"The load-bearing mechanism is the auxiliary mapping model: a BIO tagger over the question that labels condition-column and condition-value spans, followed by a pointer network that maps each word tagged as part of a condition value to a column name. It shares the question and table-schema encoders with the main generation model, so the multi-task objective $\\lambda L_{gen} + (1-\\lambda)L_{map}$ regularizes the shared representations. The generation model itself is the coarse-to-fine decoder baseline, augmented with bi-directional question-to-table attention and attentive pooling for the AGG/SEL decision. The mapping task's job is to force the shared representations to encode the alignment between question entities and schema columns, the ability that degrades on unseen tables.","core_discovery":"The paper claims that the poor zero-shot performance of neural text-to-SQL models is largely a column-mapping problem: the decoder must point from a natural-language expression to a column name it may never have seen in training. To address it, the authors add a two-step auxiliary task to a coarse-to-fine seq2seq generation model. The auxiliary model first tags question words as beginning or inside of a condition value or condition column, then points each predicted condition value to a table column. This mapping loss is added to the generation loss with a mixing weight, so the shared encoder learns question and schema representations that support explicit word-to-column alignment. The paper's evidence is that the Full-model beats the generation model alone by 4.2 points on WHERE conditions whose column name was unseen in training and by 2.1 points on seen column names, and that error analysis attributes most corrected errors to wrong condition-column predictions.","pith_inferences":["Editorial inference: the paper's zero-shot set still shares 72% of column names with training data, so the headline gain of 4.4-4.9 points mostly tests robustness to recombining familiar column names into new schemas; a stricter subset with entirely novel schemas would be a harder test of the mapping task.","Editorial inference: the same two-step mapping idea could transfer to other semantic-parsing problems where a span of input text must be aligned to an output structure, such as slot filling or knowledge-base query generation, and the gains there could be similar in size.","Editorial inference: because the mapping model learns only from words tagged as condition values, an ablation that replaces the tagging gate with raw word-to-column attention over all words would isolate whether the explicit selection of condition values is what drives the gain."],"forward_implications":["If the Full-model generalizes as the paper argues, adding an explicit word-to-column mapping task should keep helping whenever a decoder must point to schema elements, not just on WikiSQL.","Because the zero-shot testbed W-0 is a stricter evaluation than the original WikiSQL split, reported gains on W-0 indicate that part of the prior test accuracy was inflated by shared table schemas.","Since the gain is concentrated in WHERE-clause condition-column prediction, other models that struggle on WHERE should benefit most from the same auxiliary mapping regularizer.","The multi-task loss can be added to any sketch-based text-to-SQL decoder without changing the decoder's inference procedure; the only additional training-time need is span labels for condition values and columns."],"supporting_citations":[{"why":"It supplies the coarse-to-fine baseline model and the WHERE decoder that the paper augments with bi-attention and attentive pooling.","marker":"Dong and Lapata, 2018"},{"why":"It supplies WikiSQL, the dataset the paper re-splits to form the zero-shot testbed W-0.","marker":"Zhong et al., 2017"},{"why":"It supplies the SQLNet baseline and the query-match and execution-accuracy metrics used in all experiments.","marker":"Xu et al., 2017"},{"why":"It motivates the paper's concern that text-to-SQL evaluation should measure generalization to unseen data.","marker":"Finegan-Dollak et al., 2018"},{"why":"It supplies TypeSQL, the type-aware baseline whose generalization behavior the paper compares against.","marker":"Yu et al., 2018a"},{"why":"It supplies the bi-directional attention mechanism used to enhance question-schema interaction.","marker":"Seo et al., 2017"},{"why":"It supplies the attentive pooling layer that improves AGG and SEL decoding.","marker":"dos Santos et al., 2016"}],"fun_headline_variants":["Auxiliary task adds 5 points to zero-shot text-to-SQL","Column mapping key to zero-shot SQL gains of 5 points","Word-to-column mapping boosts zero-shot SQL by 5%","Zero-shot SQL: auxiliary mapping yields 5-point gain","Simple mapping task lifts zero-shot text-to-SQL accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing assumption is that 'zero-shot' can be measured by table occurrence in training; on W-0, 72% of column names on the unseen test tables do appear somewhere in training, so the reported advantage may mostly reflect handling new schemas composed of known column names rather than truly never-before-seen names.","fun_headline_variants_meta":{"raw":{"variants":["Auxiliary task adds 5 points to zero-shot text-to-SQL","Column mapping key to zero-shot SQL gains of 5 points","Word-to-column mapping boosts zero-shot SQL by 5%","Zero-shot SQL: auxiliary mapping yields 5-point gain","Simple mapping task lifts zero-shot text-to-SQL accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000311,"raw_usage":{"total_tokens":1761,"prompt_tokens":922,"completion_tokens":839,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":538,"completion_tokens_details":{"reasoning_tokens":752}},"tokens_in":538,"tokens_out":839,"duration_ms":7705,"temperature":1.0,"reasoning_tokens":752,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:25:45.295950+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the W-0 subset and further restrict it to questions whose tables contain no column name that appeared in any training table; if the Full-model's advantage over the baseline on that restricted subset is close to zero or vanishes, the claim that the auxiliary mapping task teaches generalization to novel column names is refuted.","supporting_citations":[{"cited_title":"Kummerfeld, Li Zhang, Karthik Ramanathan, Sesh Sadasivam, Rui Zhang, and Dragomir Radev","cited_arxiv_id":null,"evidence_quote":"It motivates the paper's concern that text-to-SQL evaluation should measure generalization to unseen data."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the bi-directional attention mechanism used to enhance question-schema interaction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the attentive pooling layer that improves AGG and SEL decoding."}],"review_version":1}