REVIEW 2 major objections 5 minor 16 references
Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval
T0 review · 2 major / 5 minor · reviewed 2026-08-02 · deepseek-v4-flash
Pith's one-line read Corpus-adaptive fine-tuning establishes schema retrieval as a standalone task, lifting a 305M-parameter embedder to match 4–8B models.
desk verdict The benchmark is a real contribution; the 'label-free' headline is not supported by the paper's own recipe — Appendix A mixes in labeled Spider/BIRD training pairs and no ablation isolates the synthetic data. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
Corpus-adaptive fine-tuning: a three-step recipe that (1) anchors LLM query generation on sampled schema documents so gold sets are known by construction, (2) mines hard negatives within each (source, granularity) pool to avoid cross-granularity confusion, and (3) contrastively fine-tunes an embedder. This machinery carries the argument by making label-free adaptation possible and by isolating granularity-specific signals that preserve table-level retrieval.
What would settle it
Re-annotate a random sample of queries with human-identified gold columns, including those used implicitly, and recompute recall@10 for the adapted 305M model and the 4B/8B baselines; if the adapted model no longer matches the larger models on the corrected labels, the claim of cheap parity is undermined.
Extended reading notes
Core claim
The paper establishes schema retrieval — ranking a database's tables and columns for a natural-language question — as a first-class retrieval task and shows it can be solved cheaply. From five text-to-SQL datasets recast at table and column granularity, off-the-shelf embedders transfer poorly, especially on value-only and large schemas. The paper's recipe, corpus-adaptive fine-tuning, synthesizes queries from the schema itself, mines granularity-aware hard negatives, and contrastively trains a 305M model: recall@10 rises from 60.4 to 75.6, best under 1B parameters and competitive with 4–8B embedders; the same recipe raises an 8B model to the best nDCG@10 (71.7). Leave-one-out and leakage che
Load-bearing premise
The benchmark's gold relevance sets are obtained by parsing table and column references from ground-truth SQL, which the paper's own limitations note can miss implicitly used columns; if many gold columns are absent, both the metric and the measured improvements are computed against incomplete labels.
Editorial extensions
If this is right
- Schema linking should be benchmarked as its own retrieval task rather than inferred from end-to-end text-to-SQL accuracy.
- A 305M model adapted with only the target schema corpus can rival 4–8B general embedders, so label-free corpus adaptation is a viable enterprise deployment route.
- The same recipe improves a near-state-of-the-art 8B model, showing that adaptation complements, not substitutes for, model scale.
- Granularity-aware hard negatives and table-level positives are load-bearing: removing them drops table-level recall@10 from 80.9 to 52.4, below the base model.
- Adaptation improves retrieval most on abbreviated, code-like column names, the dominant case in real schemas.
Reading between the lines
- If transferable beyond SQL, this recipe gives a general label-free route to adapting retrievers to any structured corpus — APIs, functions, knowledge-base entries — for tool-augmented agents.
- The concentration of gains on code-like identifiers suggests the adapter partly learns a mapping from business concepts to enterprise naming conventions; explicit identifier normalization could push this further.
- Because gains are steepest on value-only and large-schema settings, enterprise text-to-SQL systems could use this to avoid passing entire schemas to large models, cutting latency and cost.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces schema retrieval as a first-class retrieval task, recasting five text-to-SQL datasets (Spider, BIRD, BEAVER, and two LiveSQLBench variants) into table- and column-level retrieval benchmarks under two document representations. It evaluates a range of off-the-shelf embedders, showing that they transfer poorly, and proposes corpus-adaptive fine-tuning: the target schema corpus is used to synthesize queries with document-anchored generation, mine granularity-aware hard negatives, and contrastively fine-tune a 305M embedder (Arctic-Embed-M) or an 8B embedder (Qwen3-Embedding-8B). The authors report large gains (e.g., average recall@10 from 60.4 to 75.6 for the 305M model), competitive performance against 4–8B models, and provide a leave-one-corpus-out experiment and a leakage audit as evidence of generalization. The central claim is that schema retrieval can be solved cheaply by label-free, corpus-adaptive retrieval.
Significance. If the central claim holds, the paper makes a useful contribution: it provides a reproducible benchmark for a genuinely under-evaluated retrieval task, systematically compares baselines, and proposes a lightweight adaptation recipe with plausible practical value. The evaluation design is generally careful: it includes an ablation, a leave-one-corpus-out transfer check, a leakage audit, and per-corpus results rather than only a single aggregate. The public release of the benchmark is a practical asset. However, the paper's headline 'label-free' characterization is not supported by the training recipe as described, because labeled schema-linking pairs from Spider and BIRD are mixed into the contrastive fine-tuning data. Since Spider and BIRD are also the two largest evaluation components, the claimed gains on those corpora may be substantially attributable to supervised fine-tuning rather than to the proposed corpus-adaptive synthetic-query mechanism. This is the main load-bearing issue and needs to be resolved before the central claim can be accepted.
major comments (2)
- [§4 step (3); Appendix A; Abstract and Discussion] The paper repeatedly describes the method as 'label-free' and states that it 'requires only the schema corpus' (Abstract; Discussion). However, Appendix A says that during contrastive fine-tuning the authors 'mix the in-domain synthetic data with schema-linking pairs derived from the public text-to-SQL training splits of Spider and BIRD.' These are labeled query–schema pairs, and they come from the same two datasets that make up the largest evaluation components (Spider: 2,147 queries; BIRD: 1,534 queries). The claimed gains on these corpora (e.g., BIRD value-only 75.3→97.5; Spider value-only 67.3→91.5) may therefore be largely due to standard supervised fine-tuning, not to the proposed corpus-adaptive synthesis. The leave-one-corpus-out experiment in Appendix C does not resolve this because it only removes the held-out corpus's synthetic queries; the external labeled Spider/BIRD pairs r
- [§3; Limitations] The benchmark's gold relevance sets are obtained by 'parsing table and column references from the ground-truth SQL.' As the Limitations acknowledge, this can miss columns that are used implicitly. If a substantial fraction of gold columns are missed, then both the metric values and the model rankings are computed against an incomplete ground truth, which could change conclusions about which models or which adaptations are best. The paper should provide a quantitative audit of this parsing step, for example by comparing parsed gold sets against human annotation on a sample of queries from each dataset, and report the missed-column rate. This is important for the benchmark's validity as a standalone evaluation resource.
minor comments (5)
- [Tables 2–3] The headline 'average recall@10' is an unweighted mean over nine heterogeneous settings with very different numbers of queries (e.g., 1,534 BIRD queries vs. 332 LiveSQLBench-Large queries). The per-corpus results are reported, which is good, but the abstract and discussion emphasize the unweighted average. Consider also reporting weighted averages or per-corpus aggregates so that the headline is not dominated by small, high-variance settings.
- [Table 6] Table 6 includes only four models, but the text says the corpus-adapted models 'exceed every off-the-shelf model' on the code bucket. If this claim is based on the full set of evaluated models, the table should either include all models or refer to a fuller table in an appendix; otherwise the wording is stronger than the presented evidence.
- [Appendix B] The leakage audit defines leakage as a near-duplicate synthetic query that also shares more than half of its gold set. This is a reasonable definition, but it is loose: a synthetic query could be semantically near a test query while not meeting the cosine/sequence thresholds. The paper should state whether the reported thresholds were chosen before or after inspecting the data, and should maybe report sensitivity to the thresholds.
- [Appendix A] The implementation details say hard negatives are mined with 'voyage-multilingual-2' but no reference or version is given. Also, the 'skip the top-1 neighbor' choice is mentioned but not ablated; a one-sentence justification or ablation would help.
- [General] Minor typographical and presentation issues: 'BEA VER' spacing in tables; 'Lg.-T/Lg.-C' abbreviations are not expanded in the captions; the phrase 'more than an order of magnitude larger' is correct for 305M vs. 4–8B but could be stated as '13–26×' for precision.
Circularity Check
No circular derivation; the 'label-free' framing is internally inconsistent with Appendix A, but this is a correctness/leakage concern, not circularity.
full rationale
The paper's central empirical claim—corpus-adaptive fine-tuning improves schema retrieval—is not circular: evaluation is against external, held-out queries from Spider, BIRD, BEAVER, and LiveSQLBench; baseline comparisons are off-the-shelf embedders; and the proposed training signal is generated from schema documents with gold sets fixed by construction, independent of the benchmark labels. Leave-one-corpus-out and the query-overlap audit are genuine external checks. The main caveat is an internal inconsistency rather than circularity: the abstract and Discussion call the recipe 'label-free' ('lightweight, label-free corpus adaptation'), but Appendix A states the authors 'mix the in-domain synthetic data with schema-linking pairs derived from the public text-to-SQL training splits of Spider and BIRD.' Since Spider and BIRD are also two of the five evaluation sources, gains on those benchmarks may be partly attributable to supervised fine-tuning, and no ablation isolates the synthetic-query component. The Limitations' admission that gold relevance 'can miss columns that are used implicitly' similarly affects label completeness, not the derivation. No equation, fitted parameter, or self-citation chain reduces the reported predictions to their inputs by construction.
Assumptions & free parameters
free parameters (6)
- Learning rate (Arctic-Embed-M) =
3e-5
- Learning rate (Qwen3-Embedding-8B) =
5e-6
- Number of hard negatives per query =
6
- Batch size / epochs =
32 / 1
- Hard-negative mining top-k skip =
skip top-1
- Leakage audit thresholds =
cos>=0.95, seq ratio>=0.9, gold overlap>0.5
assumptions (5)
- domain assumption The schema corpus is available at indexing/adaptation time
- domain assumption Relevant set R(q) equals the set of table/column references parsed from the gold SQL
- domain assumption LLM-synthesized queries are representative of real human questions for the purposes of adaptation
- domain assumption Hard negatives mined by cosine similarity in the embedding space are useful contrastive examples
- domain assumption Spider/BIRD training-split schema-linking pairs do not overlap the evaluation queries/labels used in the benchmark
Cite this review
Pith. "Pith review of Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval." pith.science (2026). https://pith.science/paper/5KO73DQ5
@misc{pith2026260713311,
author = {Pith},
title = {Pith review of: Finding the Right Tables and Columns: A Benchmark and Corpus-Adaptive Embeddings for SQL Schema Retrieval},
year = {2026},
howpublished = {\url{https://pith.science/paper/5KO73DQ5}},
note = {Machine review of arXiv:2607.13311}
}
read the original abstract
Retrieval in the SQL setting has largely been studied as the task of finding, within a large collection of SQL statements, the statement that answers a natural-language question. At scale, however, a more fundamental retrieval problem precedes generation: schema retrieval, identifying the tables and columns a question requires in a database that may contain thousands of them, far more than fit in a model's context. We argue that this step warrants first-class evaluation. To this end, we recast five text-to-SQL datasets (Spider, BIRD, BEAVER, and two LiveSQLBench variants) as retrieval tasks at both table and column granularity, covering realistic and enterprise-scale schemas under two document representations, and we show that off-the-shelf text and code embedders transfer poorly to this setting. We then propose corpus-adaptive fine-tuning: natural-language queries are synthesized directly from the target schema corpus, granularity-aware hard negatives are mined, and a 305M-parameter embedder is fine-tuned contrastively. This procedure raises average recall@10 from 60.4 to 75.6 (nDCG@10 from 51.9 to 68.0), making the 305M model the strongest retriever under one billion parameters and competitive with state-of-the-art embedders of 4-8B parameters, more than an order of magnitude larger. The same recipe improves an 8B state-of-the-art embedder from 77.8 to 78.4 recall@10, matching the best result on the benchmark and indicating that the adaptation is backbone-agnostic. Leave-one-corpus-out experiments and a leakage audit show that these gains reflect a transferable schema-retrieval ability rather than memorization of the evaluation data. Our results establish schema linking as a standalone retrieval task and lightweight, label-free corpus adaptation as a practical route to deploying it at enterprise scale.
Figures
Reference graph
Works this paper leans on
-
[9]
Sentence- BERT: Sentence embeddings using Siamese BERT- networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natu- ral Language Processing (EMNLP-IJCNLP), pages 3982–3992, Hong Kong, China. Association for Com- putational Linguistics. Tarun Suresh, Revanth Gangi Re...
2019
-
[10]
https://github.com/bird- bench/livesqlbench
Livesqlbench: A dynamic and contamination-free benchmark for evaluating llms on real-world text-to-sql tasks. https://github.com/bird- bench/livesqlbench. Accessed: 2025-05-22. Nandan Thakur, Nils Reimers, Andreas Rücklé, Ab- hishek Srivastava, and Iryna Gurevych
2025
-
[11]
GPL: Generative pseudo labeling for unsupervised domain adaptation of dense retrieval. InProceedings of the 2022 Conference of the North American Chapter of the Association for Computa- tional Linguistics: Human Language Technologies, pages 2345–2360, Seattle, United States. Association for Computational Linguistics. 7 Liang Wang, Nan Yang, Xiaolong Huang...
2022
-
[12]
Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muen- nighoff, Defu Lian, and Jian-Yun Nie
Text embeddings by weakly-supervised contrastive pre-training.Preprint, arXiv:2212.03533. Shitao Xiao, Zheng Liu, Peitian Zhang, Niklas Muen- nighoff, Defu Lian, and Jian-Yun Nie
-
[13]
Approximate nearest neighbor neg- ative contrastive learning for dense text retrieval. Preprint, arXiv:2007.00808. Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingn- ing Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev
arXiv 2007
-
[15]
Qwen3 embedding: Advancing text embedding and reranking through foundation models. Preprint, arXiv:2506.05176. Victor Zhong, Caiming Xiong, and Richard Socher
-
[2017]
Seq2sql: Generating structured queries from natural language using reinforcement learning. Preprint, arXiv:1709.00103. A Implementation Details Synthetic queries are generated with a proprietary instruction-tuned LLM served via Snowflake Cor- tex; the recipe does not depend on a specific genera- tor, and we leave a systematic comparison of query- synthesi...
arXiv 2018
-
[2018]
InProceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3911–3921, Brussels, Bel- gium
Spider: A large-scale human-labeled dataset for complex and cross-domain semantic pars- ing and text-to-SQL task. InProceedings of the 2018 Conference on Empirical Methods in Natural Lan- guage Processing, pages 3911–3921, Brussels, Bel- gium. Association for Computational Linguistics. Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong ...
2018
Show all 16 references
-
[2019]
Mohammadreza Pourreza and Davood Rafiei
Document expansion by query prediction.Preprint, arXiv:1904.08375. Mohammadreza Pourreza and Davood Rafiei
1904 arXiv
-
[2020]
InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online
Dense passage retrieval for open- domain question answering. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769–6781, Online. Association for Computational Linguistics. Omar Khattab and Matei Zaharia
2020
-
[2021]
InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5835–5847, On- line
RocketQA: An optimized train- ing approach to dense passage retrieval for open- domain question answering. InProceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Hu- man Language Technologies, pages 5835–5847, On- l...
2021
-
[2022]
Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou
Promptagator: Few-shot dense retrieval from 8 examples.Preprint, arXiv:2209.11755. Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou
-
[2023]
InProceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014–2037, Dubrovnik, Croatia
MTEB: Massive text embedding benchmark. InProceedings of the 17th Conference of the European Chapter of the Association for Com- putational Linguistics, pages 2014–2037, Dubrovnik, Croatia. Association for Computational Linguistics. Rodrigo Nogueira, Wei Yang, Jimmy Lin, and K...
2014
-
[2024]
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers
Arctic-embed: Scalable, efficient, and accurate text embedding models.Preprint, arXiv:2405.05374. Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers
-
[2025]
Preprint, arXiv:2411.12644
Codexembed: A generalist embedding model fam- ily for multiligual and multi-task code retrieval. Preprint, arXiv:2411.12644. Luke Merrick, Danmei Xu, Gaurav Nuti, and Daniel Campos
-
[2026]
Zhuyun Dai, Vincent Y
Beaver: An enterprise benchmark for text-to-sql.Preprint, arXiv:2409.02038. Zhuyun Dai, Vincent Y . Zhao, Ji Ma, Yi Luan, Jianmo Ni, Jing Lu, Anton Bakalov, Kelvin Guu, Keith B. Hall, and Ming-Wei Chang
Reviewed August 2, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.