REVIEW 3 major objections 5 minor 30 references
CORE-T: one LLM pass plus a join-compatibility cache retrieves coherent table sets for text-to-SQL.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-03 09:37 UTC pith:KCEXT7O4
load-bearing objection Solid, clearly-specified open-book table retriever; benchmark gains likely hold, but the compatibility cache's coverage of realistic join paths is unquantified and the abstract overclaims. the 3 major comments →
CORE-T: COherent REtrieval of Tables for Text-to-SQL
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On the paper's own terms, the contribution is a specific decomposition: move schema understanding offline by writing a short purpose description for every table and caching a compatibility score for each table pair; keep online inference to a single LLM call over only the top-K retrieved candidates, using the cached compatibility evidence as guidance; then restore any strongly compatible tables the LLM pruned. The compatibility cache is the load-bearing new object: it scores pairs only when at least one column is unique and the other column's values are a subset of it, combining header similarity and value overlap, and it records the best join columns. The reported experiments show that this
What carries the argument
The central mechanism is the offline table-compatibility cache, a pairwise score CS(t_i, t_j) computed from column headers (exact and semantic) and value overlap, gated by a hard key–foreign-key-like constraint: at least one column must be unique and the other must be a subset. The table–table score is the best valid column match, and the argmax join columns are cached. This cache supplies the join evidence that a single LLM selection call consumes, and it drives the additive restoration step that re-adds strongly compatible tables. Around it, the pipeline also relies on LLM-generated purpose metadata to disambiguate near-duplicate tables and a dense-retrieval top-K stage to guarantee recall
Load-bearing premise
The claim collapses if the compatibility cache cannot recognize the join paths a query needs: it only scores pairs where one column is unique and the other's values are a subset, and it estimates value overlap from five randomly sampled rows, so non-key joins, self-joins, many-to-many bridge joins, and multi-hop joins can receive a zero score and the pipeline can silently drop a required table.
What would settle it
Take the BIRD, Spider, or MMQA query set and partition the multi-table queries by the type of join required (key–foreign-key vs. non-key/bridge/multi-hop). If CORE-T's perfect recall on the non-key partition is no better than plain dense retrieval's, the compatibility cache — not the LLM selection — is the binding constraint. A cheaper probe: remove the hard uniqueness/subset constraint and rerun the joinability-accuracy comparison; the reported gap would be the measure of how much the constraint buys.
If this is right
- If the central claim holds, open-book retrieval can be made join-aware with a single LLM call, removing the need for database identifiers or gold foreign keys.
- Multi-table execution accuracy should improve most where queries require two or more tables, because coherent schema slices reduce the chance of missing a required join edge.
- Token cost should stay roughly constant relative to the top-K candidate size, not the corpus size, making the method applicable to large pooled corpora.
- The compatibility cache yields a free side product: inferred join columns for any table pair, reusable outside text-to-SQL in data-lake analytics.
- The error analysis suggests that gains come primarily from removing distractor tables while holding recall, so the method's value should grow as corpora become more heterogeneous.
Where Pith is reading between the lines
- The paper does not test it, but the same offline-cache plus one-shot selection scheme should transfer to retrieving other schema-agnostic artifacts (views, JSON collections, document stores) whenever a compatibility signal can be precomputed.
- If the method is correct, the practical bottleneck shifts from retrieval architecture to the quality of the compatibility cache; a testable prediction is that retrieval F1 will degrade on query types whose join paths are non-key, self-joins, bridge tables, or multi-hop, exactly the cases the paper flags as limitations.
- The 5-row sampling for value overlap is a tunable knob: with minimal added cost, sampling more rows or using schema-only signals could reduce the chance of missing shared keys, and an ablation varying sample size would reveal the sensitivity.
- The reported gains concentrate in removing distractors while holding recall, so an enterprise data lake with cross-domain duplicates is the natural stress test for whether CORE-T's advantage survives in real deployments.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper studies open-book text-to-SQL over pooled multi-source table collections without db_ids or gold foreign keys. It proposes CORE-T, a training-free pipeline that (i) offline enriches each table with LLM-generated purpose metadata and a dense index, (ii) precomputes a table–table compatibility cache using column uniqueness/subset constraints and value/header similarity, and (iii) online retrieves top-K tables by dense retrieval, performs a single LLM selection pass conditioned on relevance and cached compatibility edges, and then additively restores strongly compatible tables pruned by the LLM. The authors report table-selection F1 gains over DR/DRR/ReAct/JAR/ARM on pooled BIRD, SPIDER, and MMQA, multi-table execution-accuracy improvements across three SQL generators, token-efficiency gains, ablations, oracle bounds, and an error analysis.
Significance. If the experimental results are accepted, the paper makes a useful practical contribution: it shows that a lightweight, training-free, cache-based compatibility signal combined with a single LLM selection call can substantially improve the precision–recall balance of multi-table retrieval and translate into downstream execution gains, while reducing LLM token usage compared to agentic and MIP-based baselines. The work is clearly presented, ships prompts and implementation details, provides ablations against DR@10, includes oracle upper bounds, and makes code/data available. The main caveat is that the core compatibility mechanism is restricted to key–foreign-key-like equality joins; the paper's own Limitations section acknowledges this, and the broader claim of applicability to realistic enterprise open-book corpora needs additional evidence.
major comments (3)
- [Eq. (2) and §3.1/§3.2.3] The compatibility cache is load-bearing: it is the only join evidence given to the LLM selector and the exclusive trigger for the additive restoration step. Eq. (2) requires at least one unique column and a subset relation between columns, so non-equi joins, self-joins, many-to-many bridge joins, and multi-hop joins all receive CS=0. The Limitations section acknowledges this ('may miss other connections common in practice...'), but the paper does not quantify how often such join patterns occur in the gold multi-table queries of BIRD/SPIDER/MMQA, nor how often a necessary table is missed because its only edge to the selected set fails the validity constraint. Without this analysis, the strong F1 and EM gains may be an artifact of benchmark join simplicity, and the paper's claim of improvement for realistic open-book/enterprise corpora is not established. I request an empirical breakdown o
- [§4.1.4, §5, Appendix F.2] The pipeline has several hand-set free parameters — K=10, τ_comp=0.3, the 5-row sample size, the column-score weights in Eq. (3), and the 0.5 joinability threshold — and the text states only that they are 'kept fixed across all datasets,' not how they were chosen. If these values were selected after inspecting the same dev sets on which all results are reported, the headline numbers are selected-on-test. The paper should either specify a validation protocol (e.g., tune on one dataset and evaluate on the others, or report performance across a plausible range of τ_comp and K) or justify the hand-set choices by sensitivity analysis. This is important because the central claims are empirical and several compared baselines are evaluated with their own tuned hyperparameters.
- [Abstract] The first abstract states the method is evaluated 'Across Bird, Spider, MMQA, and Beaver' and claims 'up to 24.4 points in multi-table execution accuracy,' while Section 1, Section 4, and Tables 2–3 report only BIRD, SPIDER, and MMQA, with multi-table execution gains of up to 5.0 points on BIRD and 6.9 points on MMQA. The body also mentions BEAVER only as a future benchmark in Appendix D. The abstract either must be corrected to match the experiments or the BEAVER experiments must be included. This is a factual reporting inconsistency that affects reader trust.
minor comments (5)
- [§4.2 / Table 4] The text says CORE-T 'reduces input tokens by 4.0–5.0× and total selection tokens by 3.6–4.2×,' but the abstract claims '1.64–4.20× fewer total selection tokens.' Please unify the numbers and clarify whether the abstract refers to input, output, or total tokens.
- [§3.1 / Appendix A] Five rows are 'sampled uniformly at random without replacement,' but no seed or sampling protocol is stated for this step, while all LLM decoding uses seed 42. Please report the sampling seed/procedure for reproducibility; if the code already fixes it, say so explicitly.
- [Appendix F.2] Minor typographical/capitalization issues: 'corerationale' in §4.1.2, 'AS these were not available' in Appendix F.2, and inconsistent 'Bird' vs 'BIRD' in captions and text.
- [§4.2 / Table 8] The paper says text-embedding-3-large is strongest but chooses UAE-Large-V1 as default for open-source reproducibility; the rationale is understandable, but the sentence could be clearer about the trade-off and about whether any downstream comparisons change if text-embedding-3-large is used.
- [§4.2 / Table 3] The MMQA EM=1T column contains only 0.4% of queries, so the Oracle numbers are highly unstable; this is a known small-sample caveat and should be acknowledged in the text, not only in the table.
Circularity Check
No circularity: CORE-T's claims rest on external benchmark comparisons; the compatibility cache is a hand-defined heuristic, not a fitted target.
full rationale
The paper's derivation chain is empirical rather than definitional. The offline compatibility cache (Eqs. 2-4) is a hand-defined heuristic used as evidence in the single LLM selection call and as the trigger for the additive restoration step; however, the headline table-selection F1 and SQL execution EM (Tables 2-3) are measured against gold table sets and gold SQL execution, not against CS values. There is no equation in which the reported prediction is, by construction, equal to a fitted parameter or to the cache score. The only fixed constants are K=10, tau_comp=0.3, and the embedding choice, and they are fixed across datasets rather than fitted to test labels (Section F.2). The compatibility score is itself evaluated against gold joinability annotations in a separate diagnostic (Table 5), and the paper explicitly acknowledges its restrictive key-FK assumption: 'Our compatibility cache focuses on key-foreign-key-like joins... It may miss other connections common in practice, including non-equi joins, self-joins, many-to-many joins via bridge tables, and multi-hop joins that require intermediate tables.' That is a scope limitation, not a circular reduction. Self-citations (JAR, ARM, EnrichIndex) are used as baselines or prior building blocks; JAR/ARM are compared against rather than invoked as evidence for CORE-T's superiority, so no load-bearing self-citation chain exists. Thus the central claim is self-contained given its stated assumptions.
Axiom & Free-Parameter Ledger
free parameters (6)
- K (dense retrieval cutoff) =
10
- tau_comp (restoration threshold) =
0.3
- Markdown row sample size =
5 rows
- Column-pair score weights =
0.5 * Jaccard + 0.5 * name similarity
- Joinable prediction threshold =
CS > 0.5
- Embedding model =
UAE-Large-V1
axioms (4)
- domain assumption A 5-row uniform random sample is representative enough for Jaccard and subset signals.
- domain assumption Key-foreign-key-like joins (one unique column, one subset column) are the join structure needed to answer open-book text-to-SQL queries.
- domain assumption Dropping db_id and pooling closed-book benchmark tables reproduces the open-book enterprise setting.
- domain assumption The instruction-tuned LLM follows the recall-oriented group-selection policy and emits parseable JSON.
read the original abstract
Realistic text-to-SQL workflows often require joining multiple tables. As a result, accurately retrieving the relevant set of tables becomes a key bottleneck for end-to-end performance. We study an open-book setting where queries must be answered over large, heterogeneous table collections pooled from many sources, without clean scoping signals such as database identifiers. Here, dense retrieval (DR) achieves high recall but returns many distractors, while join-aware alternatives often rely on extra assumptions and/or incur high inference overhead. We propose CORE-T, a scalable, training-free framework that enriches tables with LLM-generated purpose metadata and pre-computes a lightweight table-compatibility cache. At inference time, DR returns top-K candidates; a single LLM call selects a coherent, joinable subset, and a two-step additive adjustment stage restores strongly compatible tables. Across Bird, Spider, MMQA, and Beaver, CORE-T improves over DR by up to 22.7 points in table-selection F1 while returning up to 40% fewer tables, and by up to 24.4 points in multi-table execution accuracy, and uses 1.64-4.20x fewer total selection tokens than LLM-intensive baselines.
Figures
Reference graph
Works this paper leans on
-
[1]
Peter Baile Chen, Fabian Wenz, Yi Zhang, Devin Yang, Justin Choi, Nesime Tatbul, Michael Cafarella, Çağatay Demiralp, and Michael Stonebraker. 2025 a . https://arxiv.org/abs/2409.02038 Beaver: An enterprise benchmark for text-to-sql . Preprint, arXiv:2409.02038
Pith/arXiv arXiv 2025
-
[2]
Peter Baile Chen, Tomer Wolfson, Michael Cafarella, and Dan Roth. 2025 b . https://arxiv.org/abs/2504.03598 EnrichIndex : Using LLM s to enrich retrieval indices offline . In Proceedings of the Second Conference on Language Modeling ( COLM ) . COLM 2025
Pith/arXiv arXiv 2025
-
[3]
Peter Baile Chen, Yi Zhang, Mike Cafarella, and Dan Roth. 2025 c . https://doi.org/10.18653/v1/2025.acl-long.1463 Can we retrieve everything all at once? ARM : An alignment-oriented LLM -based retrieval method . In Proceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 30298--30317, Vienna, A...
-
[4]
Peter Baile Chen, Yi Zhang, and Dan Roth. 2024. https://doi.org/10.18653/v1/2024.acl-long.148 Is table retrieval a solved problem? exploring join-aware multi-table retrieval . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2687--2699, Bangkok, Thailand. Association for Computationa...
-
[5]
Zhiyu Chen, Mohamed Trabelsi, Jeff Heflin, Yinan Xu, and Brian D. Davison. 2020. https://doi.org/10.1145/3397271.3401044 Table search using a deep contextualized language model . In Proceedings of the 43rd International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 589--598, Virtual Event, China. Association for Computi...
arXiv 2020
-
[6]
Tianji Cong, James Gale, Jason Frantz, H. V. Jagadish, and C a g atay Demiralp. 2023. https://www.cidrdb.org/cidr2023/papers/p75-cong.pdf WarpGate : A semantic join discovery system for cloud data warehouses . In Proceedings of the 13th Conference on Innovative Data Systems Research ( CIDR 2023) , Amsterdam, The Netherlands
2023
-
[7]
Jonathan Herzig, Thomas M \"u ller, Syrine Krichene, and Julian Eisenschlos. 2021. https://doi.org/10.18653/v1/2021.naacl-main.43 Open domain question answering over tables via dense retrieval . In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, pages 512--519,...
-
[8]
Vladimir Karpukhin, Barlas Oguz, Sewon Min, Patrick Lewis, Ledell Wu, Sergey Edunov, Danqi Chen, and Wen - tau Yih. 2020. https://doi.org/10.18653/v1/2020.emnlp-main.550 Dense passage retrieval for open-domain question answering . In Proceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 6769--6781, Online. A...
-
[9]
Lai, Yeye He, and Surajit Chaudhuri
Eugenie Y. Lai, Yeye He, and Surajit Chaudhuri. 2025. https://doi.org/10.14778/3734839.3734856 Auto-prep: Holistic prediction of data preparation steps for self-service business intelligence . Proc. VLDB Endow. , 18(7):2212--2225
arXiv 2025
-
[10]
Chia-Hsuan Lee, Oleksandr Polozov, and Matthew Richardson. 2021. https://doi.org/10.18653/v1/2021.acl-long.176 K aggle DBQA : Realistic evaluation of text-to- SQL parsers . In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 2261--2273, Online. Association for Computational Linguistics
-
[11]
Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/46c10f6c8ea5aa6f267bcdabcb123f97-Abstract-Conference.html Spider 2.0: Evaluating language...
2025
-
[12]
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin Chen-Chuan Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. https://papers.nips.cc/paper_files/paper/2023/hash/83fc8fab1710363050bbd1d4b8cc0021-Abstract-Datasets_and_Benchmarks.html Can LLM alrea...
2023
-
[13]
Xianming Li and Jing Li. 2024. https://doi.org/10.18653/v1/2024.acl-long.101 A o E : Angle-optimized embeddings for semantic textual similarity . In Proceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1825--1839, Bangkok, Thailand. Association for Computational Linguistics
-
[14]
Llama Team . 2024. https://doi.org/10.48550/arXiv.2407.21783 The L lama 3 herd of models . Preprint, arXiv:2407.21783. Author list credited to Llama Team; detailed contributors listed in the paper appendix
-
[15]
Meta AI . 2024. https://ai.meta.com/blog/meta-llama-3-1/ Introducing L lama 3.1: Our most capable models to date
2024
-
[16]
Niklas Muennighoff, Nouamane Tazi, Loic Magne, and Nils Reimers. 2023. https://doi.org/10.18653/v1/2023.eacl-main.148 MTEB : Massive text embedding benchmark . In Proceedings of the 17th Conference of the European Chapter of the Association for Computational Linguistics, pages 2014--2037, Dubrovnik, Croatia. Association for Computational Linguistics
-
[17]
Fatemeh Nargesian, Erkang Zhu, Ren \'e e J. Miller, Ken Q. Pu, and Patricia C. Arocena. 2019. https://doi.org/10.14778/3352063.3352116 Data lake management: Challenges and opportunities . Proc. VLDB Endow. , 12(12):1986--1989
arXiv 2019
-
[18]
Qwen Team . 2025. https://doi.org/10.48550/arXiv.2412.15115 Qwen2.5 technical report . Preprint, arXiv:2412.15115
-
[19]
Zhiruo Wang, Zhengbao Jiang, Eric Nyberg, and Graham Neubig. 2022. https://aclanthology.org/2022.suki-1.5/ Table retrieval may not necessitate table-specific model design . In Proceedings of the Workshop on Structured and Unstructured Knowledge Integration (SUKI), pages 36--46, Seattle, USA. Association for Computational Linguistics
2022
-
[20]
Thomas Wolf, Lysandre Debut, Victor Sanh, Julien Chaumond, Cl \'e ment Delangue, Anthony Moi, Pierric Cistac, Tim Rault, R \'e mi Louf, Morgan Funtowicz, and Jamie Brew. 2020. https://doi.org/10.18653/v1/2020.emnlp-demos.6 Transformers: State-of-the-art natural language processing . In Proceedings of the 2020 Conference on Empirical Methods in Natural Lan...
-
[21]
Jian Wu, Linyi Yang, Dongyuan Li, Yuliang Ji, Manabu Okumura, and Yue Zhang. 2025. https://proceedings.iclr.cc/paper_files/paper/2025/hash/794a425a2e47e05d29d30f79b79a692d-Abstract-Conference.html MMQA : Evaluating LLM s with multi-table multi-hop complex questions . In Proceedings of the Thirteenth International Conference on Learning Representations
2025
-
[22]
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. 2023. https://iclr.cc/virtual/2023/oral/12647 ReAct : Synergizing reasoning and acting in language models . In International Conference on Learning Representations
2023
-
[23]
Puxuan Yu, Luke Merrick, Gaurav Nuti, and Daniel Campos. 2024. https://arxiv.org/abs/2412.04506 Arctic-embed 2.0: Multilingual retrieval without compromise . Preprint, arXiv:2412.04506
Pith/arXiv arXiv 2024
-
[24]
Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. https://doi.org/10.18653/v1/D18-1425 S pider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to- SQL task . In Proceedings of the 2018 Conference on Emp...
-
[25]
Yanzhao Zhang, Mingxin Li, Dingkun Long, Xin Zhang, Huan Lin, Baosong Yang, Pengjun Xie, An Yang, Dayiheng Liu, Junyang Lin, Fei Huang, and Jingren Zhou. 2025. https://doi.org/10.48550/arXiv.2506.05176 Qwen 3 embedding: Advancing text embedding and reranking through foundation models . arXiv preprint arXiv:2506.05176
-
[26]
Yi Zhang and Zachary G. Ives. 2020. https://doi.org/10.1145/3318464.3389726 Finding related tables in data lakes for interactive data science . In Proceedings of the 2020 ACM SIGMOD International Conference on Management of Data , pages 1951--1966, Portland, OR, USA. Association for Computing Machinery
arXiv 2020
-
[27]
Victor Zhong, Caiming Xiong, and Richard Socher. 2017. https://arxiv.org/abs/1709.00103 Seq2SQL : Generating structured queries from natural language using reinforcement learning . Preprint, arXiv:1709.00103
Pith/arXiv arXiv 2017
-
[28]
Erkang Zhu, Dong Deng, Fatemeh Nargesian, and Ren \'e e J. Miller. 2019. https://doi.org/10.1145/3299869.3300065 JOSIE : Overlap set similarity search for finding joinable tables in data lakes . In Proceedings of the 2019 International Conference on Management of Data, pages 847--864, Amsterdam, The Netherlands. Association for Computing Machinery
arXiv 2019
-
[29]
online" 'onlinestring :=
ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRING...
-
[30]
write newline
" write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.