REVIEW 4 major objections 6 minor 1 cited by
NLCTables: A Dataset for Marrying Natural Language Conditions with Table Discovery
T0 review · 4 major / 6 minor · reviewed 2026-08-16 · deepseek-v4-flash
Pith's one-line read The paper defines NL-conditional table discovery, builds a 22,080-table benchmark, and shows existing methods underperform on the combined table-plus-language input.
desk verdict A genuinely useful new benchmark for NL-conditional table discovery, with a real caveat about auto-generated ground truth; worth a serious referee. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a table-splitting construction framework. Large tables are filtered and manually annotated for column types; union queries are built by horizontal splitting on a column value (rows satisfying the condition form positives, rows failing form negatives), while join queries use vertical splitting with a shared join key, and mixed-mode queries layer two conditions sequentially. Each split is paired with a template-generated NL condition, and all positive tables are positive by construction because they come from the same original table as the query table and meet the condition used to create the query; subtable sampling fixes thresholds (80% of positive rows satisfy the condition, 50% of negative rows do not) and LLM rewriting adds linguistic diversity, while a minority of labels come from manual annotation of Spider SQL queries.
What would settle it
Run a human annotation study on a stratified random sample of at least 100 nlcTables queries across all condition types, and measure agreement between the auto-generated labels and independent human relevance judgments; if more than ten percent of auto-labeled positive tables are judged by annotators as not satisfying the stated natural language condition, or if negatives are judged as satisfying it, the dataset's validity as ground truth would be undermined.
Extended reading notes
Core claim
The paper's central claim is that table discovery should be a ternary relevance problem: a candidate table is relevant only if it is related to the query table and simultaneously satisfies the user's natural language condition. On nlcTables, the paper shows that existing keyword-based methods (GTR, StruBERT), table union methods (Santos, Starmie), and table join methods (Josie, DeepJoin) all fail to handle this combination, with NDCG@10 values dropping to roughly 0.3 to 0.55 on harder condition categories and degrading further as negative examples increase or fuzzy transformations are added. The paper concludes that the combined input modality is qualitatively harder than either keyword search or query-table search alone.
Load-bearing premise
The benchmark's ground truth is only as sound as the assumption that a table produced by the automated splitting pipeline, together with its template-filled natural language condition, reflects what a real user would judge relevant for that query.
Editorial extensions
If this is right
- Existing table discovery methods cannot be directly ported to nlcTD; the paper's baseline results establish a concrete accuracy gap that new models must close.
- Future nlcTD methods should condition retrieval on both the query table and the NL condition, since ignoring either side yields the low recall and precision the paper documents.
- Topic-level conditions are comparatively easy, while numerical, date, and mixed-mode conditions are the hardest, so benchmarks should weight toward these harder types.
- Dataset composition matters: increasing the positive-to-negative ratio or scaling the repository changes measured accuracy and efficiency, so reports on nlcTables should state which configuration was used.
- The released scripts let researchers generate custom nlcTD datasets with adjustable hyperparameters, enabling controlled stress tests of new methods.
Reading between the lines
- The auto-labeling pipeline assumes that a subtable satisfying a template condition is what a human would call relevant; if that assumption fails for a nontrivial fraction of queries, the reported performance gaps could shrink or grow, and a human relevance study on a sample of nlcTables queries would settle this.
- Because query tables are always subtables of a larger original table, the benchmark may not cover scenarios where the query table has independent provenance, so constructing queries from externally sourced tables would test generalization.
- The template conditions often reference a specific column and value, whereas real user language is messier; an extension that replaces templates with free-form LLM-generated conditions, validated by human judges, would increase realism.
- If a simple condition-aware reranker (for example, an LLM that checks whether candidate tables satisfy the condition) substantially improves over the reported baselines, that would identify the missing ingredient precisely.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes NL-conditional table discovery (nlcTD), a task in which a user supplies both a query table and a natural-language condition, and defines a 16-category taxonomy covering NL-only, union, join, and fuzzy conditions. It introduces nlcTables, a benchmark constructed by splitting 193 original tables from OpenData, WebTable, and GitTable, using templates and LLM-based semantic augmentation to generate NL conditions, plus a smaller set of SQL-derived labels from Spider. The paper reports 627 queries in the abstract and conclusion but 647 in Table 2, with 22,080 candidate tables and 21,200 relevance annotations, and evaluates six existing table-discovery methods, finding substantial performance gaps. The dataset, construction framework, and baseline implementations are publicly released.
Significance. If the dataset and evaluation are valid, nlcTables fills a real gap: existing table-discovery benchmarks cover only keyword or query-table inputs, while the nlcTD setting matches the interactive data-assistant paradigm. The paper has several strengths: a detailed automated construction pipeline with configurable hyperparameters, a 16-category taxonomy, public code and baselines, and multi-dimensional evaluation across condition categories, fuzzy variants, positive-to-negative ratios, and dataset scale. However, the validity of the benchmark depends on the auto-labeled ground truth and on the way baselines were adapted to a new input modality; both need additional support before the reported performance gaps can be attributed to the nlcTD task rather than to construction artifacts.
major comments (4)
- [Section 4.2 (Query Construction) and (Labeling Queries)] The ground-truth generation is circular in a way that threatens benchmark validity. Positive labels are produced by the same splitting procedure that creates the queries: horizontal splitting for union queries applies the row-level condition derived from the same column and value used in the NL template, and join queries split one original table so the new column referenced in the template is present in the positive ground truth. Negative tables are sampled so that only 50% of their rows violate the condition, meaning the other 50% still satisfy it, yet these tables are binarized as irrelevant. No external human relevance judgments are reported for the auto-labeled majority; the only human input is column-type annotation, SQL-based labeling for Spider-derived queries, and manual review of the LLM semantic augmentation. The large performance gaps in Section 5.2 (Figures 5 and 6) could therefore reflect the 80%/50% sampling thresholds and shared provenance rather than the intrinsic difficulty of nlcTD. Please provide a human-annotated subset with inter-annotator agreement, report how the baseline gaps change when the 80% and 50% thresholds are varied, and state explicitly what fraction of ground truths come from each labeling source.
- [Section 5.1 and 5.2] The paper never specifies how each baseline receives the two-part input (query table, NL condition). Gtr and StruBert accept text queries but are applied to nlcTables-U and nlcTables-J; it is unclear whether the query table is serialized into the text and how the NL condition is attached. Josie and DeepJoin take a single query column; the paper does not state which column is used or how the NL condition filters candidates. Santos and Starmie take a query table; the paper does not say how the NL condition is incorporated. Without a precise statement of these adapters, the conclusion that existing methods are not well-suited is not reproducible and cannot be separated from arbitrary choices of input serialization. Please document the exact input construction for each baseline, or point to versioned code and configuration files, in the paper.
- [Section 4.3, Table 2, Table 5, abstract, and conclusion] Dataset statistics are internally inconsistent. The abstract and conclusion state 627 queries, while Table 2's nlcTables-Full row and the sum of the five type rows give 647 queries. Furthermore, Table 5 reports nlcTables-J Large (100%, |T|=9,347), which is nearly twice the nlcTables-J total repository of 4,871 in Table 2. These discrepancies affect the headline dataset scale and must be reconciled before the benchmark statistics can be trusted.
- [Section 4.3 and Section 5.3] The paper states that refined classification proportions are omitted due to page limit, yet RQ2 in Section 5.3 reports performance per condition category. Without counts for the 16 taxonomy subcategories, a reader cannot determine whether the comparisons in Table 4 are based on enough queries or whether the dataset's difficulty is an artifact of category imbalance. Please provide the distribution of queries across the 16 subcategories and across column types, either in the paper or in an appendix.
minor comments (6)
- [Section 3.1] The phrase 'they typically accept sorely keywords' should read 'they typically accept solely keywords'.
- [Figures 2 and 3] The text and labels in Figures 2 and 3 are very small in the PDF; please enlarge them or provide a vector version for readability.
- [Section 4.2] Hyperparameters such as L_scale, L_col, dup_rate, and temp_num are named but not defined numerically or listed with defaults; a table of default values would improve reproducibility.
- [Section 5.1] No hyperparameters or training details are given for the six baselines; please state whether official implementations and default settings were used.
- [Figures 6 and 7] The claims about performance drops under fuzzy queries and different positive-to-negative ratios would be stronger with error bars or significance tests across query subsets.
- [Table 2] The row for nlcTables-K shows backslashes for query table rows and columns, which is correct because K queries have no query table, but the caption should explain this explicitly.
Circularity Check
No circularity: nlcTables operationalizes relevance via an explicit table-splitting pipeline; the reported baseline gaps are empirical measurements, not predictions derived from the construction.
full rationale
The paper's central claims are (a) nlcTD is a new task and (b) six existing table discovery methods perform poorly on nlcTables. Neither claim is derived from an input in a way that makes the conclusion equivalent to the premise. The dataset is built by an explicitly described splitting procedure: query tables and ground-truth tables are generated from the same original tables, with positive tables required to satisfy the NL condition and negative tables sampled so that 50% of rows do not. This is an operational definition of relevance, not a hidden reuse of fitted values. The baseline numbers in Tables 3-5 and Figures 5-7 are measured on that operationalized benchmark; no parameter is fitted to the evaluation subset and then renamed as a prediction. The auto-labeling loop (Section 4.2, step 3) is transparent and is the same technique used by prior datasets (Tus, Santos, LakeBench), which the paper explicitly cites. The concern that the 80%/50% thresholds and co-provenance make the task easier or harder is a validity/difficulty critique of the benchmark, not circularity. The only self-citation ([9], used for the claim that table augmentation improves AI training data) is background and not load-bearing. Therefore no circular step is identified.
Assumptions & free parameters
free parameters (3)
- Positive and negative condition satisfaction thresholds =
80% and 50%
- Table filtering thresholds =
at least 50 rows and 8 columns
- Dataset construction hyperparameters =
not specified
assumptions (4)
- domain assumption Tables produced by splitting a common source table are genuinely relevant to each other for union and join discovery.
- domain assumption LLM semantic augmentation preserves unionability and joinability semantics after manual review.
- ad hoc to paper The 16-category taxonomy covers the majority of real nlcTD scenarios.
- domain assumption Spider SQL queries provide a representative source of real table discovery use cases.
Cite this review
Pith. "Pith review of NLCTables: A Dataset for Marrying Natural Language Conditions with Table Discovery." pith.science (2026). https://pith.science/paper/FY6ORHDT
@misc{pith2026250415849,
author = {Pith},
title = {Pith review of: NLCTables: A Dataset for Marrying Natural Language Conditions with Table Discovery},
year = {2026},
howpublished = {\url{https://pith.science/paper/FY6ORHDT}},
note = {Machine review of arXiv:2504.15849}
}
read the original abstract
With the growing abundance of repositories containing tabular data, discovering relevant tables for in-depth analysis remains a challenging task. Existing table discovery methods primarily retrieve desired tables based on a query table or several vague keywords, leaving users to manually filter large result sets. To address this limitation, we propose a new task: NL-conditional table discovery (nlcTD), where users combine a query table with natural language (NL) requirements to refine search results. To advance research in this area, we present nlcTables, a comprehensive benchmark dataset comprising 627 diverse queries spanning NL-only, union, join, and fuzzy conditions, 22,080 candidate tables, and 21,200 relevance annotations. Our evaluation of six state-of-the-art table discovery methods on nlcTables reveals substantial performance gaps, highlighting the need for advanced techniques to tackle this challenging nlcTD scenario. The dataset, construction framework, and baseline implementations are publicly available at https://github.com/SuDIS-ZJU/nlcTables to foster future research.
Figures
Figures from the paper (4 more)
Forward citations
Cited by 1 Pith paper
-
Something's Fishy In The Data Lake: A Critical Re-evaluation of Table Union Search Benchmarks
Simple baselines rival or surpass specialized table union search models on current benchmarks, indicating these benchmarks reward surface overlap and general embeddings rather than isolating semantic understanding.
Reference graph
Works this paper leans on
-
[1]
[n. d.]. Copilot. https://github.com/features/copilot
-
[2]
[n. d.]. OpenData. https://open.canada.ca/
-
[3]
Alex Bogatu, Alvaro A. A. Fernandes, Norman W. Paton, and Nikolaos Konstanti- nou. 2020. Dataset Discovery in Data Lakes. In Proc. 36th IEEE ICDE . 709–720. doi:10.1109/ICDE48307.2020.00067
arXiv 2020
-
[4]
Michael J Cafarella, Alon Halevy, and Nodira Khoussainova. 2009. Data integra- tion for the relational web. Proc. VLDB Endow. 2, 1 (2009), 1090–1101
work page 2009
-
[5]
Chengliang Chai, Yuhao Deng, Yutong Zhan, Ziqi Cao, Yuanfang Zhang, Lei Cao, Yuping Wang, Zhiwei Zhang, Ye Yuan, Guoren Wang, and Nan Tang. 2024. LakeCompass: An End-to-End System for Data Maintenance, Search and Analysis in Data Lakes. Proc. VLDB Endow. 17, 12 (2024), 4381–4384. doi:10.14778/3685800. 3685880
doi:10.14778/3685800 2024
-
[6]
Si-An Chen, Lesly Miculicich, Julian Eisenschlos, Zifeng Wang, Zilong Wang, Yanfei Chen, YASUHISA FUJII, Hsuan-Tien Lin, Chen-Yu Lee, and Tomas Pfister
-
[7]
Zhiyu Chen, Mohamed Trabelsi, Jeff Heflin, Yinan Xu, and Brian D. Davison
-
[8]
Nadiia Chepurko, Ryan Marcus, Emanuel Zgraggen, Raul Castro Fernandez, Tim Kraska, and David Karger. 2020. ARDA: automatic relational data augmentation for machine learning. Proc. VLDB Endow. 13, 9 (2020), 1373–1387. doi:10.14778/ 3397230.3397235
arXiv 2020
Show all 47 references
- [9]
-
[10]
Xiang Deng, Huan Sun, Alyssa Lees, You Wu, and Cong Yu. 2020. TURL: table understanding through representation learning. Proc. VLDB Endow. 14, 3 (2020), 307–319. doi:10.14778/3430915.3430921
2020
-
[11]
Yuhao Deng, Chengliang Chai, Lei Cao, Qin Yuan, Siyuan Chen, Yanrui Yu, Zhaoze Sun, Junyi Wang, Jiajun Li, Ziqi Cao, et al. 2024. LakeBench: A Benchmark for Discovering Joinable and Unionable Tables in Data Lakes. Proc. VLDB Endow. 17, 8 (2024), 1925–1938
2024
-
[12]
Yuyang Dong and Masafumi Oyamada. 2022. Table Enrichment System for Machine Learning. In Proc. 45th SIGIR. ACM. 3267–3271. doi:10.1145/3477495. 3531678
2022 doi
-
[13]
Yuyang Dong, Kunihiro Takeoka, Chuan Xiao, and Masafumi Oyamada. 2021. Efficient Joinable Table Discovery in Data Lakes: A High-Dimensional Similarity- Based Approach. In Proc. 37th IEEE ICDE . 456–467. doi:10.1109/ICDE51399.2021. 00046
2021
-
[14]
Yuyang Dong, Chuan Xiao, Takuma Nozawa, Masafumi Enomoto, and Masafumi Oyamada. 2023. DeepJoin: Joinable Table Discovery with Pre-Trained Language Models. Proc. VLDB Endow. 16, 10 (2023), 2458–2470. doi:10.14778/3603581. 3603587
2023 doi
-
[15]
Grace Fan, Jin Wang, Yuliang Li, and Renée J. Miller. 2023. Table Discovery in Data Lakes: State-of-the-art and Future Directions. In Companion of SIGMOD. 69–75. doi:10.1145/3555041.3589409
2023
-
[16]
Grace Fan, Jin Wang, Yuliang Li, Dan Zhang, and Renée J. Miller. 2023. Semantics- aware Dataset Discovery from Data Lakes with Contextualized Column-based Representation Learning. Proc. VLDB Endow. 16, 7 (2023), 1726–1739. doi:10. 14778/3587136.3587146
2023
-
[17]
Xuming Hu, Shen Wang, Xiao Qin, Chuan Lei, Zhengyuan Shen, Christos Faloutsos, Asterios Katsifodimos, George Karypis, Lijie Wen, and Philip S. Yu
-
[18]
Madelon Hulsebos, Çagatay Demiralp, and Paul Groth. 2023. GitTables: A Large- Scale Corpus of Relational Tables. Proc. ACM Manag. Data 1, 1 (2023), 1–17. doi:10.1145/3588710
2023 doi
-
[19]
Miller, and Mirek Riedewald
Aamod Khatiwada, Grace Fan, Roee Shraga, Zixuan Chen, Wolfgang Gatter- bauer, Renée J. Miller, and Mirek Riedewald. 2023. SANTOS: Relationship- based Semantic Table Union Search. Proc. ACM Manag. Data 1, 1 (2023), 1–25. doi:10.1145/3588689
2023 doi
-
[20]
Aristotelis Leventidis, Martin Pekár Christensen, Matteo Lissandrini, Laura Di Rocco, Katja Hose, and Renée J. Miller. 2024. A Large Scale Test Corpus for Semantic Table Search. In Proc. 47th SIGIR. ACM. https://api.semanticscholar. org/CorpusID:271114571
2024
-
[21]
Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al . 2024. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. NeurIPS 36
2024
-
[22]
Jiabin Liu, Chengliang Chai, Yuyu Luo, Yin Lou, Jianhua Feng, and Nan Tang
-
[23]
Malkov and D
Yu A. Malkov and D. A. Yashunin. 2020. Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs. IEEE Trans. Pattern Anal. Mach. Intell. 42, 4 (2020), 824–836. doi:10.1109/TPAMI.2018.2889473
2020
-
[24]
Pu, and Renée J
Fatemeh Nargesian, Erkang Zhu, Ken Q. Pu, and Renée J. Miller. 2018. Table union search on open data. Proc. VLDB Endow. 11, 7 (2018), 813–825. doi:10. 14778/3192965.3192973
2018
-
[25]
Paton, Jiaoyan Chen, and Zhenyu Wu
Norman W. Paton, Jiaoyan Chen, and Zhenyu Wu. 2024. Dataset Discovery and Exploration: A Survey. ACM Comput. Surv.56, 4 (2024), 1–37. doi:10.1145/3626521
2024 doi
-
[26]
Fabio Petroni, Federico Siciliano, Fabrizio Silvestri, and Giovanni Trappolini
-
[27]
Victor Sanh, Lysandre Debut, Julien Chaumond, and Thomas Wolf. 2019. Dis- tilBERT, a distilled version of BERT: smaller, faster, cheaper and lighter. ArXiv abs/1910.01108 (2019). https://api.semanticscholar.org/CorpusID:203626972
2019 arXiv
-
[28]
Aecio Santos, Aline Bessa, Christopher Musco, and Juliana Freire. 2022. A Sketch- based Index for Correlated Dataset Search. In Proc. 38th IEEE ICDE . 2928–2941. doi:10.1109/ICDE53745.2022.00264
2022
-
[29]
Roee Shraga, Haggai Roitman, Guy Feigenblat, and Mustafa Cannim. 2020. Web Table Retrieval using Multimodal Deep Learning. In Proc. 43rd SIGIR. ACM . 1399–1408. doi:10.1145/3397271.3401120
2020
-
[30]
Kaitao Song, Xu Tan, Tao Qin, Jianfeng Lu, and Tie-Yan Liu. 2020. Mpnet: Masked and permuted pre-training for language understanding. NeurIPS 33, 16857–16867
2020
-
[31]
IR-RAG @ SIGIR24: Information Retrieval’s Role in RAG Systems. In Proc. 47th SIGIR. ACM. 3036–3039. doi:10.1145/3626772.3657984
-
[32]
Petros Venetis, Alon Halevy, Jayant Madhavan, Marius Paşca, Warren Shen, Fei Wu, Gengxin Miao, and Chung Wu. 2011. Recovering semantics of tables on the web. Proc. VLDB Endow. 4, 9 (2011), 528–538. doi:10.14778/2002938.2002939
2011
-
[33]
Fei Wang, Kexuan Sun, Muhao Chen, Jay Pujara, and Pedro Szekely. 2021. Re- trieving Complex Tables with Multi-Granular Graph Representation Learning. In Proc. 44th SIGIR. ACM. 1472–1482. doi:10.1145/3404835.3462909
2021
-
[34]
Qiming Wang and Raul Castro Fernandez. 2023. Solo: Data Discovery Using Natural Language Questions Via A Self-Supervised Approach. Proc. ACM Manag. Data 1, 4 (2023), 262:1–262:27. doi:10.1145/3626756
2023 doi
-
[35]
Mohamed Yakout, Kris Ganjam, Kaushik Chakrabarti, and Surajit Chaudhuri
-
[36]
Davison, and Jeff Heflin
Mohamed Trabelsi, Zhiyu Chen, Shuo Zhang, Brian D. Davison, and Jeff Heflin
-
[37]
In Proceedings of the ACM Web Conference 2022
StruBERT: Structure-aware BERT for Table Search and Matching. In Proceedings of the ACM Web Conference 2022 . 442–451. doi:10.1145/3485447. 3511972
2022 doi
-
[38]
Shuo Zhang and Krisztian Balog. 2018. Ad Hoc Table Retrieval using Semantic Similarity. In Proc. WWW. 1553–1562. doi:10.1145/3178876.3186067
2018
-
[39]
Shuo Zhang and Krisztian Balog. 2021. Semantic Table Retrieval using Keyword and Table Queries. ACM Trans. Web 15 (2021), 11:1–11:33
2021
-
[40]
Xinyang Zhao, Xuanhe Zhou, and Guoliang Li. 2024. Chat2Data: An Interactive Data Analysis System with RAG, Vector Databases and LLMs.Proc. VLDB Endow. 17, 12 (2024), 4481–4484. doi:10.14778/3685800.3685905
2024
-
[41]
Erkang Zhu, Dong Deng, Fatemeh Nargesian, and Renée J. Miller. 2019. JOSIE: Overlap Set Similarity Search for Finding Joinable Tables in Data Lakes. In Proc. ACM SIGMOD. 847–864. doi:10.1145/3299869.3300065 10
2019
-
[43]
Li Zhang, Shuo Zhang, and Krisztian Balog. 2019. Table2Vec: Neural Word and Entity Embeddings for Table Population and Retrieval. In Proc. 42nd SIGIR. ACM. 1029–1032. doi:10.1145/3331184.3331333
2019
-
[44]
Shuo Zhang and Krisztian Balog. 2017. EntiTables: Smart Assistance for Entity- Focused Tables. In Proc. 40th SIGIR. ACM. 255–264. doi:10.1145/3077136.3080796
2017
-
[2012]
InfoGather: entity augmentation and attribute discovery by holistic match- ing with web tables. In Proc. ACM SIGMOD. 97–108. doi:10.1145/2213836.2213848
-
[2020]
Table Search Using a Deep Contextualized Language Model. In Proc. SIGIR. ACM. 589–598. doi:10.1145/3397271.3401044
-
[2023]
In Findings of the Association for Computational Linguistics: ACL 2023
Automatic Table Union Search with Tabular Representation Learning. In Findings of the Association for Computational Linguistics: ACL 2023 . 3786–3800. doi:10.18653/v1/2023.findings-acl.233
2023 doi
-
[2024]
In NeurIPS, Vol
TableRAG: Million-Token Table Understanding with Language Models. In NeurIPS, Vol. 37. 74899–74921
Reviewed August 16, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.