{"id":"8d8212d9-f2c5-4b7f-9efb-a958ee7bfcc3","arxiv_id":"2504.15849","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper defines NL-conditional table discovery, releases the nlcTables benchmark, and shows that six existing table search methods underperform on it.","lead":"This paper introduces a new benchmark task that combines a user's table with a natural language condition to search for other tables, and releases a dataset of roughly 22,000 candidate tables with 21,200 relevance labels. It reports that six existing table discovery methods perform poorly on this combined task, pointing to a gap in current table retrieval tools.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Auto-labeled ground truth from §4.2 lacks external relevance validation; negative tables are sampled so 50% of rows still satisfy the condition, so the reported baseline gaps may be artifacts of construction thresholds.","rationale":"The paper proposes a new task, a dataset, and an evaluation of six baselines. For the central claim to hold, the benchmark must accurately reflect real user relevance for query-table-plus-NL-condition inputs. The weakest point is the auto-labeling pipeline in Section 4.2: positives and negatives are generated by splitting the same original table, and the NL condition is filled from the same column/value used for splitting. This creates a closed loop where labels are correct by construction with respect to the pipeline, but there is no evidence that these labels match human judgments of relevance. The reader identified exactly this assumption, and we agree. We additionally sharpen the concern: the subtable sampling thresholds (80% of positive rows satisfying, 50% of negative rows not satisfying) mean negative tables contain a substantial fraction of rows that do satisfy the NL condition. Such tables are arguably partially relevant, yet they are treated as hard negatives. If a user asks for 'grade > 80', a table where half the rows have grade > 80 is not clearly irrelevant. The reported performance gaps could therefore be inflated by these design choices. We are not claiming the authors acted improperly or that the dataset is useless; rather, the benchmark's validity is conditional on external validation. The proposed human study directly tests whether auto-labels align with human relevance; if agreement is high, the concern is resolved. Given the moderate confidence and the need for this validation, the verdict should remain CONDITIONAL.","tokens_in":16958,"tokens_out":8079,"duration_ms":77317,"concrete_test":"Conduct an external human relevance study on a stratified sample of 100 auto-labeled queries (25 each from nlcTables-U, nlcTables-U-fz, nlcTables-J, and nlcTables-J-fz). For each query, have three annotators judge binary relevance for the query table plus NL condition over all positive and negative tables and a random sample of 50 unlabeled repository tables. Compute inter-annotator agreement and agreement with the auto-labels. If Cohen's kappa between auto-labels and majority human labels is below 0.6, or if more than 10% of auto-negatives are judged relevant, the benchmark's validity as a measure of nlcTD is not established.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that nlcTables is a reliable benchmark and that six methods are inadequate depends on the correctness of the auto-generated relevance labels. In Section 4.2 (Query Construction, Subtable Sampling), positive and negative ground truths are both produced by splitting the same original table from which the query table is drawn, using the same column/value that appears in the template-filled NL condition. This makes positives positive by construction. The paper also states that negative tables are sampled so that '50% of rows in the negative table do not [satisfy the condition]', meaning half of the rows in a negative table do satisfy the NL condition. Such tables are at best partially relevant, yet they are binarized as irrelevant. No human relevance judgments are reported for this auto-labeled majority; the only human input is column-type annotation, SQL-based labeling for Spider-derived queries, and manual review of the LLM semantic augmentation. Consequently, the large performance gaps in Section 5 could reflect these design choices (shared provenance, 80%/50% sampling thresholds) rather than the intrinsic difficulty of NL-conditional table discovery. This is load-bearing because the paper's main contribution is the benchmark itself and the conclusion that current methods are inadequate for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NL-conditional table discovery (nlcTD), a task in which a user supplies both a query table and a natural-language condition, and defines a 16-category taxonomy covering NL-only, union, join, and fuzzy conditions. It introduces nlcTables, a benchmark constructed by splitting 193 original tables from OpenData, WebTable, and GitTable, using templates and LLM-based semantic augmentation to generate NL conditions, plus a smaller set of SQL-derived labels from Spider. The paper reports 627 queries in the abstract and conclusion but 647 in Table 2, with 22,080 candidate tables and 21,200 relevance annotations, and evaluates six existing table-discovery methods, finding substantial performance gaps. The dataset, construction framework, and baseline implementations are publicly released.","tokens_in":17215,"tokens_out":5843,"duration_ms":51909,"significance":"If the dataset and evaluation are valid, nlcTables fills a real gap: existing table-discovery benchmarks cover only keyword or query-table inputs, while the nlcTD setting matches the interactive data-assistant paradigm. The paper has several strengths: a detailed automated construction pipeline with configurable hyperparameters, a 16-category taxonomy, public code and baselines, and multi-dimensional evaluation across condition categories, fuzzy variants, positive-to-negative ratios, and dataset scale. However, the validity of the benchmark depends on the auto-labeled ground truth and on the way baselines were adapted to a new input modality; both need additional support before the reported performance gaps can be attributed to the nlcTD task rather than to construction artifacts.","major_comments":[{"comment":"The ground-truth generation is circular in a way that threatens benchmark validity. Positive labels are produced by the same splitting procedure that creates the queries: horizontal splitting for union queries applies the row-level condition derived from the same column and value used in the NL template, and join queries split one original table so the new column referenced in the template is present in the positive ground truth. Negative tables are sampled so that only 50% of their rows violate the condition, meaning the other 50% still satisfy it, yet these tables are binarized as irrelevant. No external human relevance judgments are reported for the auto-labeled majority; the only human input is column-type annotation, SQL-based labeling for Spider-derived queries, and manual review of the LLM semantic augmentation. The large performance gaps in Section 5.2 (Figures 5 and 6) could therefore reflect the 80%/50% sampling thresholds and shared provenance rather than the intrinsic difficulty of nlcTD. Please provide a human-annotated subset with inter-annotator agreement, report how the baseline gaps change when the 80% and 50% thresholds are varied, and state explicitly what fraction of ground truths come from each labeling source.","section":"Section 4.2 (Query Construction) and (Labeling Queries)"},{"comment":"The paper never specifies how each baseline receives the two-part input (query table, NL condition). Gtr and StruBert accept text queries but are applied to nlcTables-U and nlcTables-J; it is unclear whether the query table is serialized into the text and how the NL condition is attached. Josie and DeepJoin take a single query column; the paper does not state which column is used or how the NL condition filters candidates. Santos and Starmie take a query table; the paper does not say how the NL condition is incorporated. Without a precise statement of these adapters, the conclusion that existing methods are not well-suited is not reproducible and cannot be separated from arbitrary choices of input serialization. Please document the exact input construction for each baseline, or point to versioned code and configuration files, in the paper.","section":"Section 5.1 and 5.2"},{"comment":"Dataset statistics are internally inconsistent. The abstract and conclusion state 627 queries, while Table 2's nlcTables-Full row and the sum of the five type rows give 647 queries. Furthermore, Table 5 reports nlcTables-J Large (100%, |T|=9,347), which is nearly twice the nlcTables-J total repository of 4,871 in Table 2. These discrepancies affect the headline dataset scale and must be reconciled before the benchmark statistics can be trusted.","section":"Section 4.3, Table 2, Table 5, abstract, and conclusion"},{"comment":"The paper states that refined classification proportions are omitted due to page limit, yet RQ2 in Section 5.3 reports performance per condition category. Without counts for the 16 taxonomy subcategories, a reader cannot determine whether the comparisons in Table 4 are based on enough queries or whether the dataset's difficulty is an artifact of category imbalance. Please provide the distribution of queries across the 16 subcategories and across column types, either in the paper or in an appendix.","section":"Section 4.3 and Section 5.3"}],"minor_comments":[{"comment":"The phrase 'they typically accept sorely keywords' should read 'they typically accept solely keywords'.","section":"Section 3.1"},{"comment":"The text and labels in Figures 2 and 3 are very small in the PDF; please enlarge them or provide a vector version for readability.","section":"Figures 2 and 3"},{"comment":"Hyperparameters such as L_scale, L_col, dup_rate, and temp_num are named but not defined numerically or listed with defaults; a table of default values would improve reproducibility.","section":"Section 4.2"},{"comment":"No hyperparameters or training details are given for the six baselines; please state whether official implementations and default settings were used.","section":"Section 5.1"},{"comment":"The claims about performance drops under fuzzy queries and different positive-to-negative ratios would be stronger with error bars or significance tests across query subsets.","section":"Figures 6 and 7"},{"comment":"The row for nlcTables-K shows backslashes for query table rows and columns, which is correct because K queries have no query table, but the caption should explain this explicitly.","section":"Table 2"}],"recommendation":"major_revision","confidential_remarks":"The paper is a potentially useful dataset contribution, but the circular auto-labeling and the unspecified baseline input adaptation are load-bearing issues that require substantive revision. The 627/647 inconsistency and the Table 5 vs Table 2 repository-size conflict also suggest the manuscript needs a careful consistency pass. If the authors provide human validation, threshold-sensitivity analysis, and full baseline input details, the paper could be acceptable; I do not see grounds for rejection at this stage."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper does something concrete: it formalizes a task that has been floating around—query table plus natural language condition—gives it a 16-category taxonomy, and releases a reproducible construction framework and a dataset of ~22k tables and ~21k labels. The six-baseline evaluation is a fair first cut, and the finding that existing methods stumble on the combined input is credible. The GitHub release with 12k+ lines of code and documented manual annotation hours is a real asset. Credit where due: this is a plausible community benchmark in the making.\n\nThe soft spots are real and not minor. The stress-test note holds up. For the auto-labeled majority, positives are positive by construction: they come from splitting the same table that produced the query, and the NL condition is built from the same column and value. Negatives are sampled so that 50% of their rows still satisfy the condition, then binarized as irrelevant. That makes the measured performance gap partly an artifact of the sampling thresholds, not just the intrinsic difficulty of the task. There is no external human relevance judgment on the auto-labeled part, only manual review of the LLM augmentation. The paper should report how much of the test set is auto-labeled versus manually SQL-derived, and should validate a sample against independent judges.\n\nThe 627-vs-647 query count discrepancy between abstract, Table 2, and conclusion is sloppy and needs fixing. Category proportions being omitted due to page limit is understandable but should be in an appendix; the taxonomy is a contribution and the proportions affect interpretation.\n\nNone of this kills the contribution. The task is new, the dataset is the first of its kind, and the baselines' inadequacy likely stands even if the absolute numbers shift after better labels. But the dataset's current labels are load-bearing for the main claim, so the validity questions need to be answered before it becomes a community standard.\n\nBottom line: send it to peer review. A good referee will ask for the external validation and the count fix, but the core is sound enough to justify the time.","headline":"A genuinely useful new benchmark for NL-conditional table discovery, with a real caveat about auto-generated ground truth; worth a serious referee.","tokens_in":17699,"tokens_out":1437,"would_cite":true,"duration_ms":15431,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper defines NL-conditional table discovery, builds a 22,080-table benchmark, and shows existing methods underperform on the combined table-plus-language input.","keywords":["nlcTD","table discovery","table union","table join","natural language conditions","benchmark dataset","semantic search","data lake"],"falsifier":"Run a human annotation study on a stratified random sample of at least 100 nlcTables queries across all condition types, and measure agreement between the auto-generated labels and independent human relevance judgments; if more than ten percent of auto-labeled positive tables are judged by annotators as not satisfying the stated natural language condition, or if negatives are judged as satisfying it, the dataset's validity as ground truth would be undermined.","tokens_in":16786,"feed_emoji":"📊","tokens_out":5432,"duration_ms":46297,"temperature":0.7,"pith_summary":"The paper introduces NL-conditional table discovery (nlcTD), a task in which a user supplies both a query table and a natural language condition, such as 'unionable tables with grade above 90,' and the system returns a ranked list of relevant tables from a repository. To support this task, the authors construct nlcTables, a benchmark of 22,080 candidate tables, 627 queries covering NL-only, union, join, and fuzzy conditions, and 21,200 relevance annotations produced by an automated table-splitting pipeline supplemented with manual SQL-derived labels. They evaluate six established table discovery methods and find that all of them underperform, especially on numerical, date, and mixed-mode conditions, which the paper interprets as evidence that the combined table-plus-condition input requires new techniques. A sympathetic reader would care because, if the benchmark is valid, it provides a reusable testbed that measures exactly the missing capability of current table search systems.","feed_headline":"Benchmark marries natural language to table discovery","feed_subtitle":"nlcTables pairs 627 queries with 22,080 tables; six search methods all fall short on the combined input.","key_machinery":"The load-bearing mechanism is a table-splitting construction framework. Large tables are filtered and manually annotated for column types; union queries are built by horizontal splitting on a column value (rows satisfying the condition form positives, rows failing form negatives), while join queries use vertical splitting with a shared join key, and mixed-mode queries layer two conditions sequentially. Each split is paired with a template-generated NL condition, and all positive tables are positive by construction because they come from the same original table as the query table and meet the condition used to create the query; subtable sampling fixes thresholds (80% of positive rows satisfy the condition, 50% of negative rows do not) and LLM rewriting adds linguistic diversity, while a minority of labels come from manual annotation of Spider SQL queries.","core_discovery":"The paper's central claim is that table discovery should be a ternary relevance problem: a candidate table is relevant only if it is related to the query table and simultaneously satisfies the user's natural language condition. On nlcTables, the paper shows that existing keyword-based methods (GTR, StruBERT), table union methods (Santos, Starmie), and table join methods (Josie, DeepJoin) all fail to handle this combination, with NDCG@10 values dropping to roughly 0.3 to 0.55 on harder condition categories and degrading further as negative examples increase or fuzzy transformations are added. The paper concludes that the combined input modality is qualitatively harder than either keyword search or query-table search alone.","pith_inferences":["The auto-labeling pipeline assumes that a subtable satisfying a template condition is what a human would call relevant; if that assumption fails for a nontrivial fraction of queries, the reported performance gaps could shrink or grow, and a human relevance study on a sample of nlcTables queries would settle this.","Because query tables are always subtables of a larger original table, the benchmark may not cover scenarios where the query table has independent provenance, so constructing queries from externally sourced tables would test generalization.","The template conditions often reference a specific column and value, whereas real user language is messier; an extension that replaces templates with free-form LLM-generated conditions, validated by human judges, would increase realism.","If a simple condition-aware reranker (for example, an LLM that checks whether candidate tables satisfy the condition) substantially improves over the reported baselines, that would identify the missing ingredient precisely."],"forward_implications":["Existing table discovery methods cannot be directly ported to nlcTD; the paper's baseline results establish a concrete accuracy gap that new models must close.","Future nlcTD methods should condition retrieval on both the query table and the NL condition, since ignoring either side yields the low recall and precision the paper documents.","Topic-level conditions are comparatively easy, while numerical, date, and mixed-mode conditions are the hardest, so benchmarks should weight toward these harder types.","Dataset composition matters: increasing the positive-to-negative ratio or scaling the repository changes measured accuracy and efficiency, so reports on nlcTables should state which configuration was used.","The released scripts let researchers generate custom nlcTD datasets with adjustable hyperparameters, enabling controlled stress tests of new methods."],"supporting_citations":[{"why":"Supplies the table-splitting methodology that this paper adapts to generate union and join ground truths at scale.","marker":"[11]"},{"why":"Early table union search with a splitting-based dataset that informs the union query construction.","marker":"[24]"},{"why":"Knowledge-graph-based union search method used as a baseline on nlcTables-U.","marker":"[19]"},{"why":"Set-similarity join search method used as a baseline on nlcTables-J.","marker":"[41]"},{"why":"Embedding-based join search method used as a baseline on nlcTables-J.","marker":"[14]"},{"why":"Graph-transformer keyword-based table retrieval method used as a baseline across dataset types.","marker":"[33]"},{"why":"Structure-aware BERT keyword-based table retrieval method used as a baseline across dataset types.","marker":"[31]"},{"why":"Source of real SQL queries used for the manual labeling component that complements the automated ground truths.","marker":"[21]"}],"fun_headline_variants":["NL-conditional table discovery benchmark stumps six methods","Natural language requirements trip up table discovery","nlcTables: NL conditions add a new hurdle to table search","Table discovery methods fail on natural language conditions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark's ground truth is only as sound as the assumption that a table produced by the automated splitting pipeline, together with its template-filled natural language condition, reflects what a real user would judge relevant for that query.","fun_headline_variants_meta":{"raw":{"variants":["NL-conditional table discovery benchmark stumps six methods","Natural language requirements trip up table discovery","nlcTables: NL conditions add a new hurdle to table search","Table discovery methods fail on natural language conditions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000574,"raw_usage":{"total_tokens":2670,"prompt_tokens":862,"completion_tokens":1808,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":478,"completion_tokens_details":{"reasoning_tokens":1747}},"tokens_in":478,"tokens_out":1808,"duration_ms":12027,"temperature":1.0,"reasoning_tokens":1747,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:16:05.292706+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human annotation study on a stratified random sample of at least 100 nlcTables queries across all condition types, and measure agreement between the auto-generated labels and independent human relevance judgments; if more than ten percent of auto-labeled positive tables are judged by annotators as not satisfying the stated natural language condition, or if negatives are judged as satisfying it, the dataset's validity as ground truth would be undermined.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the table-splitting methodology that this paper adapts to generate union and join ground truths at scale."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Source of real SQL queries used for the manual labeling component that complements the automated ground truths."}],"review_version":1}