{"id":"16a752c6-1384-410c-9564-cfa5204e7282","arxiv_id":"2505.11545","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"TARGET provides a unified benchmark for table retrieval in question answering, fact verification, and text-to-SQL, revealing that dense embedding retrievers outperform BM25 and that retrieval quality predicts downstream accuracy.","lead":"The paper introduces TARGET, a benchmark for evaluating how well retrieval systems find relevant tables before an AI generates answers. Tests on five datasets show dense embedding models outperform keyword search, and better table retrieval directly improves answer quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Spider text-to-SQL results rely on the test split, but the official hidden test lacks gold SQL; the sqlglot extraction in Sec 3.3 cannot be reproduced, making the reported text-to-SQL recall/accuracy unverifiable.","rationale":"Reading the paper in good faith, TARGET is a valuable and carefully designed benchmark, and the dense-over-BM25 finding is likely robust on OTTQA, FeTaQA, and TabFact. However, the text-to-SQL component is the novel extension of the benchmark, and its validity rests on ground-truth table labels. The reader's weakest assumption already flagged sqlglot extraction as unvalidated; I agree, but the more elementary problem is the dataset split: the official Spider test split does not include gold SQL, so the sqlglot extraction described in Sec. 3.3 cannot be applied to the split named in Table 2. This is a concrete, checkable concern. If the authors used the dev split but mislabeled it, the mismatch is a correctness issue; if they used a third-party version with test SQL, that version is not the official benchmark and needs disclosure. In either case, the benchmark's reproducibility for text-to-SQL is at stake. This concern does not overturn the paper's main empirical trend across the QA and fact-verification tasks, so a conditional verdict remains appropriate, pending a clear statement and validation of the Spider data provenance.","tokens_in":13774,"tokens_out":13991,"duration_ms":137878,"concrete_test":"Inspect the released code (anonymous.4open.science/r/target-B7821) and the data-loading script for Spider to identify the exact split and the source of the 'ground-truth query' strings. Then independently parse the Spider dev split (which has gold SQL) with sqlglot, extract table sets, and compare them against the tables actually needed by the gold SQL for a random sample of 100 queries; report the parse success rate and any mismatches. If the extraction is intended for the test split, check whether the SQL file exists; if it does not, recompute Table 3's CR@10 and Table 5's EX on the dev split and compare. If the dense-vs-BM25 ordering varies after this correction, the central claim needs qualification.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparison of dense vs. sparse retrievers depends on correct ground-truth labels for the text-to-SQL tasks. Section 3.3 states: 'To evaluate retrieval for text-to-SQL, we extract all the tables referenced in the ground-truth query using sqlglot and consider them as ground-truth.' Table 2 reports that the authors use the 'test' split of Spider, with 2.1K queries. However, the official Spider test split is used for the leaderboard: its questions and databases are public, but the gold SQL queries are not released. Without the gold SQL, sqlglot cannot extract the tables referenced by the query, so the described pipeline cannot be executed on the claimed split. The paper neither reports the provenance of the test split SQL nor validates the sqlglot extraction (e.g., parse success rate, manual audit). If the authors actually used a non-standard split with leaked SQL, or the dev split under a different name, the reported CR@10 and execution accuracy for Spider are not reproducible as claimed. This is more fundamental than the accuracy of sqlglot itself: the input to that step is missing. Even if a corrected split is available, the lack of any validation of the extraction leaves the text-to-SQL retrieval labels—and hence the benchmark's utility as a reference standard—at risk.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces TARGET, a benchmark for evaluating table retrieval in retrieval-augmented generation (RAG) pipelines over structured data. It integrates five datasets spanning three downstream tasks: question answering (OTTQA, FeTaQA), fact verification (TabFact), and text-to-SQL (Spider, BIRD). The benchmark evaluates retrieval in isolation with recall/capped recall and time, and end-to-end with a GPT-4o-mini generator using task-specific metrics (SacreBleu, P/R/F1, execution accuracy). The authors compare sparse lexical retrievers (BM25, TF-IDF), dense metadata embeddings, dense table embeddings (three models), and dense row-level embeddings. Main findings are that dense embedding-based retrievers generally outperform sparse lexical baselines, metadata matters for lexical retrievers, retrieval performance degrades with corpus size, and downstream accuracy correlates with the rank of the gold table in retrieval results. The paper also includes a No Context baseline to measure memorization effects.","tokens_in":14056,"tokens_out":6683,"duration_ms":60412,"significance":"If the results hold, TARGET fills a clear gap by providing a unified, extensible benchmark for table retrieval across diverse tasks and data characteristics, with standard metrics and a modular API. The inclusion of the No Context baseline, capped recall for multi-table text-to-SQL, and the study of metadata sensitivity are thoughtful design choices. The finding that BM25 is less effective over tables than over text is a useful empirical contribution. The paper ships code and a project website, which supports reproducibility, though the issues below must be addressed before the benchmark can serve as a reliable reference standard.","major_comments":[{"comment":"The manuscript states that the Spider test split (2.1K queries) is used and that ground-truth tables are extracted from the ground-truth query via sqlglot. However, the official Spider test split does not include gold SQL queries, so the described extraction cannot be executed on the claimed split. The paper does not report the provenance of the SQL used for labeling, nor whether a different split (e.g., dev) was used under the name 'test'. This directly undermines the reproducibility of the Spider retrieval recall (Table 3) and execution accuracy (Table 5), and hence the text-to-SQL conclusions.","section":"Section 3.3, Table 2"},{"comment":"Even for BIRD, where gold SQL is publicly available, the paper does not validate the sqlglot-based extraction of ground-truth tables. No parse success rate, manual audit, or error analysis is reported, and the exact sqlglot version is not specified. If the extraction fails or is incomplete for any queries, the capped recall and downstream execution results are corrupted. Since the text-to-SQL relevance labels are a novel contribution of TARGET, this validation is essential for the benchmark's credibility.","section":"Section 3.3"},{"comment":"The claim that 'dense embedding-based retrievers far outperform a BM25 baseline' is too strong. In Table 3, BM25 with table title achieves R@10 of 0.967 on OTTQA, slightly above the best dense table embedding (0.963), and on Spider the gap is modest (CR@10 0.544 vs 0.657). The claim should be qualified as applying on average or for specific datasets, and the per-dataset exceptions should be discussed to avoid overgeneralization.","section":"Abstract, Table 3"},{"comment":"All results are reported from single runs with no variance, error bars, or significance testing. For LLM-based generators, sampling temperature and decoding parameters are not reported, so downstream accuracy differences (e.g., FeTaQA SacreBleu 12.569 vs 13.231) may not be statistically reliable. As a benchmark intended for reference use, TARGET should provide multiple seeds or confidence intervals to support its comparisons.","section":"Table 5, Section 4.2"}],"minor_comments":[{"comment":"There is a typo in the No Context baseline paragraph: 'soely' should be 'solely'.","section":"Section 3.4"},{"comment":"In the 'Benchmarks and Datasets' paragraph, 'rerievers' should be 'retrievers'.","section":"Section 2"},{"comment":"The phrase 'strong negative correlation between retriever performance and downstream task performance' is imprecise; Figure 5 plots the rank of the gold table in the retrieval results against downstream accuracy, so the correlation is between gold-table rank and accuracy, not between retriever performance and accuracy.","section":"Section 4.2"},{"comment":"The caption reads 'Best scores are inbold'; a space is missing between 'in' and 'bold'.","section":"Table 3 caption"},{"comment":"The sentence 'We use the test splits of included datasets for our evaluations' is slightly misleading because OTTQA and BIRD use validation splits; Table 2 is accurate, but the sentence should explicitly say 'test splits when available, otherwise validation splits'.","section":"Section 3.3"},{"comment":"In the Limitations section, 'TARGETdoes' is missing a space; it should read 'TARGET does'.","section":"Section 7"}],"recommendation":"major_revision","confidential_remarks":"The Spider test-split issue is the main blocker: the paper must clarify the provenance of the gold SQL used for labeling and ideally release the processed queries and extracted ground-truth tables. If the authors cannot provide this, I would recommend rejection. I also encourage the editor to ask for validation of the sqlglot extraction (parse success rate, manual audit) for both Spider and BIRD, and to ensure that the generated metadata summaries and any non-standard splits are released with the benchmark. The paper's central ideas are solid, but the current lack of reproducibility for a key component is a serious concern."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nTARGET is worth taking seriously. It does a real service by packaging five table corpora under one API and measuring retrieval both in isolation and end-to-end across three tasks. The dense-versus-sparse finding is clean and intuitively right for tabular data, and the metadata sensitivity and corpus-scale experiments are solid additions. The no-context baseline is a sensible control, and the rank-accuracy correlation figure gives the community a practical signal about gold-table position.\n\nBut the text-to-SQL leg has a load-bearing reproducibility problem. The paper says it evaluates on the Spider test split, and the official Spider test split has no released gold SQL. Section 3.3 says ground-truth tables are extracted from the gold query via sqlglot; on the official test split that extraction has no input. The paper doesn't say where the SQL came from, doesn't report parse success rate, and doesn't include a manual audit. This may be an honest fix — maybe they used the dev split under a different name, or a third-party release with gold SQL — but as written, the reported CR@10 and execution accuracy for Spider cannot be reproduced. The same uncertainty doesn't hit BIRD, whose validation split is public, so the problem is specific to Spider.\n\nBeyond that, the empirical base is thin in the usual way: single runs, no variance, no significance tests. For a benchmark paper that's not always disqualifying, but here the central claim about dense retrievers relies on comparisons across models, and confidence intervals would help. The paper also doesn't link the actual code and data at review time (only an anonymous URL), and the \"strong negative correlation\" statement in the abstract is a little loose — it's actually the rank of the gold table driving downstream accuracy, not retriever performance per se. Minor.\n\nOn the plus side, the authors are transparent about limitations: the metric issue for OTTQA, the lack of row/column labels, the omission of many known retrieval methods. No sign of circularity; the empirical claims are measurements against external datasets. I'd trust the dense-over-sparse story to hold after the Spider fix.\n\nWho is this for? Anyone building or evaluating table retrievers for RAG, and benchmark designers in the structured-data space. It's a solid measurement instrument, not a theoretical contribution.\n\nRecommendation: send it to peer review with the expectation of major revision. Require the authors to clarify the Spider split provenance, validate or replace the sqlglot extraction, and release the exact code and data. The benchmark idea is valuable enough to deserve referee time.","headline":"A genuinely useful table-retrieval benchmark whose text-to-SQL results are currently unreproducible until the Spider test-split provenance and sqlglot extraction are fixed.","tokens_in":14562,"tokens_out":2501,"would_cite":true,"duration_ms":24843,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces TARGET, a benchmark that measures table retrieval for generative tasks and shows that dense table embeddings outperform sparse lexical retrieval while the rank of the retrieved table strongly drives downstream answer…","keywords":["table retrieval","retrieval-augmented generation","dense embeddings","sparse lexical retrieval","BM25","text-to-SQL","fact verification","question answering"],"falsifier":"Manually audit a random sample of Spider and BIRD queries: compare the table sets that sqlglot extracts from the ground-truth SQL against human-annotated ground truth. If even a few percent of queries miss or include extra tables, the reported retrieval recall and downstream execution accuracy figures for text-to-SQL would shift measurably.","tokens_in":13604,"feed_emoji":"📊","tokens_out":10713,"duration_ms":84218,"temperature":0.7,"pith_summary":"The paper introduces TARGET, a benchmark that evaluates table retrieval in isolation and as the first step of end-to-end generative tasks: question answering, fact verification, and text-to-SQL, across five datasets. It claims that dense embeddings of tables, rows, or table metadata reliably outperform sparse lexical retrieval (BM25/TF-IDF) over tabular data, and that lexical search is less effective on tables than on unstructured text. It also reports a strong negative correlation (average Spearman's $\\rho = -0.85$) between the rank of a ground-truth table in retrieval results and the accuracy of the downstream LLM answer, meaning that better retrieval rank directly improves generated answers. A sympathetic reader would care because the benchmark gives the understudied retrieval step in open-domain tabular RAG a standard yardstick and identifies where current retrievers fail, for instance when table titles are uninformative or corpora grow large.","feed_headline":"Dense retrievers beat BM25 on tables and rank predicts answer quality","feed_subtitle":"New benchmark shows dense embeddings find the right tables and that table rank drives LLM answer accuracy.","key_machinery":"The central object is the TARGET benchmark itself: a pipeline that takes a natural-language query and a table corpus, runs a pluggable retriever, hands the top-$k$ tables to a GPT-4o-mini generator, and scores retrieval separately from the downstream task. The load-bearing measurement is capped recall at $k=10$, with denominator $\\min(k, |T_i|)$ when a query needs multiple tables, and the main analytical instrument is the rank-to-accuracy curve: instances are grouped by the rank of the ground-truth table (1-2, 3-4, …, not in top-10), and the monotone drop in downstream accuracy quantifies how much generation depends on retrieval. The retrievers compared are sparse lexical representations (BM25/TF-IDF over columns, rows, and titles), dense metadata embeddings (GPT-4o-mini-generated table summaries embedded with text-embedding-ada-002), dense table embeddings (three text embedding models over column names with or without sampled rows, formatted as markdown), and dense row-level embeddings (per-row sentences embedded with stella_en_400M_v5, then mapped to tables).","core_discovery":"On the paper's own terms, TARGET establishes that dense embedding-based retrievers far outperform a BM25 baseline for table retrieval across question answering, fact verification, and text-to-SQL, and that sparse lexical representations—so effective for free-form text—are markedly less robust for structured data. Across the five datasets, dense table embeddings (particularly stella_en_400M_v5 embeddings of column names plus sampled rows) give the best retrieval recall, while LLM-generated metadata summaries help when table titles are absent or uninformative. The paper further shows that the rank at which the correct table appears in the retrieval list strongly predicts downstream generation quality: the average Spearman's $\\rho = -0.85$ between retriever performance and task performance indicates that relying on a long-context LLM to sort through many tables is a poor substitute for retrieving the right table early.","pith_inferences":["A testable extension: combining BM25-style exact matching with dense embeddings in a hybrid retriever should help on text-to-SQL queries, where the paper notes that queries resemble column names and including rows barely changes recall.","The $-0.85$ rank-accuracy correlation implies that investing in a reranker on top of the initial dense retriever may yield larger end-to-end gains than upgrading the generator, a claim TARGET does not itself test.","The benchmark's reliance on generated metadata summaries suggests a feedback loop worth probing: if the summary generator makes a factual error, retrieval may succeed on the summary while downstream generation fails on the true table contents; a targeted study of summary fidelity could quantify this failure mode.","Because row-level retrieval was not evaluated on BIRD due to scale, the open question remains whether hierarchical retrieval (database first, then table, then row) can make row-level methods practical for analytical text-to-SQL workloads."],"forward_implications":["Dense table embeddings, especially stella_en_400M_v5 embeddings of column names plus sampled rows, are the best current default for table retrieval, outperforming BM25 by large margins on datasets without descriptive titles.","Sparse lexical retrievers become much weaker when descriptive metadata such as table titles is unavailable; LLM-generated table summaries recover much of that gap.","Row-level dense embeddings can match or slightly beat table-level dense embeddings in recall, but their search cost becomes impractical on very large tables such as those in BIRD.","Because retrieval rank correlates with downstream accuracy (average Spearman's $\\rho = -0.85$), feeding many tables into a long-context LLM is not a reliable substitute for getting the correct table near the top of the retrieval list.","Retrieval quality degrades as the corpus grows beyond roughly 10K added tables, so table retrievers need to be tested at scale, with lexical retrieval degrading fastest."],"supporting_citations":[{"why":"Supplies the sparse lexical retriever formulation and the OTTQA corpus used for open-domain question answering.","marker":"Chen et al., 2021"},{"why":"Supplies stella_en_400M_v5, the embedding model that gives the best dense table and row retrieval results.","marker":"Zhang et al., 2025"},{"why":"Defines capped recall, the metric TARGET uses to evaluate text-to-SQL retrieval when a query needs multiple tables.","marker":"Thakur et al., 2021"},{"why":"Provides the prompt template for text-to-SQL generation with retrieved database schemas.","marker":"Talaei et al., 2024"},{"why":"Supplies the BIRD dataset and its execution-accuracy evaluation implementation for text-to-SQL.","marker":"Li et al., 2024a"},{"why":"Supplies the Spider dataset used for text-to-SQL retrieval and generation.","marker":"Yu et al., 2018"},{"why":"Supplies TabFact, the corpus and statements for the fact-verification task.","marker":"Chen et al., 2020"},{"why":"Supplies FeTaQA, the free-form table question answering dataset used in the question answering task.","marker":"Nan et al., 2022"},{"why":"Serves as the design template for the benchmark and as the reference point showing that lexical retrieval works well on text.","marker":"Muennighoff et al., 2023"},{"why":"Documents the GPT-4o-mini model used as the generator for all downstream task evaluations.","marker":"Hurst et al., 2024"}],"fun_headline_variants":["Dense retrievers outperform BM25 on tables, rank predicts answer quality","Table retrieval: dense beats sparse, and rank drives LLM accuracy","For tables, retrieval rank predicts LLM answer quality, dense wins","Dense embeddings beat BM25 on tables; early rank means better LLM answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The text-to-SQL conclusions rest on the assumption that sqlglot automatically extracts the complete and correct set of ground-truth tables for every Spider and BIRD query, and this extraction (Section 3.3) is not validated.","fun_headline_variants_meta":{"raw":{"variants":["Dense retrievers outperform BM25 on tables, rank predicts answer quality","Table retrieval: dense beats sparse, and rank drives LLM accuracy","For tables, retrieval rank predicts LLM answer quality, dense wins","Dense embeddings beat BM25 on tables; early rank means better LLM answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000501,"raw_usage":{"total_tokens":2440,"prompt_tokens":928,"completion_tokens":1512,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":544,"completion_tokens_details":{"reasoning_tokens":1433}},"tokens_in":544,"tokens_out":1512,"duration_ms":11860,"temperature":1.0,"reasoning_tokens":1433,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T21:24:49.643139+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Manually audit a random sample of Spider and BIRD queries: compare the table sets that sqlglot extracts from the ground-truth SQL against human-annotated ground truth. If even a few percent of queries miss or include extra tables, the reported retrieval recall and downstream execution accuracy figures for text-to-SQL would shift measurably.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies TabFact, the corpus and statements for the fact-verification task."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies FeTaQA, the free-form table question answering dataset used in the question answering task."}],"review_version":1}