{"id":"e70437d8-478c-4428-9923-3e1565b8b5c3","arxiv_id":"2501.16220","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"The paper introduces DB routing, a task of ranking databases for answerability, with synthesized benchmarks from Spider and BIRD showing that LLMs outperform embeddings but both struggle with many or similar databases.","lead":"This paper defines a new task, sending a natural-language question to the correct database among many, and creates two benchmarks from existing text-to-SQL datasets. It compares open-source LLMs with embedding models and finds the task grows harder with more databases, with similar-domain databases, and with less domain knowledge.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Dataset labels are provenance-based, not answerability-based: the paper's own error examples show the exactly-one-answerable-DB assumption fails, so the benchmark's ground truth may not measure the task it defines.","rationale":"The reader's weakest assumption is exactly this label-validity problem; my read agrees and I found no additional concern that changes the verdict. The paper's qualitative findings (LLMs beat embeddings when context fits, task-specific fine-tuning helps, more DBs and closer domains make routing harder) could survive such an audit, but the quantitative benchmark numbers cannot be fully trusted until labels are validated. No code/data release and the mAP metric issue are secondary; they affect reproducibility and reporting, not the core argument's correctness. So the appropriate verdict remains CONDITIONAL, i.e., UNCHANGED from the reader's verdict.","tokens_in":14971,"tokens_out":3215,"duration_ms":31147,"concrete_test":"Perform an answerability audit on the test splits. Sample 200 questions from Spider-Route in-domain, 200 from Spider-Route cross-domain, and 200 from Bird-Route in-domain. For each sample, (i) execute the gold SQL against the gold DB to confirm answerability, (ii) use an independent schema-matching tool plus the authors' top-5 candidates to identify all other DBs whose schema and data can support a correct answer (e.g., by adapting the gold SQL or asking two annotators to verify), and (iii) have two annotators label whether each candidate DB answers the question. Recompute R@1/R@3/mAP counting every answerable DB as correct; compare with the provenance-based labels.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that the benchmark measures routing to any DB that can answer a query. Section 2 defines relevance as 'answerability' and assumes every question maps to exactly one DB; Section 3.1 sets the ground-truth DB to be the DB on which the question was posed in Spider/Bird. But answerability and provenance are different. A question like 'Find the number of albums.' is answerable from multiple music DBs in the same vertical, as the paper's own error analysis (Section 7.5.1) admits; 'Count the number of tracks.' is ambiguous between songs and race tracks. Section 3.1 filters only the in-domain test split to avoid questions shared across DBs in the original train set; it does not verify that no other DB can answer the remaining questions, and the cross-domain split is not filtered at all. Consequently, the labels are a set of sufficient provenance assignments, not the set of all answerable DBs. A model that returns any genuinely answerable DB is scored as wrong, so R@1/mAP understate true answerability performance. The findings that difficulty increases with more DBs and with closer domains (RQ2/RQ3) may be partly an artifact of more candidate DBs creating more plausible-but-marked-wrong alternatives, rather than a genuine property of the routing task. Because every headline number and research-question conclusion is computed against these labels, this is the most load-bearing vulnerability in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper defines a new task, database routing: given a natural-language query and a set of databases, rank the databases by whether they can answer the query. The authors construct two benchmarks, Spider-Route and Bird-Route, by converting the Spider and BIRD-SQL NL-to-SQL datasets into routing test sets, taking the database on which each question was originally posed as the ground-truth database. They evaluate three families of approaches: zero-shot Llama3-70B prompting, pre-trained SBERT cosine-similarity retrieval, and task-specific SBERT models fine-tuned with contrastive pairs. The experiments address four research questions: whether in-domain versus cross-domain training helps, whether domain similarity hurts, whether increasing the number of databases hurts, and whether external metadata helps. The main reported findings are that Llama3 outperforms embedding-based approaches when it can be applied, task-specific fine-tuning helps especially in-domain, closer domains make routing harder, more databases make routing harder, and metadata improves Bird-Route performance.","tokens_in":15227,"tokens_out":8431,"duration_ms":73089,"significance":"If the benchmark and its label assumptions were sound, this would be a useful first benchmark for an enterprise-relevant problem that is distinct from document retrieval and API/tool routing. The paper names a genuinely under-studied setting, and the research-question structure is sensible. The error analysis in Section 7.5 is informative, and the comparison of pre-trained versus fine-tuned embeddings across in-domain and cross-domain settings is a useful empirical contribution. The benchmark construction from existing NL-to-SQL datasets is also a practical way to bootstrap training data. However, the central issue is that the ground-truth labels are provenance-based rather than answerability-based, which conflicts with the paper's own task definition and with examples in its error analysis. The metric mislabeling and the hybrid nature of the Llama3 baseline further weaken the quantitative conclusions as currently stated. The benchmark could become a valuable resource after relabeling and clarification.","major_comments":[{"comment":"The ground-truth labels do not implement the task definition as stated. Section 2 defines relevance as answerability and assumes every question is answerable by exactly one database, and Section 3.1 sets the ground truth to the database on which the question was posed in the original NL-to-SQL dataset. Section 7.5.1 then documents examples such as 'Find the number of albums.' being answerable by multiple databases (music_2 and chinook_1) and 'Count the number of tracks.' admitting multiple readings. The filter described in Section 3.1 applies only to the in-domain split and ensures that questions shared across databases in the original training set are not in that split; no equivalent check is applied to the cross-domain split or to Bird-Route. Consequently, the labels are a set of sufficient provenance assignments, not the set of all answerable databases, and a model that returns any genuinely answerable database is scored as wrong. This affects every headline number and the RQ2/RQ3 conclusions, because adding more or closer databases creates more plausible-but-marked-wrong alternatives. A concrete fix is to re-annotate a sample of test questions (or verify answerability by executing candidate SQL) and report an any-answerable-DB variant of R@1/R@3 alongside the provenance-based metric.","section":"Sections 2, 3.1, and 7.5.1"},{"comment":"The Llama3 baseline is not a pure language-model ranker for the in-domain splits. Section 4.1 states that when the databases do not fit into the 8K context, the authors first retrieve the top-10 databases for Spider-Route using pre-trained SBERT and top-3 tables, and then ask Llama3 to re-rank those candidates; for Bird-Route the approach is not used at all. The Llama3 rows in Table 3 therefore describe a hybrid retriever-plus-LLM pipeline whose R@1 is upper-bounded by the retriever's recall@10. The abstract and conclusion claim that 'open-source LLMs perform better than embedding based approach' without this caveat. The paper should report the candidate-retriever recall, ablate the LLM re-ranker against the retriever alone, and qualify the conclusion to the hybrid setting.","section":"Section 4.1, Tables 3 and 5, Abstract and Conclusion"},{"comment":"The metric called mAP is not mean average precision as that term is standardly used. The text defines mAP for a question as 1/i where i is the rank of the correct database; averaged over questions this is mean reciprocal rank (MRR). Mean average precision for a ranking task would average precision over the ranked list of databases, not the reciprocal rank of a single relevant item. Since every result table reports this quantity, the numbers cannot be compared with mAP values elsewhere in the literature. Please rename the metric to MRR or provide the correct mAP computation.","section":"Section 5.1, Tables 2-8"},{"comment":"The negative-pair count in the contrastive training data is inconsistent with the described procedure. With 5,959 training questions and 140 databases, pairing each question with 'all DBs except the one it originally mapped to' yields 5,959 × 139 = 828,301 pairs, not the stated 19,460. If negatives were sampled, restricted to vertical clusters, or otherwise filtered, that procedure must be described. As written, the reported number contradicts the text and prevents reproduction of the task-specific embedding models.","section":"Section 3.1"},{"comment":"The within-vertical and across-vertical definitions appear inverted and are inconsistent with the table values. The text says that when the top-ranked database and the ground-truth database are in the same cluster, across-vertical Recall@1 is 1 and within-vertical Recall@1 is 0, and the reverse when the clusters differ. Under that reading, the two metrics are complementary for each question, yet Table 2 shows W-V R1 = 96.71 and A-V R1 = 98.64, which cannot both hold if they are complements. The names also conflict with the RQ2 discussion, where same-cluster confusions are described as confusions 'within domain'. Please define each metric over a precise question subset and recompute or relabel the columns before the RQ2 conclusion can be supported.","section":"Section 5.1 and RQ2, Tables 2, 3, and 5"}],"minor_comments":[{"comment":"The sentence 'Here in Table ??, we provide the details of the prompt...' has a broken cross-reference; it should point to Table 10.","section":"Appendix 7.2"},{"comment":"The contrastive-loss margin m is never given a value; please report the chosen margin for the reported experiments.","section":"Section 4.3"},{"comment":"The filtering step that keeps shared questions in the training set is described only for the in-domain test split; the same check should be documented for the cross-domain split or its absence justified.","section":"Section 3.1"},{"comment":"There are several typos: 'set-of data-sources' in the abstract, 'with a the model' in Section 2, and 'constrastive' in Section 4.3.","section":"Abstract and general exposition"},{"comment":"The vertical-cluster listings have formatting issues, including trailing commas and entries split across lines without clear delimiters, which makes the clusters harder to parse.","section":"Appendix 7.3, Table 12"},{"comment":"Section 4.1 refers to an instruction-tuned Llama3 70B model, but reference [16] is the original LLaMA paper; the specific model card or the Llama-3-70B-Instruct release should be cited.","section":"Section 4.1 and References"}],"recommendation":"major_revision","confidential_remarks":"The provenance-versus-answerability issue is the main reason this is a major revision rather than a reject: it is serious, but it is in principle fixable by re-annotating or SQL-verifying test labels and by reporting an any-answerable-DB metric. The paper also does not mention releasing the synthesized benchmarks or the fine-tuning code; given that the benchmark is the main contribution, a release would substantially strengthen a revision. I would not recommend acceptance until the label validity and metric-definition issues are resolved."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is a reasonable first step on a real problem: routing a natural-language query to the database that can answer it. The task definition is genuinely new relative to prior routing work (LLM routing, tool/API routing), and synthesizing Spider-Route and Bird-Route from existing NL-to-SQL datasets is a legitimate contribution. The baselines are sensible — embedding similarity, fine-tuned SBERT, and Llama3 prompting — and the authors do some careful experimental hygiene, especially the controlled 20-DB sampling in RQ1 and the manual vertical clusters for the difficulty analysis. The main qualitative findings (fine-tuning helps; more DBs and closer domains hurt; metadata helps) are plausible and worth reporting.\n\nThe central problem is the label definition. Section 2 assumes every question maps to exactly one answerable DB, and Section 3.1 uses the original dataset's provenance as ground truth. But the paper's own error analysis shows the assumption is false in the evaluation data: \"Find the number of albums\" is answerable by both music_2 and chinook_1, and \"Count the number of tracks\" is ambiguous between songs and race tracks. So the benchmark measures whether a model retrieves the original DB, not the set of answerable DBs. A model that returns a genuinely answerable DB is scored as wrong. This understates true answerability performance and likely inflates the difficulty trends in RQ2 and RQ3, since more candidates mean more plausible-but-marked-wrong alternatives. The cross-domain split is not even filtered for shared questions. That is the load-bearing flaw.\n\nTwo fixable issues add noise. First, what they call mAP is actually mean reciprocal rank (1/i where i is the position of the correct DB); that should be renamed or recomputed. Second, for the large repository splits, the Llama3 results are not pure LLM ranking — they rely on a pre-trained embedding candidate filter (top-10 DBs, top-3 tables). That limits the \"LLMs beat embeddings\" claim to the small-repository case. Also, no code or data is released, which is a real gap for a benchmark paper.\n\nThese problems are all fixable. The qualitative findings are likely to survive a re-labeling; it's the quantitative benchmark values that need care. This paper deserves a serious referee, not a desk reject. If the authors tighten the labels, release the data, and fix the metric, it becomes a citable resource for the text-to-SQL and retrieval communities.","headline":"A useful first cut at a real enterprise problem, with a load-bearing label-design flaw that the authors nearly admit in their own error analysis.","tokens_in":15792,"tokens_out":1928,"would_cite":false,"duration_ms":19630,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper establishes a new benchmark task, routing natural-language questions to the database that can answer them, and shows current LLM and embedding baselines degrade sharply as the number of candidate databases grows.","keywords":["database routing","natural language querying","enterprise search","query routing","text-to-SQL","sentence embeddings","large language models","benchmark construction"],"falsifier":"Take the questions that at least one method routes incorrectly and check answerability directly: execute the ground-truth SQL on each plausible candidate database, or have annotators say which databases can answer the question. If a large share of the errors are questions answerable by two or more databases, as the paper's own 'Find the number of albums' example suggests, then the reported Recall@1 understates routing quality and the benchmark labels need revision.","tokens_in":14741,"feed_emoji":"🗄️","tokens_out":5336,"duration_ms":47603,"temperature":0.7,"pith_summary":"The paper defines a new task: given a natural-language question and a repository of enterprise databases, rank the databases by whether they can answer the question. It builds two benchmark sets, Spider-Route and Bird-Route, by recycling the questions and schemas of existing text-to-SQL datasets and treating the database a question was originally posed on as the correct answer. Using these benchmarks, the paper shows that an open-source 70B language model routes more accurately than embedding-based similarity but cannot scale to large repositories because of context-length limits, and that a small embedding model fine-tuned with contrastive pairs recovers much of the gap. The paper also shows the task grows predictably harder with more candidate databases, with databases from similar domains, and with questions that are ambiguous or need domain knowledge. A sympathetic reading is that the routing task itself is real and measurable, and none of the current baselines solve it at realistic enterprise scale.","feed_headline":"Routing a question to the right database gets harder at scale","feed_subtitle":"New benchmarks from text-to-SQL datasets show 70B LLMs beat embeddings but hit context limits as database counts rise.","key_machinery":"The mechanism that carries the argument is a ranking pipeline over database schemas expressed as Data Definition Language (DDL) strings. The scoring function $f(q_i, S_j)$ is implemented in three ways: zero-shot instruction-tuned Llama3 ranking databases in a prompt; cosine similarity between pre-trained Sentence-BERT embeddings of the question and the schema string; and Sentence-BERT models fine-tuned with a contrastive margin loss on synthesized positive and negative question-schema pairs. For Bird-Route, whose schemas exceed the embedding context, the pipeline becomes two-stage: retrieve relevant domain-statement evidence from each database's metadata, then retrieve relevant tables, then pool table scores into a database score. The DDL-plus-metadata text is the object the scoring functions operate on, so the paper's conclusions are tied to how faithfully that text represents a database.","core_discovery":"The central claim is that DB routing is a distinct, learnable task and that current retrieval and ranking baselines are not sufficient. The paper constructs Spider-Route (140 train databases, 20 cross-domain test databases) and Bird-Route (69 train, 11 cross-domain) by converting Spider and Bird-SQL schemas to DDL and using each question's original database as ground truth. Empirically, Llama3 70B achieves 95.45% Recall@1 on Spider-Route cross-domain where all schemas fit in context, versus 87.71% for pre-trained embeddings and 91.78% for fine-tuned embeddings; on the 140-database in-domain split Llama3 drops to 59.84%, while fine-tuned embeddings reach 55.04%. The same split shows within-vertical Recall@1 much lower than across-vertical Recall@1, meaning mistakes mostly confuse databases from similar domains. The authors conclude that routing difficulty scales with source count and domain overlap, benefits from domain metadata, and is degraded by ambiguous queries.","pith_inferences":["The paper's single-ground-truth labels are inherited from text-to-SQL provenance, so a stronger evaluation would annotate multi-database answerability and include unanswerable queries; until then, reported recall may understate or overstate true routing quality depending on how often the label is ambiguous.","The two-stage retrieve-evidence-then-tables recipe for Bird-Route suggests a scalable enterprise architecture: cheap embedding prefiltering to shrink the candidate set, followed by an LLM reranker that sees only short schemas; the paper stops short of testing this combination.","The same contrastive fine-tuning approach could transfer to heterogeneous sources such as knowledge graphs and document stores if each source is represented by a comparable schema-like metadata string, which the paper lists as future work."],"forward_implications":["A benchmark with 160 databases across many domains now exists for query-to-database routing, built from two standard text-to-SQL datasets.","Routing accuracy falls as the number of candidate databases grows: on the Spider-Route test split, task-specific embeddings drop from 91.78% Recall@1 with 20 databases to 60.38% with 160 databases.","Databases that look similar confuse all tested routers; within-vertical Recall@1 is consistently lower than across-vertical Recall@1, so domain overlap is a primary difficulty.","Providing database-specific domain knowledge improves routing: on Bird-Route cross-domain, pre-trained embeddings' Recall@1 rises from 91.00% to 97.71% when retrieved domain statements are added.","Because Llama3's 8K context cannot hold even a few Bird-Route schemas, context-length limits, not reasoning alone, currently block direct LLM routing on realistic enterprise repositories."],"supporting_citations":[{"why":"Supplies the Spider questions, database schemas, and question-to-database mappings used to build the Spider-Route benchmark.","marker":"[18]"},{"why":"Supplies the Bird-SQL questions, schemas, evidence statements, and SQL queries used to build the Bird-Route benchmark.","marker":"[7]"},{"why":"Supplies the Sentence-BERT base model and cosine-similarity scoring used by both the pre-trained and fine-tuned embedding baselines.","marker":"[12]"},{"why":"Supplies the open-source language model family, including the instruction-tuned 70B model, used for the zero-shot Llama3 ranking baseline.","marker":"[16]"}],"fun_headline_variants":["LLMs beat embeddings for DB query routing but hit context walls","Routing queries to databases gets harder with more sources and overlap","DB routing: Scale and domain similarity trip up both LLMs and embeddings","70B LLMs top embeddings for database routing until context runs out"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every benchmark question is assumed to be answerable by exactly one database, and that database is inherited from the original text-to-SQL dataset rather than verified by actually answering the question; if many questions are naturally answerable by several databases, the accuracy numbers are not true routing errors.","fun_headline_variants_meta":{"raw":{"variants":["LLMs beat embeddings for DB query routing but hit context walls","Routing queries to databases gets harder with more sources and overlap","DB routing: Scale and domain similarity trip up both LLMs and embeddings","70B LLMs top embeddings for database routing until context runs out"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000206,"raw_usage":{"total_tokens":1416,"prompt_tokens":986,"completion_tokens":430,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":602,"completion_tokens_details":{"reasoning_tokens":357}},"tokens_in":602,"tokens_out":430,"duration_ms":4358,"temperature":1.0,"reasoning_tokens":357,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T13:36:25.928364+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the questions that at least one method routes incorrectly and check answerability directly: execute the ground-truth SQL on each plausible candidate database, or have annotators say which databases can answer the question. If a large share of the errors are questions answerable by two or more databases, as the paper's own 'Find the number of albums' example suggests, then the reported Recall@1 understates routing quality and the benchmark labels need revision.","supporting_citations":[],"review_version":1}