{"id":"aed1def0-b6c6-4903-8656-a99bf3028542","arxiv_id":"2505.23039","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Adding workload-derived hints and tailored retrieval from past SQL queries improves NL2SQL execution accuracy and reduces prompt size on three benchmarks.","lead":"TailorSQL mines a database's past SQL query logs to generate hints, such as common join paths and filter patterns, and feeds them into an LLM prompt when translating natural language to SQL. On three NL2SQL benchmarks it reports higher execution accuracy and matched accuracy with fewer prompt tokens and lower latency.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The evaluation removes PK/FK constraints from the schema while keeping join-path hints from the query log, so the reported accuracy gains may largely reflect a handicapped baseline rather than real workload specialization.","rationale":"I looked for the condition that must hold for the central claim to be true: past query logs must contain transferable signal that schema-only RAG misses. The paper's evaluation is designed to maximize that signal by deleting PK/FK constraints from the schema while retaining hints derived from query logs. The reader's stated weakest assumption (synthetic question realism) is real but secondary: if synthetic questions were poor, the embedding and context-allocation optimization would degrade, yet the method still outperforms; that suggests robustness, not fatal weakness. The PK/FK removal, by contrast, directly inflates the comparison: the schema-only baselines are deprived of the very join-path information that the hints supply. The paper's rationale for this setup is plausible for some deployments but is not validated, and no experiment measures performance when FK metadata is present. The abstention policy and ablations are valuable, and the system is well-engineered, but the headline quantitative claim is conditional on this evaluation choice. A single rerun with intact FKs would resolve whether the benefit is real specialization or an artifact of the modified schema.","tokens_in":19135,"tokens_out":5227,"duration_ms":55844,"concrete_test":"Rerun the end-to-end evaluation (Section 8.3) on Bird-Union and Spider-Union with PK/FK constraints restored in the schema documents (i.e., the original CREATE TABLE statements from BIRD/Spider), keeping TailorSQL and all baselines unchanged. Compare execution accuracy and prompt-token counts. If the gap between TailorSQL and the SBERT baseline at its best prompt size narrows by more than 5 absolute points on either benchmark, the headline improvement is substantially attributable to the removed schema constraints rather than to workload specialization.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central accuracy result (Section 8.3, Fig. 5) is obtained under a modified schema: footnote 2 in Section 8.1 states 'We exclude primary key-foreign key (PK-FK) information from the schema.' Meanwhile, Section 8.2.2 keeps join-path hints extracted from the query log. Join-path hints directly encode the join relationships that PK/FK metadata would provide, and group-by hints can substitute for missing primary keys. The schema-only baselines (SBERT, BM25) therefore compete without information that is routinely available in real database catalogs. The paper justifies this as simulating 'real-world scenarios where this data is typically unavailable,' but it provides no evidence about how common that scenario is. In a database with intact FK constraints, the LLM already sees the join path in the schema document, so TailorSQL's most valuable hint class becomes redundant; the measured 12.5%-22.7% accuracy advantage could shrink to a much smaller margin. This does not make the method internally inconsistent, but it means the central claim that 'past queries provide information not apparent from the schema alone' is only demonstrated in a setting where that information was artificially removed from the schema.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TailorSQL, a retrieval-augmented NL2SQL system that specializes to a database's past query workload. Offline, it builds schema documents and query-hint documents (join-path, filter, and group-by hints) from historical SQL logs, then computes document embeddings as a weighted sum of raw SBERT embeddings and proxy embeddings derived from co-occurring documents, past SQL queries, and LLM-generated synthetic questions; the weights are optimized on the logged workload. It also uses Bayesian optimization to allocate the LLM prompt context among document classes. At runtime, TailorSQL retrieves documents per class, builds a prompt, and generates SQL; a bandit-based abstention policy decides when to fall back to a generic schema-only pipeline. The evaluation on three benchmark-derived \"Union\" databases reports 12.5%, 10.9%, and 22.7% higher execution accuracy than the best baseline, with substantially fewer prompt tokens at matched accuracy, and shows that TailorSQL can improve existing systems such as DIN-SQL and MAC-SQL.","tokens_in":19321,"tokens_out":6401,"duration_ms":68549,"significance":"If the reported results hold, the core idea is valuable: past query logs contain transferable semantic information, such as common join paths and column semantics, that schema-only RAG misses. The paper's system design is coherent, the ablations isolate the main components, and using the same LLM across all compared systems is a good control. The abstention-policy idea is a sensible safeguard against workload drift. However, the headline quantitative claims currently rest on an evaluation setup that removes standard schema metadata (PK-FK constraints) while preserving log-derived join hints, and on a random split that makes the test workload similar to the logged workload by construction. The synthetic questions used as the training signal are not validated. These are load-bearing concerns for the central claim that workload specialization, rather than restoration of routine metadata or benchmark distribution overlap, drives the reported gains.","major_comments":[{"comment":"The experimental setup excludes primary-key/foreign-key (PK-FK) information from the schema, while TailorSQL's hint documents include join-path hints extracted from the past query log. In real database catalogs, FK constraints are routinely available; the SBERT and BM25 baselines are therefore deprived of information that TailorSQL effectively restores through its join-path hints, and group-by hints can similarly substitute for missing primary keys. The 12.5%, 10.9%, and 22.7% accuracy advantages over the best baseline in §8.3 may thus reflect restoration of standard catalog metadata rather than novel information from past queries. To support the paper's central claim, the authors should rerun the comparison with PK-FK constraints included in the schema documents for all systems, or at least report baseline accuracy with intact PK-FK metadata.","section":"§8.1, footnote 2; §8.2.2"},{"comment":"The headline end-to-end accuracy is reported under the Random Split, where the logged and test halves are drawn from the same benchmark distribution. The paper itself calls this setup 'ideal for showcasing the benefits of TailorSQL' (§8.1), but a real deployment may face a log that only partially overlaps the future workload. The Disjoint Split is used only to evaluate the abstention policy (§8.5), not to report TailorSQL's end-to-end accuracy against the baselines. Without an evaluation under a less favorable split (for example, BIRD train versus dev, or a temporal split), the claim that TailorSQL specializes to real workloads rather than to the benchmark distribution is not established.","section":"§8.1, §8.3, §8.5"},{"comment":"The embedding weights and the context allocation are optimized on synthetic questions generated by an LLM from the logged SQL queries, but the paper does not validate the fidelity of these synthetic questions to real user phrasing. There is no report of the generation prompt, the number of synthetic questions, or any comparison of their distribution with real user questions. If the synthetic questions are systematically more aligned with the logged SQL than real user paraphrases are, the learned weights in Eq. (2) and the Bayesian-optimization objective in §6 could be overfit to the synthetic workload, inflating the reported accuracy. Please add a validation study of synthetic-question quality (for example, human judgments, or an ablation that uses real logged questions instead of synthetic ones) and disclose the generation details.","section":"§5.1, Eq. (2), §6"}],"minor_comments":[{"comment":"The term 'query' is used both for natural-language user questions and for SQL queries, which can be confusing in Section 5 where 'past queries' are SQL and 'user questions' are natural language; consistent terminology would improve readability.","section":"§1, §2"},{"comment":"The construction of Bird-Union and Spider-Union (combining all dev-set databases into one) is not described in detail; it is unclear whether same-named tables or columns across original databases were renamed and how this affects PK-FK relationships, so the schema-removal decision is hard to reproduce.","section":"§8.1"},{"comment":"The paper reports median values over five runs but does not report variance or significance for the end-to-end accuracy differences; given the modest dataset sizes (300-1200 questions) and possible split variance, confidence intervals would strengthen the comparisons.","section":"§8.3"},{"comment":"The epsilon value and sliding-window length used in the bandit-based abstention policy are not reported, which makes the workload-drift experiment difficult to reproduce.","section":"§7, §8.5"},{"comment":"The end-to-end ablation is reported only on Bird-Union; reporting the same ablations on at least one other benchmark would strengthen the claim that each component is necessary.","section":"Table 1"},{"comment":"The paper does not report the number of hint documents generated per benchmark or the distribution across the three hint types, which would help assess the relative contribution of each hint type to the accuracy gains.","section":"§8.2.2"}],"recommendation":"major_revision","confidential_remarks":"The PK-FK removal and the Random Split evaluation are the main risks to the paper's central claim. If the authors can show that the accuracy advantage persists with PK-FK constraints included in the schema for all systems and under a more realistic split, I would support acceptance. The topic is well within the scope of a database systems journal, and the system is described in sufficient detail that the requested experiments are feasible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Workload-aware NL2SQL is a real idea, and TailorSQL is a competent engineering of it: mine query logs for hints, tailor embeddings via proxy averages, allocate context per document class, and fall back with a bandit. The paper's ablations isolate each component, and the latency numbers (2–15x fewer tokens at matched accuracy) are the strongest evidence. I think the reader's conditional verdict is about right, though the stress-test concern about PK-FK removal is the one that should be front and center in any review.\n\nWhat's new: prior RAG NL2SQL retrieves schema only; TailorSQL adds hint documents from logs and specializes retrieval to the workload. That is a legitimate gap, and the combination of weighted proxy embeddings and per-class context allocation is non-obvious. The DIN-SQL/MAC-SQL integration (Tables 2-3) is a nice complementarity check, not just a leaderboard cherry-pick.\n\nWhere I'd push back: the evaluation's favorable setup. Footnote 2 removes PK-FK constraints from the schema, while join-path hints from logs remain. In any real database with FK metadata, the LLM already sees the join path; the headline 12.5-22.7% gain is therefore measured against baselines deprived of information that is routinely available. That doesn't kill the method—filter-format and group-by hints are genuinely schema-independent—but it means the main accuracy claim is likely an upper bound on the real-world benefit. Also, Random Split is the default; test questions mirror the log distribution, which is the best case for specialization. The drift experiment shows the bandit recovers, but it doesn't report what TailorSQL's accuracy is under Disjoint Split relative to baselines.\n\nThe synthetic questions used to fit embeddings and context allocation are never validated. The paper treats them as a faithful proxy for real user phrasing; that's a plausible assumption but unexamined. A quick manual comparison or a sensitivity check would have helped.\n\nNo code or data is released, which in this area makes replication hard. The paper is well-written and honest about its choices; it just doesn't address the PK-FK asymmetry.\n\nWho it's for: DB/NL2SQL researchers working on RAG and retrieval. It deserves a serious referee; the core idea is sound and the engineering is solid, but the evaluation needs to be rebalanced (either keep PK-FK in the schema or justify their absence better) before the accuracy claims are publishable as stated.","headline":"TailorSQL is a competent, genuinely novel workload-aware NL2SQL system, but its headline accuracy gains are measured against baselines deprived of PK-FK metadata that real databases usually have.","tokens_in":19918,"tokens_out":2333,"would_cite":true,"duration_ms":23221,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Workload-specific retrieval built from past SQL logs improves NL2SQL execution accuracy by up to 2× and uses up to 15× fewer prompt tokens at matched accuracy.","keywords":["NL2SQL","text-to-SQL","retrieval-augmented generation","query workload","workload specialization","document retrieval","execution accuracy","abstention policy"],"falsifier":"Run TailorSQL on a production database with a time-ordered split: train the specialized embeddings and context allocation on the first several months of real query logs and evaluate on the next several months of real user questions, comparing against schema-only retrieval with the same LLM. If execution accuracy on genuinely future questions does not beat the baseline, the central claim that past logs transfer to future questions fails.","tokens_in":18907,"feed_emoji":"🗄️","tokens_out":9360,"duration_ms":78325,"temperature":0.7,"pith_summary":"TailorSQL tries to establish that the SQL queries a database has already executed are a reusable resource for NL2SQL, not just the schema itself. It claims that by mining past logs for common join paths, filter expressions, and group-by clauses, and by using those logs to shape both document embeddings and per-class prompt-token budgets, a retrieval-augmented text-to-SQL pipeline can generate more accurate SQL while consuming less context. On three benchmarks the paper reports execution-accuracy gains of 12.5, 10.9, and 22.7 percentage points over the strongest retrieval baseline, and matched accuracy with 1.8×, 15×, and 1.9× fewer prompt tokens. If this is right, existing databases carry transferable semantic signal—what cryptic tables mean, which joins are conventional, how column values are formatted—that schema-only retrieval misses.","feed_headline":"Past SQL logs lift text-to-SQL accuracy up to 2x","feed_subtitle":"Mining past queries for join paths and filters improves accuracy and cuts prompt size by up to 15x.","key_machinery":"The central object is the workload-tailored document embedding $E_{\\mathrm{doc}} = w_1 E_{\\mathrm{raw}} + w_2 E_{\\mathrm{co\\text{-}occur}} + w_3 E_{\\mathrm{SQL}} + w_4 E_{\\mathrm{synthQ}}$, with weights optimized once per workload by gradient descent on a cosine-loss objective over a synthetic question workload generated from past queries. It is paired with a context allocator that uses Bayesian optimization to split the prompt token budget across table, column, and hint document classes, and a bandit-based abstention policy that switches to a generic schema-only pipeline when user feedback indicates the specialized pipeline no longer matches the workload.","core_discovery":"The paper's central discovery is that the historical query workload contains reusable semantic structure for NL2SQL, and that this structure can be injected into a retrieval-augmented generation pipeline at three points: document content, document embeddings, and retrieval budgets. Concretely, past queries are broken into clause-level hint documents covering join paths, filters, and group-by conditions; each document's embedding is reweighted as a blend of its raw embedding, embeddings of co-occurring documents, embeddings of the SQL queries that used it, and embeddings of synthetic user questions generated from those queries; and a Bayesian optimizer chooses how many prompt tokens to give to table, column, and hint documents. Evaluated as execution accuracy, TailorSQL reports consistent gains over workload-agnostic retrieval, with the largest gains on the benchmark with the most cryptic table names and complex nested queries.","pith_inferences":["Editorial inference: TailorSQL's Random Split evaluation likely overstates production gains; a time-ordered split, where training logs precede test questions, would be a stricter test of whether past logs transfer to genuinely future questions.","Editorial inference: the realism of LLM-generated synthetic questions is untested; having human annotators rate or rewrite a sample and measuring how retrieval accuracy changes would expose how much the method depends on that link.","Editorial inference: the same embedding-blend recipe could transfer to other retrieval-augmented settings where historical artifacts encode conventions—e.g., code generation from repository commit logs—but this is outside the paper's evidence.","Editorial inference: because fine-tuned embeddings achieved higher recall than TailorSQL's weighted blend, a hybrid that periodically fine-tunes a small per-workload adapter could combine TailorSQL's low offline cost with higher retrieval recall."],"forward_implications":["Deployed NL2SQL systems on existing databases can improve accuracy without new data collection: the query logs are already there.","At matched execution accuracy, the prompt can be 1.8×–15× smaller, which directly lowers per-query LLM latency and cost.","Workload specialization composes with existing reasoning-based NL2SQL pipelines, so it can be added on top of current state-of-the-art systems rather than replacing them.","The abstention policy gives a safe deployment story: when the workload shifts, the system can fall back to generic retrieval instead of silently degrading.","Hint documents for joins, filters, and group-bys carry most of the value, so the scheme is extensible to other SQL clause types."],"supporting_citations":[{"why":"Supplies the BIRD benchmark used to build the Bird-Union combined database and the chemistry-database join-path example.","marker":"[13]"},{"why":"Supplies the Spider benchmark used to build the Spider-Union evaluation set.","marker":"[37]"},{"why":"Supplies the FIBEN benchmark whose cryptic names and nested queries produce TailorSQL's largest accuracy gains.","marker":"[25]"},{"why":"Provides Sentence-BERT, the pretrained embedding model whose raw embeddings TailorSQL blends into tailored ones.","marker":"[22]"},{"why":"Provides the specific all-MiniLM-L6-v2 embedding model used for question and document embeddings.","marker":"[4]"},{"why":"Defines retrieval-augmented generation, the pipeline framework that TailorSQL specializes.","marker":"[11]"},{"why":"DIN-SQL, a state-of-the-art NL2SQL system that TailorSQL's prompt is combined with to show complementarity.","marker":"[20]"},{"why":"MAC-SQL, the second state-of-the-art NL2SQL system used in the combination experiment.","marker":"[32]"},{"why":"Evidence that LLMs are distracted by irrelevant context, motivating the context allocator's precision-aware token budgets.","marker":"[26]"},{"why":"Lost-in-the-middle evidence that a high concentration of irrelevant documents hurts LLM accuracy, justifying Bayesian optimization over allocation.","marker":"[14]"}],"fun_headline_variants":["Mining past SQL queries boosts text-to-SQL accuracy 2x","Past queries teach NL2SQL: accuracy up 2x, prompt 15x smaller","TailorSQL learns from your SQL history to improve NL2SQL","Workload-aware NL2SQL: past queries lift accuracy 2x","Your SQL logs can boost text-to-SQL accuracy by 2x"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that LLM-generated synthetic questions derived from past SQL queries faithfully resemble the natural-language questions real users will ask later; if those synthetic questions are unrepresentative, the tailored embeddings and context allocation are tuned to the wrong signal.","fun_headline_variants_meta":{"raw":{"variants":["Mining past SQL queries boosts text-to-SQL accuracy 2x","Past queries teach NL2SQL: accuracy up 2x, prompt 15x smaller","TailorSQL learns from your SQL history to improve NL2SQL","Workload-aware NL2SQL: past queries lift accuracy 2x","Your SQL logs can boost text-to-SQL accuracy by 2x"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000174,"raw_usage":{"total_tokens":1267,"prompt_tokens":918,"completion_tokens":349,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":534,"completion_tokens_details":{"reasoning_tokens":250}},"tokens_in":534,"tokens_out":349,"duration_ms":3769,"temperature":1.0,"reasoning_tokens":250,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:55:22.597981+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run TailorSQL on a production database with a time-ordered split: train the specialized embeddings and context allocation on the first several months of real query logs and evaluate on the next several months of real user questions, comparing against schema-only retrieval with the same LLM. If execution accuracy on genuinely future questions does not beat the baseline, the central claim that past logs transfer to future questions fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the FIBEN benchmark whose cryptic names and nested queries produce TailorSQL's largest accuracy gains."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides Sentence-BERT, the pretrained embedding model whose raw embeddings TailorSQL blends into tailored ones."},{"cited_title":"https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2","cited_arxiv_id":null,"evidence_quote":"Provides the specific all-MiniLM-L6-v2 embedding model used for question and document embeddings."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Evidence that LLMs are distracted by irrelevant context, motivating the context allocator's precision-aware token budgets."}],"review_version":1}