{"id":"6d71e98f-c71c-47db-aa7b-8c3913735985","arxiv_id":"2508.19807","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"LLM-based synthetic SQL generation can train a learned cost model with fewer, more diverse queries than mechanical generation, though the measured accuracy gains are small and the comparison is not matched by training size.","lead":"The paper uses LLM-powered synthetic SQL query generation to create training data for learned cost models, claiming this works with 45% fewer queries than mechanical generation. If true, it could speed up bootstrapping of cost-based query routing in cloud databases without waiting for real workload history.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 45% fewer queries claim rests on one unmatched comparison (2,200 SDG vs 4,000 mechanical) with no repeated trials; the 0.02 qmedian and 15-min routing differences are within plausible run-to-run and training-size noise.","rationale":"I read the paper as a preliminary extended abstract: the contribution is an SDG pipeline and a first measurement that it can help train learned cost models. For the headline claim to hold, the accuracy/routing advantage must be attributable to SDG rather than to training-size, seed noise, or test-set composition. The experiment conflates generation method with dataset size (2,200 vs 4,000) and has no repeated trials, so the headline numbers have no estimated variance. I do not consider this a fatal flaw; the authors explicitly label the results as initial and the pipeline is grounded in their DiGiT work. But the central quantitative claim is not yet verified. The needed test is straightforward: matched sizes, multiple seeds, and report distributions. This is essentially the same concern the Reader raised, so the conditional verdict remains appropriate.","tokens_in":8500,"tokens_out":4748,"duration_ms":50811,"concrete_test":"Run a matched-size, repeated-trial comparison. For seeds 1-10, train ErrGNN on: (a) 2,200 SDG, (b) 2,200 mechanical, (c) 4,000 SDG, (d) 4,000 mechanical, keeping the same 1,000-query test set and engine provisioning. Report mean and standard deviation of qmedian, qmean, qp95, and total routing time for each condition, and disclose how the test queries were generated. If (a) does not beat (b) by more than the seed spread, the 45% claim is unsupported; if (c) does not beat (d), the apparent advantage is a training-size effect, not a property of SDG. If the test set is mechanical, repeat the comparison on an SDG-generated test set to check distribution bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central quantitative claim is not established by Section 5. ErrGNN+SDG is trained on 2,200 SDG queries and ErrGNN+Mech on 4,000 mechanical queries, so training size and generation method change simultaneously. The qmedian improvement (1.20 to 1.18) and qp95 improvement (2.38 to 2.37) are tiny; the larger qmean drop (1.41 to 1.34) is driven by a few slow queries and is reported without error bars or repeated seeds. The routing result (165 to 150 min on 1,000 test queries) is a single observation with no confidence interval or significance test. Table 2 in the appendix reports only per-engine qmean, again without variance. The composition of the 1,000-query test set is not stated, so it is unknown whether the test distribution matches the mechanical training distribution, the SDG distribution, or a third workload. Because the abstract claims a 45% query reduction, the experiment must show that SDG reaches the same or better accuracy with 45% fewer queries at matched sizes and across repeated runs. The current design cannot rule out that the observed differences are due to random seeds, optimizer nondeterminism, or simply having 2,200 instead of 4,000 training examples.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This extended abstract proposes using LLM-based synthetic data generation (SDG) to create diverse SQL training queries for learned cost models (LCMs). The authors describe a pipeline based on the DiGiT framework: schema preprocessing, subschema creation, few-shot prompting of a Granite instruct model, validation/filtering, and Apache Calcite-based coverage analysis. They evaluate the LCM from [27] on the TPC-H schema, comparing ErrGNN+SDG trained on 2,200 SDG-generated queries with ErrGNN+Mech trained on 4,000 mechanically generated queries. They report a slight Q-error improvement (qmedian 1.20 to 1.18, qmean 1.41 to 1.34, qp95 2.38 to 2.37) and a 165- to 150-minute routing improvement on 1,000 test queries, concluding that SDG enables training with 45% fewer queries.","tokens_in":8831,"tokens_out":4113,"duration_ms":45982,"significance":"If validated, the result is practically significant: LLM-based SDG could reduce the bootstrapping cost of learned cost models by replacing large volumes of mechanical or historical training queries with fewer, more diverse synthetic ones. The paper has concrete strengths: it ships an open-source artifact, describes a real pipeline, and uses Calcite-based coverage analysis to characterize query diversity. The comparison is also not circular: the same LCM architecture and training procedure are used for both arms, so only the training-data source and size differ. However, the central quantitative claim rests on a small, single-run experiment in which training size and generation method are confounded. The significance is therefore conditional on the evaluation being strengthened.","major_comments":[{"comment":"The headline claim of '45% fewer queries' is not established because the two arms differ in both generation method and training set size: ErrGNN+SDG uses 2,200 queries while ErrGNN+Mech uses 4,000. The reported Q-error and routing differences could be due to the size difference alone. To support the abstract's claim, the authors should train on matched sizes (e.g., 2,200 and 4,000 for both generators) or provide a learning-curve ablation over training sizes so the reader can see how much of the improvement comes from SDG rather than fewer or more samples.","section":"§5, Table 1"},{"comment":"All results are from a single run with no error bars, repeated seeds, or significance tests. The differences are small (qmedian 1.20 vs. 1.18; qp95 2.38 vs. 2.37) and the routing result (165 vs. 150 minutes) is a single observation on one 1,000-query test set. These differences are within plausible run-to-run or optimizer-nondeterminism noise. The qmean drop is larger (1.41 to 1.34), but without per-query distributions or confidence intervals it is unclear whether it reflects broad improvement or a few long-tail queries. I recommend reporting repeated runs (at least 5--10 seeds), confidence intervals, and/or per-query Q-error distributions.","section":"§5, Table 1 and §A.3, Table 2"},{"comment":"The composition of the 1,000-query test set is not described. If the test queries were generated by the same SDG pipeline, the comparison may favor the SDG-trained model; if they were generated mechanically, it may favor the mechanical baseline. The authors should state how the test set was produced, how it relates to the training distributions, and how representative it is of the intended workload. This is load-bearing for interpreting both the accuracy and the routing results.","section":"§5, 'Bootstrapping Learned Cost Models'"}],"minor_comments":[{"comment":"A few reference typos should be fixed: [19] lists 'Lerner Lerner', [24] lists 'Pedreira Pedreira' and 'Binnig Binnig' appears in [4]; [25] has '!rst' instead of 'first'.","section":"References"},{"comment":"The qmedian formula is defined in-line, but qmean and qp95 are used without definitions. Add one sentence defining them so the tables are self-contained.","section":"§5"},{"comment":"There is a duplicated phrase: 'the LCM trained when trained with the 2200 queries' should read 'the LCM trained with the 2200 queries'.","section":"§5"},{"comment":"Table 2 reports only qmean per engine/provisioning. For consistency with Table 1, reporting qmedian and qp95 as well would make the per-engine picture clearer.","section":"§A.3, Table 2"},{"comment":"Figure 1 is referenced but appears only as a pipeline diagram without detailed caption text; a short caption explaining the artifacts at each stage would help readers.","section":"§4, Figure 1"}],"recommendation":"major_revision","confidential_remarks":"This is a workshop-style extended abstract with a useful artifact and a clear motivation. The core problem is that the evidence in Section 5 is too weak for the strength of the abstract's claim: the comparison is size-confounded, unrepeated, and the test set is undefined. These issues are fixable within the manuscript's scope (e.g., by adding a matched-size ablation and repeated runs), so I recommend major revision rather than rejection. If the authors cannot add such experiments, the claims should be softened to 'initial, indicative results' and the 45% framing should be removed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a readable extended abstract with a genuinely useful idea—use an LLM-based synthetic data pipeline, with Calcite-driven coverage steering, to bootstrap training data for learned cost models. The pipeline description is concrete and builds on released tooling (DiGiT), and the diversity analysis (Figures 3–6) is a reasonable preliminary look at what the generator produces. The authors are honestly labeling these as initial results.\n\nWhat's actually new: prior synthetic workload generators for LCM training are mechanical or historical; using an LLM with a coverage loop to cut required training queries is not in the cited literature. That's a real contribution to the subfield.\n\nWhere the paper is soft: the headline claim—45% fewer queries—is supported by one table and one routing number. Training sets are unmatched (2,200 SDG vs. 4,000 mechanical), so size and generation method change together. The reported improvements (qmedian 1.20→1.18, qp95 2.38→2.37) are tiny and could easily be noise; the qmean drop (1.41→1.34) is driven by a few slow queries. There are no error bars, no repeated seeds, and no significance test. The test set composition isn't stated, so we don't know whether it matches the mechanical distribution, the SDG distribution, or something else. I agree with the stress-test note: the central claim is not established by this experiment. On the other hand, the direction is plausible, and I don't see a fatal flaw in the approach itself—just insufficient evidence.\n\nOne thing the paper does well: it gives enough detail about the prompt settings and the coverage loop that someone could reproduce or extend the pipeline. The artifact link helps.\n\nWho this is for: people working on learned cost models or workload generation for database tuning. It's a workshop-level contribution that suggests a path, not a definitive result.\n\nRecommendation: I would send this to peer review for a workshop, but with clear revision expectations—matched training sizes, repeated runs, significance testing, and a stated test distribution. The idea deserves referee time; the current evaluation doesn't yet back the abstract's strong claim.","headline":"A plausible LLM-based synthetic query generation pipeline, but the central 45% claim rests on one unmatched, unrepeated comparison.","tokens_in":9332,"tokens_out":1900,"would_cite":false,"duration_ms":19381,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that an LLM-driven synthetic SQL generator can train a learned cost model more accurately using 45% fewer queries than a mechanical generator, and can cut cross-engine routing time by 10%.","keywords":["learned cost models","synthetic data generation","SQL query generation","large language models","query routing","workload diversity","TC-H benchmark","bootstrapping"],"falsifier":"Run the same 1,000-query test with several independently generated synthetic and mechanical training sets at matched sizes, and compare the distributions of median Q-error and total routing time; if the synthetic advantage does not persist across seeds, or vanishes when sizes are equalized, the 45%-fewer-queries claim is not supported.","tokens_in":8379,"feed_emoji":"🤖","tokens_out":8317,"duration_ms":84906,"temperature":0.7,"pith_summary":"Learned cost models predict how long a query will take on a given database engine, but they are only as good as the workload they train on. Real query histories take months to collect, and mechanical generators produce large but narrow sets of queries. This paper tries to solve that cold-start problem with a synthetic-data pipeline that uses a large language model to generate diverse, schema-specific SQL queries, validated and steered by coverage metrics. On a standard decision-support benchmark schema, training the authors' learned cost model on 2,200 such queries beat training it on 4,000 mechanically generated queries, improving median prediction error and cutting the time to route 1,000 test queries from 165 to 150 minutes. If the result holds, database deployments could bootstrap efficient cross-engine query routing from the schema alone, without waiting for historical workload data.","feed_headline":"LLM-made SQL queries beat mechanical training sets with 45% less data","feed_subtitle":"Synthetic queries cut prediction error and cross-engine routing time on 1,000 test queries.","key_machinery":"The load-bearing mechanism is the coverage-steered synthetic query generation loop. It starts by cutting the schema into 187 connected subschemas joined by foreign keys, so prompts are small and semantically coherent. A few-shot prompt shows the LLM CREATE statements and mechanically built example queries, optionally biased toward group by or order by clauses. Validators filter for syntax and relevance; then a SQL parser computes how often each table, column, operator, join, and sub-select appears, and the pipeline biases the next round toward uncovered areas. The other half of the machinery is the multi-predictor-head GNN cost model from the authors' earlier work, which turns the generated","core_discovery":"The paper reports that the distribution of queries used to train a learned cost model matters more than the sheer number, and that an LLM-based generator with coverage steering can produce a better distribution from scratch. The pipeline decomposes the schema into 187 foreign-key-connected subschemas, prompts an instruction-tuned model with CREATE statements and mechanically built seed examples, filters the output for syntactic correctness and semantic relevance, and parses accepted queries to count table, column, and operator coverage; any gaps trigger another generation round. The resulting 2,200-query training set yields a median Q-error of 1.18 versus 1.20 for the 4,000-query mechanical","pith_inferences":["The paper does not run a matched-size control: training on 2,200 mechanically generated queries would separate the effect of query diversity from the effect of dataset size, which the current comparison conflates.","The larger mean Q-error improvement (1.41 to 1.34) compared with the median suggests the benefit is concentrated in slow or otherwise hard-to-predict queries; reporting error separately per runtime bucket would test whether the even runtime distribution is the causal driver.","A natural extension the authors only gesture at is to close the loop using the learned model itself: let its prediction errors or uncertainty choose which queries to generate next, turning coverage metrics into an acquisition function.","The same pipeline could generalize to other cost-sensitive database tasks, such as learning join-order choices or physical plan selection, where diverse schema-bound queries are equally scarce."],"forward_implications":["If the result holds, a new database deployment can train a learned cost model immediately from its schema, eliminating the months of suboptimal bootstrapping operation that historical-query approaches require.","Training-data volume can shrink by 45%, meaning fewer queries have to be actually executed to get ground-truth labels, the expensive step in building these models.","Coverage steering means the generator can be biased toward under-represented SQL constructs, so training sets can be deliberately balanced rather than reflecting whatever queries happened to be logged.","The same diverse, validated query sets can serve other schema-dependent tasks such as stress testing and vulnerability testing, which also need realistic workloads."],"supporting_citations":[{"why":"Supplies the multi-predictor GNN learned cost model used for all accuracy and routing evaluations; its training setup defines the comparison.","marker":"[27]"},{"why":"Supplies the mechanical query generation technique used to create the 4,000-query baseline training set.","marker":"[12]"},{"why":"The open-source synthetic data generation framework the pipeline builds on, providing LLM-, template-, and component-based generators and validators.","marker":"[8]"},{"why":"The SQL parser used in the coverage step to count table, column, and operator references and steer generation toward gaps.","marker":"[10]"},{"why":"The benchmark schema and workload on which all experiments are run.","marker":"[6]"},{"why":"The self-instruct loop that justifies iteratively generating and validating new queries to enlarge the dataset.","marker":"[31]"},{"why":"The instruction-tuned LLM used to generate the synthetic SQL queries from subschema prompts.","marker":"[29]"}],"fun_headline_variants":["Coverage-steered synthetic SQL trains models with 45% less data","Bootstrapping cost models with LLM-generated SQL: 45% less data","LLM-synthesized SQL beats mechanical sets, 45% less data","Better query distributions beat more queries: LLM-crafted SQL","LLM-generated SQL query distribution trims 45% off training data"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The result rests on a single comparison run with different training-set sizes: if the 1.20-to-1.18 Q-error gap and the 165-to-150-minute routing gap are within run-to-run noise, the claimed advantage of synthetic queries disappears.","fun_headline_variants_meta":{"raw":{"variants":["Coverage-steered synthetic SQL trains models with 45% less data","Bootstrapping cost models with LLM-generated SQL: 45% less data","LLM-synthesized SQL beats mechanical sets, 45% less data","Better query distributions beat more queries: LLM-crafted SQL","LLM-generated SQL query distribution trims 45% off training data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001507,"raw_usage":{"total_tokens":5822,"prompt_tokens":630,"completion_tokens":5192,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":374,"completion_tokens_details":{"reasoning_tokens":5105}},"tokens_in":374,"tokens_out":5192,"duration_ms":40324,"temperature":1.0,"reasoning_tokens":5105,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T15:26:55.701597+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same 1,000-query test with several independently generated synthetic and mechanical training sets at matched sizes, and compare the distributions of median Q-error and total routing time; if the synthetic advantage does not persist across seeds, or vanishes when sizes are equalized, the 45%-fewer-queries claim is not supported.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the mechanical query generation technique used to create the 4,000-query baseline training set."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The open-source synthetic data generation framework the pipeline builds on, providing LLM-, template-, and component-based generators and validators."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The SQL parser used in the coverage step to count table, column, and operator references and steer generation toward gaps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The benchmark schema and workload on which all experiments are run."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The instruction-tuned LLM used to generate the synthetic SQL queries from subschema prompts."}],"review_version":1}