{"id":"a8e8e8b0-503b-4f4a-8dee-05211cad2cbf","arxiv_id":"2504.14837","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A three-team multi-agent framework (GPT-4o exploration, lightweight local expansion, adaptive management) generates over 300,000 diverse SQL queries for under $200 and improves downstream text-to-SQL and clustering baselines.","lead":"This paper describes SQL-Factory, a multi-agent system that first uses a powerful language model to invent diverse SQL queries and then uses a cheaper local model to expand them into hundreds of thousands of variants. The authors report generating over 300,000 queries on four database benchmarks for under $200 and say these corpora improve downstream text-to-SQL and clustering models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified","rationale":"The stress-test pass found the paper's strongest claim to be an empirical one: that SQL-Factory generates large, diverse, low-cost SQL corpora that improve downstream models. None of the identified weaknesses are fatal. The cost claim ($200 for 300,000+ queries) is a direct output of the pipeline accounting; the reader's alternative calculation of roughly $1,000 if all queries used GPT-4o actually supports the framework's cost-efficiency value proposition. The diversity claim is supported by both a proposed hybrid similarity metric and an external Vendi Score computed with two encoders, and the paper includes a comparison with single-agent baselines (Table 3) showing that agent decomposition, not just model strength, drives the diversity gains. The schema-aware allocation analysis (Figures 7-8) provides concrete evidence for the table-selection component. The main residual concern, matching the reader's weakest assumption, is contamination of the downstream Text-to-SQL training pairs: GPT-4o may have memorized Spider/BIRD schemas and queries, and the SQL-to-NL pipeline retrieves similar examples from the official training set as in-context demonstrations, which could bias the generated pairs toward test-like content. This is a legitimate risk that should be tested, but it is not a demonstrated error, and the paper's artifact release enables such testing. The other reader concerns—unequal data volume between official and generated training sets, reuse of the filtering metric as an evaluation metric, and absent variance estimates—are methodological limitations that warrant caution in interpreting the magnitude of gains but do not undermine the central claim that the framework produces useful, low-cost, diverse SQL corpora. The verdict of CONDITIONAL with moderate confidence therefore remains appropriate, as it already reflects the need for external validation of the contamination and fairness assumptions.","tokens_in":20266,"tokens_out":1580,"duration_ms":12573,"concrete_test":"Re-run the Text-to-SQL fine-tuning ablation (Table 4) with the SQL-Factory-generated pairs for Spider and BIRD examined for sequence-level overlap with the official train and test sets; if the generated pairs contain exact or near-duplicate test queries from those benchmarks, the reported accuracy gains would need to be re-evaluated for contamination rather than data-quality effects.","verdict_should_be":"UNCHANGED","load_bearing_attack":"No single load-bearing concern emerged from this pass beyond the concerns already raised in the reader's verdict, and I do not believe those concerns overturn the paper's central claim. The three empirical pillars—(1) over 300,000 executable SQL queries generated across four benchmarks for under $200 in API cost, (2) higher diversity than SQLSmith, LearnedSQLGen, and OmniSQL on a 10,000-query TPC-DS comparison, and (3) downstream Text-to-SQL and query-clustering gains—are each supported by explicit experimental tables and ablations (Tables 1-5). The reader's weakest assumption about GPT-4o memorization of Spider/BIRD is a genuine risk, since any LLM-based data-generation pipeline for benchmark-derived schemas could leak test-like content into the generated training pairs; however, the paper's open-source artifact release (https://github.com/LJHzju/SQL-Factory) makes this assumption checkable, and under the no-ad-hominem and good-faith reading rules this is a verification gap rather than a demonstrated flaw. The confounds noted by the reader (unequal data volume, reuse of the hybrid similarity filter as an evaluation metric, missing variance estimates) are real but affect only the strength of the comparison, not the existence of the measured gains.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SQL-Factory, a multi-agent framework for synthesizing large volumes of executable SQL queries from a database schema. The Generation Team (backed by GPT-4o) explores novel query structures, the Expansion Team (backed by Qwen2.5-Coder-14B) scales promising patterns from selected seeds, and the Management Team filters queries by executability and a hybrid token/AST/embedding similarity metric while scheduling between exploration and exploitation. The authors report generating over 300,000 queries across TPC-DS, IMDB, Spider, and BIRD for under $200 in API cost, higher diversity than SQLSmith, LearnedSQLGen, and OmniSQL, and improved downstream performance on Text-to-SQL and SQL query clustering after fine-tuning on the generated data.","tokens_in":20490,"tokens_out":7324,"duration_ms":66843,"significance":"If the claims hold, SQL-Factory is a practically useful contribution: it converts a modest API budget into a large corpus of executable SQL, and the downstream results suggest that such corpora can rival or supplement manually curated Text-to-SQL training sets. The paper ships an open-source artifact, reports concrete cost and throughput numbers, and uses Vendi Scores with two independent encoders as diversity evidence, which is a strength because that evidence does not depend on the framework's own similarity filter. The main weaknesses are experimental-control issues in the downstream evaluation, specifically unequal training volumes and the absence of a decontamination check, rather than demonstrated failures of the generation method itself.","major_comments":[{"comment":"The comparison between '+Official-Data' and '+SQL-Factory' is confounded by training-set volume. The paper does not report the number of <NL, SQL> pairs used in each fine-tuning setting, and the official Spider and BIRD training sets are substantially smaller than the generated corpora (130,000 and 160,000 SQL queries before NL conversion). The claim that generated data 'rivals or surpasses expert-curated sets' therefore requires an equal-volume control, for example fine-tuning on a random subset of SQL-Factory pairs matched to the size of the official training set, or reporting the number of retained synthetic pairs after the semantic-consistency filter and adding an official-plus-synthetic condition. Without such a control, the observed gains in Table 4 could be driven largely by the amount of training data rather than by the quality of SQL-Factory's output.","section":"§5.3, Table 4"},{"comment":"The downstream Text-to-SQL evaluation lacks a decontamination check. The Generation Agent uses GPT-4o on Spider and BIRD schemas, and the SQL-to-NL pipeline retrieves similar examples from the official training sets of those benchmarks; Spider and BIRD are public benchmarks whose test queries and schemas may have appeared in LLM pretraining data. Because the central claim that SQL-Factory improves Text-to-SQL accuracy depends on the generated pairs not leaking test-like content, the paper should report an overlap or membership analysis (for instance, n-gram overlap between generated SQL/NL and the test sets, or an experiment that excludes schemas appearing in the official splits) or explicitly justify why contamination cannot occur.","section":"§5.1.4, §5.3.1"},{"comment":"The hybrid-similarity row of Table 2 is partially circular. The same metric with hand-set weights (alpha=0.6, beta=0.3, gamma=0.1) is used inside SQL-Factory's Critical Agent to filter and deduplicate the SQL Pool, while the baselines' outputs were not filtered with it; a lower average hybrid similarity for SQL-Factory is therefore partly by construction. The independent Vendi Scores with SQL-Encoder and SimCSE support the diversity claim, so this issue is not fatal, but the hybrid-similarity comparison should be supplemented by an evaluation on the unfiltered pool, or the hybrid-similarity claim should be dropped or de-emphasized in favor of the encoder-based diversity measures.","section":"§4.1.2, Eq. (3); §5.2.3, Table 2"}],"minor_comments":[{"comment":"The text says 'the expansion ratio rapidly decrease while the average similarity among queries steadily rising'; the verb forms are inconsistent, and the direction of the expansion-ratio change should be stated precisely relative to Figure 9.","section":"§5.2.6"},{"comment":"For Deepseek-Coder-1.3B on BIRD, '+SQL-Factory' executability (79.8) is slightly below '+Official-Data' (80.3), which weakens the statement that SQL-Factory 'consistently improves' over the official-data condition; this exception should be acknowledged or explained.","section":"Table 4"},{"comment":"The method name 'LearnedGen' in Table 2 should be aligned with 'LearnedSQLGen' used in the text, and the duplicated column header 'Vendi Score Vendi Score' should be corrected.","section":"Table 2"},{"comment":"The schema-aware allocation analysis is qualitative: the claim that SQL-Factory's query distribution 'aligns well' with table complexity is based on visual inspection of the figures, so a quantitative measure such as the Spearman correlation between per-table query counts and Complexity(T) should be reported for both SQL-Factory and OmniSQL.","section":"§5.2.5, Figures 7 and 8"},{"comment":"Tables 4 and 5 report point estimates without variance or repeated-run statistics; because fine-tuning and clustering evaluation are stochastic, the paper should report standard deviations over multiple runs or at least state the number of runs.","section":"Tables 4 and 5"},{"comment":"The hand-set weights in the hybrid similarity metric are not accompanied by a sensitivity analysis; since the weights affect both the internal filtering and the diversity comparison, a brief robustness check (e.g., varying alpha/beta/gamma over a small grid) would strengthen the presentation.","section":"Eq. (3)"}],"recommendation":"major_revision","confidential_remarks":"The artifact release and the independent Vendi Score evidence are significant strengths, and I see the main risks as verification and control issues rather than fundamental errors. The equal-volume control and decontamination analysis are essential before the paper can support the Text-to-SQL claims; I believe these are addressable within a revision, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is a real engineering contribution, and the central claim—300k executable queries across four benchmarks for under $200—holds up better than most similar claims. The genuinely new piece is the division of labor: GPT-4o explores structural patterns, a local 14B model expands them, and a management agent schedules between the two using schema-complexity statistics. I have not seen that decomposition in the cited literature, and it makes sense. The Vendi Score measurements are concrete, and the comparison against SQLSmith, LearnedSQLGen, and OmniSQL is fair enough on the diversity axis. The paper also ships code and data on GitHub, so the results are checkable; that is real evidence and should be credited.\n\nThe soft spots are in the downstream claims. Table 4 compares SQL-Factory fine-tuning against official-data fine-tuning without an equal-volume control; if one side gets many more training pairs, part of the gain could just be volume. More importantly, the paper never checks whether GPT-4o has memorized Spider and BIRD queries from pretraining. Since those same benchmarks are used for evaluation, contamination would inflate the reported improvements. This is a verification gap, not a demonstrated flaw—the artifact release makes it testable—but the authors should have addressed it head-on.\n\nThere is also a metric-reuse issue: the hybrid similarity from Eq. (3) is used both as a filter inside the pipeline and as one of the diversity metrics in Table 2. That gives SQL-Factory an advantage the baselines did not have. The independent Vendi scores temper this, but it should be acknowledged and ideally re-evaluated with a held-out metric. Minor issues: no variance estimates across runs, 15% empty-result queries are reported without much discussion, and the stopping criterion is empirical rather than principled.\n\nWho is this for? Database systems and data-centric NLP people who need cheap, large SQL corpora for benchmarking, representation learning, or training data. It deserves serious peer review: the cost-diversity claim is important enough, and the artifact release makes it falsifiable. I would want the authors to add a decontamination analysis and an equal-volume control before acceptance, but I would not desk-reject it.","headline":"A credible systems paper that makes a concrete cost-diversity claim for SQL corpus generation, with solid diversity evidence but downstream Text-to-SQL comparisons that are weaker than they look.","tokens_in":21038,"tokens_out":1912,"would_cite":true,"duration_ms":19782,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SQL-Factory claims that a three-team multi-agent pipeline can synthesize over 300,000 executable, structurally diverse SQL queries across four benchmarks for under $200 in API cost, and that this synthetic corpus rivals official benchmark…","keywords":["SQL generation","multi-agent framework","text-to-SQL","query diversity","schema-aware generation","large language models","data augmentation","query clustering"],"falsifier":"Compute token-level and AST-level overlap between the generated corpus and the official Spider and BIRD test sets, then retrain the same text-to-SQL models on a version of the corpus stripped of any query whose schema fragments resemble a test query; if the accuracy gains shrink to the official-data level or below, the reported improvements come from benchmark memorization rather than from SQL-Factory's generation mechanism.","tokens_in":20066,"feed_emoji":"🏭","tokens_out":10049,"duration_ms":83703,"temperature":0.7,"pith_summary":"The paper argues that the hard trade-off in automatic SQL generation—between structural diversity, schema coverage, and cost—can be broken by dividing the work among three cooperating agent teams. A strong language model generates novel, join-heavy queries; a lightweight local model scales those patterns into variants; and a management team evaluates executability and redundancy in real time, switching between the two. Applied to TPC-DS, IMDB, Spider, and BIRD, the system reports over 300,000 generated queries for under $200 in API cost, with higher diversity than existing generators. The payoff the paper pursues is practical: synthetic SQL corpora used as fine-tuning data improve small text-to-SQL models beyond what official benchmark training sets achieve, and improve SQL clustering representations.","feed_headline":"Three-agent factory writes 300,000 varied SQL queries for under $200","feed_subtitle":"A cheap synthetic corpus rivals human-curated data for training text-to-SQL and SQL clustering models.","key_machinery":"The load-bearing mechanism is the explore–exploit scheduling loop gated by a hybrid-similarity evaluator. A centralized SQL Pool stores each query's text, referenced tables, and precomputed BERT embedding, and the Management Agent switches between the Generation Team and the Expansion Team through LLM-based inference over the state transition $S_{t+1}=\\mathrm{AgentM}(S_t,E_t)$. The Critical Agent combines an execution checker with a retrieval tool that scores redundancy as $\\mathrm{Sim}(q,q')=\\alpha\\,\\mathrm{SimTok}+\\beta\\,\\mathrm{SimAST}+\\gamma\\,\\mathrm{SimEmb}$, with $\\alpha=0.6$, $\\beta=0.3$, $\\gamma=0.1$, where token-level, AST edit-distance, and BERT cosine similarities each contribute. This gating is what permits a cheap local 14B model to produce most of the volume while a frontier model seeds only novel structures, keeping the reported API cost under $200.","core_discovery":"SQL-Factory's central discovery is that large-scale, high-quality SQL synthesis can be organized as an exploration–exploitation loop instead of a single monolithic generation call. In the exploration phase, a Table Selection Agent prioritizes tables by schema complexity and current coverage, and a powerful generation model writes novel join-heavy queries; in the exploitation phase, a lightweight local model rewrites selected seed queries into variants. A management layer filters candidates by executability and hybrid similarity, then decides when to switch modes. The paper reports 380,000 generated queries across TPC-DS, IMDB, Spider, and BIRD for $179.91 total API cost, of which 78.6% contain at least one join and 15.34% return empty result sets, and shows fine-tuning and contrastive-learning experiments where this corpus improves downstream text-to-SQL and clustering models.","pith_inferences":["The paper does not test whether the same explore–exploit scheduling transfers to other structured-output domains; a natural extension is to apply the three-team loop to Python, Cypher, or SPARQL generation, where a frontier model seeds skeletons and a cheap local model scales them.","The hybrid-similarity gate could serve as a general-purpose filter for synthetic code corpora; one testable question is whether its 0.6/0.3/0.1 token/AST/embedding weighting is close to optimal for predicting downstream model accuracy.","The reported text-to-SQL gains are consistent with GPT-4o having memorized benchmark test queries; an extension the authors do not run is to measure n-gram and AST overlap between generated queries and the Spider/BIRD test sets and rerun the pipeline with an open-weight generator.","The stopping-criterion argument suggests a broader data-centric rule: keep generating while marginal diversity exceeds a threshold, then stop; this saturation detector could be lifted into any LLM-based data augmentation pipeline."],"forward_implications":["Fine-tuning a 1B-parameter text-to-SQL model on SQL-Factory data lifts Spider exact-match accuracy from 41.3% to 70.6% for LLaMA3.2-1B, exceeding the gain from official benchmark training data.","Contrastive fine-tuning of BERT-Large on the generated corpus improves clustering quality on all three academic benchmarks tested, lowering the BetaCV compactness metric below every compared baseline.","The GPT-4o-and-expansion split cuts API cost to $179.91 across four benchmarks, about one-fifth of the estimated $1,000 if GPT-4o generated everything alone.","Schema-aware table selection aligns query frequency with table complexity, so structurally rich tables receive dense coverage instead of being starved by uniform sampling.","The observed saturation point—expansion ratio rising and average similarity trending upward—gives a data-driven stopping criterion for when further synthesis yields diminishing returns."],"supporting_citations":[{"why":"SQLSmith is the grammar-driven random generator whose low pairwise similarity serves as the incoherence baseline in the diversity comparison.","marker":"[46]"},{"why":"LearnedSQLGen is the reinforcement-learning generator used as the redundant-pattern baseline in the diversity comparison.","marker":"[64]"},{"why":"OmniSQL is the LLM-based large-scale generator to which SQL-Factory compares schema-aware allocation and cost.","marker":"[27]"},{"why":"The Vendi Score is the eigenvalue-entropy diversity metric used to score the generated corpora.","marker":"[13]"},{"why":"SimCSE is the general-purpose sentence encoder used as one backbone for computing Vendi Scores.","marker":"[15]"},{"why":"SQL-Encoder is the SQL-specialized encoder used as the second Vendi Score backbone.","marker":"[44]"},{"why":"Spider supplies the schemas, official training data, and text-to-SQL evaluation used in the fine-tuning experiments.","marker":"[60]"},{"why":"BIRD supplies the realistic text-to-SQL schemas and evaluation used alongside Spider.","marker":"[29]"},{"why":"TPC-DS is the 24-table decision-support benchmark used as a generation target and diversity testbed.","marker":"[37]"}],"fun_headline_variants":["Multi-agent SQL factory: 380k queries for under $200","AI agents collaborate to write 380k SQL queries cheaply","SQL-Factory: diverse SQL at scale for cents per query","Three teams, one goal: cheap diverse SQL generation","Exploration-exploitation loop yields 380k SQL queries"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"All reported downstream gains rest on the assumption that GPT-4o has not memorized the Spider and BIRD test queries or schemas during its own training, so the synthetic SQL-to-natural-language pairs do not leak test-like content into the fine-tuned models.","fun_headline_variants_meta":{"raw":{"variants":["Multi-agent SQL factory: 380k queries for under $200","AI agents collaborate to write 380k SQL queries cheaply","SQL-Factory: diverse SQL at scale for cents per query","Three teams, one goal: cheap diverse SQL generation","Exploration-exploitation loop yields 380k SQL queries"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000782,"raw_usage":{"total_tokens":3456,"prompt_tokens":948,"completion_tokens":2508,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":564,"completion_tokens_details":{"reasoning_tokens":2424}},"tokens_in":564,"tokens_out":2508,"duration_ms":16144,"temperature":1.0,"reasoning_tokens":2424,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:39:28.763708+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute token-level and AST-level overlap between the generated corpus and the official Spider and BIRD test sets, then retrain the same text-to-SQL models on a version of the corpus stripped of any query whose schema fragments resemble a test query; if the accuracy gains shrink to the official-data level or below, the reported improvements come from benchmark memorization rather than from SQL-Factory's generation mechanism.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Spider supplies the schemas, official training data, and text-to-SQL evaluation used in the fine-tuning experiments."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SQLSmith is the grammar-driven random generator whose low pairwise similarity serves as the incoherence baseline in the diversity comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"LearnedSQLGen is the reinforcement-learning generator used as the redundant-pattern baseline in the diversity comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OmniSQL is the LLM-based large-scale generator to which SQL-Factory compares schema-aware allocation and cost."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The Vendi Score is the eigenvalue-entropy diversity metric used to score the generated corpora."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"SQL-Encoder is the SQL-specialized encoder used as the second Vendi Score backbone."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"BIRD supplies the realistic text-to-SQL schemas and evaluation used alongside Spider."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"TPC-DS is the 24-table decision-support benchmark used as a generation target and diversity testbed."}],"review_version":1}