{"id":"e8f540be-a408-425d-8c58-65db3f0c5b08","arxiv_id":"2603.07841","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"FusionSQL predicts a Text2SQL model's execution accuracy on unseen unlabeled workloads from embedding-distance shift descriptors, reaching about 4-point MAE on benchmark transfers.","lead":"FusionSQL estimates how accurate a text-to-SQL model will be on a new, unlabeled dataset by comparing the model's internal representations of its training data with those of the new questions. It offers a label-free accuracy estimate—about four percentage points of error on benchmark transfers—that could cut the cost of pre-deployment evaluation.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Central claim rests on synthetic-to-real transfer of the shift→accuracy mapping; only benchmark-specific validation is shown.","rationale":"The reader's weakest_assumption exactly identifies the synthetic-to-real transfer of the shift→accuracy relationship as the load-bearing premise. I agree with that assessment after reviewing the method. The central experiment (Table III) is strong evidence on five benchmark transfers, but those transfers are curated and may share structure with FusionDataset (e.g., both are derived from similar SQL synthesis pipelines). The paper does not provide a held-out real deployment test nor a formal out-of-distribution guarantee; the qualitative coverage plots are not sufficient to establish that the conditional relationship learned on synthetic targets holds for arbitrary real shifts. The concern is not that the method is wrong—the reported results are plausible and reproducible code is claimed—but that the generality claim exceeds the evidence. The label-free issue for unseen models is secondary: the meta-learning adaptation uses labeled FusionDataset subsets, which weakens the 'any Text2SQL model' framing but does not invalidate the fixed-model transfer result that is the paper's main contribution. A concrete held-out benchmark test would settle whether the synthetic training distribution spans real deployment shifts, so the verdict remains CONDITIONAL pending that check.","tokens_in":24495,"tokens_out":5779,"duration_ms":62052,"concrete_test":"Run FusionSQL on three additional real-world labeled benchmarks not used in the five reported transfers—EHRSQL, ScienceBenchmark, and KaggleDBQA—as target workloads, using the same frozen models (e.g., Qwen2.5-72B) and the released code. Compute MAE between predicted and true execution accuracy. If MAE on these holds at or near the 3–5 point range reported in Table III, the synthetic-to-real transfer concern is substantially mitigated; if MAE jumps significantly (e.g., >8–10 points), the mapping does not generalize beyond the selected benchmark transfers.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim (§IV-C, Eq. 10) is that a regressor g_θ trained on shift descriptors Δ computed between a model's training workload and synthetic FusionDataset subsets will, at deployment, map a real train–test Δ_test to a reliable accuracy estimate. This requires the joint distribution of (Δ, execution accuracy) seen during training to cover real deployment shifts. FusionDataset (§IV-A) is LLM-generated from TabLib/KaggleDBQA tables, with synthetic SQL and questions; it may not contain the value-grounding, numeric reasoning, or domain-specific phrasings that dominate real targets like BIRD or Spider 2.0. The coverage evidence in §V-A (t-SNE, radar) demonstrates marginal embedding coverage of individual benchmarks, not that the conditional Δ→accuracy relationship trained on synthetic targets matches real-world conditions. The five transfers in Table III are a selected, finite set; good MAE there does not guarantee acceptable MAE on a genuinely new deployment workload. Fig. 10b actually shows the shift-sensitivity is not monotonic (distant-but-easy transfers are flat), so the regressor must learn shift-direction-specific effects—this makes extrapolation beyond the training support especially fragile. The paper itself acknowledges no formal guarantee, and the 3.3M-example FusionDataset is not released, so independent reproduction of the training distribution is impossible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes FusionSQL, a method to estimate the execution accuracy of a fixed Text2SQL model on an unseen, unlabeled workload without gold SQL. It computes shift descriptors (Fréchet, Mahalanobis, sliced Wasserstein distances) between pooled embeddings of the model's training workload and the target workload, and trains a small MLP regressor on a large synthetic corpus (FusionDataset, 3.3M examples) to map these descriptors to accuracy. Experiments on five public benchmark transfers report MAE around 4–5 percentage points, outperforming confidence-based and LLM-judge baselines. A meta-learning extension (FusionSQL-ML) is proposed for unseen models, and ablations address scalability and non-neural systems.","tokens_in":24723,"tokens_out":10398,"duration_ms":91901,"significance":"If the transfer result is robust, FusionSQL addresses a real deployment gap and is one of the first dataset-level label-free evaluators for Text2SQL. Strengths include a clearly stated problem formulation, a large synthetic training corpus, a lightweight and interpretable descriptor set, and an efficiency analysis. The code release is a positive step. However, the paper's headline claims are weakened by an unspecified MAE unit, a labeled auxiliary set in the unseen-model protocol, an unevaluated uncertainty objective, and a transfer evidence base that is finite and descriptive rather than a stress test of out-of-support shifts. These issues need to be addressed before the results can be taken at face value.","major_comments":[{"comment":"The unit over which MAE is computed is not defined. For a transfer such as Spider→BIRD there is a single target workload, so Eq. (15)'s N cannot be one unless you split the target into multiple datasets (per database or per subset). The paper reports MAE with 95% CIs averaged over five seeds, but this does not explain how a dataset-level MAE is formed. Please specify what N is, how shift descriptors are computed for each unit, and report per-unit results. Without this, the central numerical claim in Table III is not interpretable.","section":"§V-C, Table III, Eq. (15)"},{"comment":"The generalization-to-unseen-models protocol is not label-free. The text says 'At deployment, for a previously unseen model m_new, we adapt the evaluator using S_i. Since S_i is known, we can compute the model's accuracy and form the loss L_{m_new,S_i}(θ).' Computing accuracy on S_i requires gold SQL and execution. Thus FusionSQL-ML uses reference labels on an auxiliary set, contradicting the abstract's 'without reference labels.' The paper should either scope the claim to 'no labels on the target workload' or provide a label-free adaptation procedure, and Table IV should be re-described accordingly.","section":"§IV-C, Table IV"},{"comment":"The problem formulation promises calibrated uncertainty estimates: a prediction interval [M̂−δ_α, M̂+δ_α] with P(M* ∈ ...) ≥ 1−α. No experiment in §V reports interval half-widths, empirical coverage, or calibration plots. Since this is listed as a required property of the evaluator, the claim is unsupported. Add uncertainty-calibration experiments or remove/soften the property.","section":"§III-C, §V"},{"comment":"The central transfer mechanism trains g_θ on synthetic shifts from FusionDataset and applies it to real benchmarks. The evidence is limited to five selected transfers, and the coverage analysis (t-SNE, radar) is descriptive. Fig. 10b shows the shift–accuracy relationship is not monotonic, so extrapolation to shift directions outside FusionDataset's support is fragile. Please add an explicit out-of-support test (e.g., a deployment workload whose schema/question style is deliberately absent from FusionDataset) or a leave-one-benchmark-out protocol, and report MAE and per-transfer errors. This would make the 'unseen workload' claim more than a finite-set demonstration.","section":"§IV-A, §V-A, Fig. 10b"}],"minor_comments":[{"comment":"The symbols ŷ_i and y_i are used for SQL queries in Eq. (14) and then for predicted/true accuracies in Eq. (15); use different notation to avoid ambiguity.","section":"Eq. (14)–(15)"},{"comment":"Most 'unseen' models are size variants of families already in the meta-training pool (CodeLlama, StarCoder2, Phi, DeepSeek-Coder); only Mistral-7B is a genuinely new family. Please clarify how much of the reported gain is due to cross-family generalization.","section":"Table IV"},{"comment":"Pearson r=0.53–0.70 is described as 'strong'; this is more accurately moderate correlation. Please qualify the wording.","section":"§V-C, Fig. 10a"},{"comment":"FusionDataset is not released. Since the training distribution is central to the method, please include a data-release plan or a detailed generation script in the repository.","section":"§IV-A"},{"comment":"The sentence 'FusionSQL is trained purely from shift descriptors ... zero-label transfer setting' conflicts with the labeled adaptation described in §IV-C for unseen models. Clarify the scope.","section":"§V-C"}],"recommendation":"major_revision","confidential_remarks":"The MAE construction and the label requirement in the unseen-model protocol are the two most important points to verify in revision. A statistician's review of how the reported CIs are derived from the transfer design would be valuable. The dataset release is essential for reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper introduces FusionSQL, a method to estimate a frozen Text2SQL model's execution accuracy on an unlabeled target workload by regressing distribution-shift descriptors (Frechet distance, Mahalanobis tail stats, sliced Wasserstein distance) against accuracy. The headline result is real: on five benchmark transfers, FusionSQL achieves MAE around 4.2 percentage points with zero target labels, beating confidence-based (ATC, DoC) and judge-based (BugJudge, ArenaCmp) baselines by a large margin. That is the core contribution, and it is well supported by Table III and the latency/ablation analysis.\n\nWhat is genuinely new here is the Text2SQL-specific package: the shift-descriptor triple, the large synthetic FusionDataset (3.3M pairs), and the meta-learning extension to unseen models. The paper is careful about efficiency and includes a non-neural baseline, which is a nice touch. The writing is honest about what the method requires and does not oversell the architecture.\n\nThe soft spots are in the generalization claim. The central assumption is that the shift-to-accuracy mapping learned on LLM-generated FusionDataset transfers to arbitrary deployment databases. The evidence for that is descriptive—t-SNE and radar plots—not a formal guarantee. The five benchmark transfers are a small, selected set, and Figure 10b shows the mapping is not monotonic, so extrapolation beyond the training support is genuinely fragile. Also, the unseen-model protocol in §IV-C adapts on labeled FusionDataset subsets, so it is not fully label-free in that setting. The uncertainty-interval objective stated in §III-C is never measured. And FusionDataset is not released, making independent reproduction impossible. These are real limitations, but they do not break the zero-label transfer result on the tested benchmarks.\n\nThis paper is for researchers and practitioners working on Text2SQL evaluation and deployment monitoring. It deserves a serious referee, but the authors should be pushed to address the generalization concern, either by releasing the dataset or by providing a more systematic analysis of when the mapping holds. I would accept it for peer review with major revisions in mind.\n\nI would bring this to a reading group that focuses on model evaluation, and I would cite it in my own work on label-free evaluation.","headline":"A solid label-free accuracy estimator for Text2SQL with a strong zero-label transfer result, but the synthetic-to-real generalization story is asserted more than proven.","tokens_in":25321,"tokens_out":1912,"would_cite":true,"duration_ms":20485,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fixed Text2SQL model's execution accuracy on an unseen, unlabeled workload can be estimated from distribution-shift descriptors alone — no gold SQL or model retraining needed.","keywords":["Text2SQL","label-free evaluation","distribution shift","execution accuracy","model-agnostic evaluator","synthetic dataset","meta-learning"],"falsifier":"Take two target workloads whose pooled embedding sets yield identical SD_F, SD_M, and SD_SW relative to the same source, but whose true execution accuracies differ by more than 10 points — for instance, one set of simple single-table questions and an identically phrased set over a schema with many semantically overlapping columns. If the regressor outputs the same estimate for both, the descriptor set cannot be carrying the accuracy-relevant information, and the central claim collapses.","tokens_in":24303,"feed_emoji":"📊","tokens_out":6687,"duration_ms":55646,"temperature":0.7,"pith_summary":"The paper asks whether a fixed Text2SQL model's accuracy on a brand-new, unlabeled database can be estimated before any gold SQL exists. It answers yes: by pooling the model's own embeddings on its training data and on the new workload, computing three distribution-shift descriptors (global drift, tail risk, shape change), and feeding them to a small regressor, the framework predicts execution accuracy without labels or model retraining. A reader should care because text-to-SQL systems are increasingly deployed against evolving databases where labeled test sets are expensive or privacy-blocked; a reliable label-free estimate turns deployment from a gamble into a measurable decision. The paper trains and validates the approach on a large synthetic corpus and reports mean absolute errors of about four percentage points across five benchmark transfers, with similar behavior for unseen model families via meta-learning.","feed_headline":"Text2SQL accuracy on new data predicted without labels","feed_subtitle":"Shift-based estimator lands within ~4 points of true accuracy on five benchmark transfers, no labels required.","key_machinery":"The load-bearing object is the shift descriptor Δ = [SD_F, SD_M, SD_SW] plus the regression head g_θ. SD_F = (∥μ_T−μ_S∥₂², variance ratios) captures global drift; SD_M whitens target embeddings with source statistics to expose tail behavior; SD_SW averages 1D Wasserstein distances between sliced projections to capture shape changes. These are computed on pooled last-layer embeddings of the model applied to its training workload and to the target workload, so the estimator needs no labels, no execution, and no access to model weights. A 3-layer MLP regressor (and a Reptile meta-learning wrapper for unseen models) maps Δ to an execution-accuracy estimate; a Hybrid SWD approximation keeps runti","core_discovery":"The paper's central claim is that the dataset-level execution accuracy of a frozen Text2SQL model on an unseen, unlabeled workload, M*, can be estimated as M̂ = g_θ(Δ_test), where Δ_test = h(ϕ_src, ϕ_tgt) is a fixed-length shift descriptor comparing pooled embeddings of the model's training workload and the target workload, and g_θ is a three-layer MLP trained on synthetic shift–accuracy pairs. The descriptor h combines three complementary statistics: a Fréchet term for global domain drift (mean and variance shifts), a Mahalanobis term for tail risk (rare, failure-prone queries), and a sliced Wasserstein term for distributional shape change (schema restructuring, collapsed query templates).","pith_inferences":["The same shift-descriptor recipe (global + tail + shape statistics on pooled embeddings) could plausibly transfer to other structured prediction tasks — code generation, semantic parsing — where accuracy is defined by execution and labels are scarce; the paper does not test this.","The paper's meta-adaptation for a previously unseen model requires computing the model's accuracy on known sample sets, which presupposes labels for those samples; the 'fully label-free' framing strictly holds for models whose behavior is already covered by the meta-training pool.","The stated objective of calibrated prediction intervals is not reported in the experiments; a testable extension is to check coverage of the reported MAE intervals on held-out transfers."],"forward_implications":["Pre-deployment checks become possible without labels: teams can estimate whether a Text2SQL model is ready for a new database before any gold SQL is written.","Continuous monitoring of evolving schemas is feasible: as a database changes, shift descriptors can be recomputed and an accuracy drop flagged before users report failures.","The estimator is model-agnostic: it works with five large neural systems and with classic non-neural Text2SQL engines, so one evaluation tool can cover an organization's whole model fleet.","New Text2SQL model families can be handled after a few gradient steps, because the meta-learned initialization adapts rapidly without target labels.","Because the descriptor pipeline is a matrix multiplication plus a small MLP, evaluation is faster than LLM-as-a-judge baselines, making routine readiness checks practical."],"fun_headline_variants":["Estimate Text2SQL accuracy on new data, no labels needed","FusionSQL: predict Text2SQL accuracy without gold labels","No labels? FusionSQL still gauges Text2SQL accuracy","Text2SQL accuracy on unseen data predicted via shift analysis"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The learned relationship between shift descriptors and accuracy transfers from the synthetic FusionDataset to real deployment targets, including shift directions not present in the synthetic corpus.","fun_headline_variants_meta":{"raw":{"variants":["Estimate Text2SQL accuracy on new data, no labels needed","FusionSQL: predict Text2SQL accuracy without gold labels","No labels? FusionSQL still gauges Text2SQL accuracy","Text2SQL accuracy on unseen data predicted via shift analysis"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000355,"raw_usage":{"total_tokens":1753,"prompt_tokens":717,"completion_tokens":1036,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":461,"completion_tokens_details":{"reasoning_tokens":974}},"tokens_in":461,"tokens_out":1036,"duration_ms":7401,"temperature":1.0,"reasoning_tokens":974,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T18:34:35.280869+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take two target workloads whose pooled embedding sets yield identical SD_F, SD_M, and SD_SW relative to the same source, but whose true execution accuracies differ by more than 10 points — for instance, one set of simple single-table questions and an identically phrased set over a schema with many semantically overlapping columns. If the regressor outputs the same estimate for both, the descriptor set cannot be carrying the accuracy-relevant information, and the central claim collapses.","supporting_citations":[],"review_version":1}