Pith. sign in

REVIEW 4 major objections 5 minor 2 cited by

A fixed Text2SQL model's execution accuracy on an unseen, unlabeled workload can be estimated from distribution-shift descriptors alone — no gold SQL or model retraining needed.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-02 18:34 UTC pith:APYB5ZRV

load-bearing objection A solid label-free accuracy estimator for Text2SQL with a strong zero-label transfer result, but the synthetic-to-real generalization story is asserted more than proven. the 4 major comments →

arxiv 2603.07841 v2 pith:APYB5ZRV submitted 2026-03-08 cs.CL

An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data

Trinh Pham , Thanh Tam Nguyen , Viet Huynh , Hongzhi Yin , Quoc Viet Hung Nguyen This is my paper
classification cs.CL
keywords Text2SQLlabel-free evaluationdistribution shiftexecution accuracymodel-agnostic evaluatorsynthetic datasetmeta-learning
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper asks whether a fixed Text2SQL model's accuracy on a brand-new, unlabeled database can be estimated before any gold SQL exists. It answers yes: by pooling the model's own embeddings on its training data and on the new workload, computing three distribution-shift descriptors (global drift, tail risk, shape change), and feeding them to a small regressor, the framework predicts execution accuracy without labels or model retraining. A reader should care because text-to-SQL systems are increasingly deployed against evolving databases where labeled test sets are expensive or privacy-blocked; a reliable label-free estimate turns deployment from a gamble into a measurable decision. The paper trains and validates the approach on a large synthetic corpus and reports mean absolute errors of about four percentage points across five benchmark transfers, with similar behavior for unseen model families via meta-learning.

Core claim

The paper's central claim is that the dataset-level execution accuracy of a frozen Text2SQL model on an unseen, unlabeled workload, M*, can be estimated as M̂ = g_θ(Δ_test), where Δ_test = h(ϕ_src, ϕ_tgt) is a fixed-length shift descriptor comparing pooled embeddings of the model's training workload and the target workload, and g_θ is a three-layer MLP trained on synthetic shift–accuracy pairs. The descriptor h combines three complementary statistics: a Fréchet term for global domain drift (mean and variance shifts), a Mahalanobis term for tail risk (rare, failure-prone queries), and a sliced Wasserstein term for distributional shape change (schema restructuring, collapsed query templates).

What carries the argument

The load-bearing object is the shift descriptor Δ = [SD_F, SD_M, SD_SW] plus the regression head g_θ. SD_F = (∥μ_T−μ_S∥₂², variance ratios) captures global drift; SD_M whitens target embeddings with source statistics to expose tail behavior; SD_SW averages 1D Wasserstein distances between sliced projections to capture shape changes. These are computed on pooled last-layer embeddings of the model applied to its training workload and to the target workload, so the estimator needs no labels, no execution, and no access to model weights. A 3-layer MLP regressor (and a Reptile meta-learning wrapper for unseen models) maps Δ to an execution-accuracy estimate; a Hybrid SWD approximation keeps runti

Load-bearing premise

The learned relationship between shift descriptors and accuracy transfers from the synthetic FusionDataset to real deployment targets, including shift directions not present in the synthetic corpus.

What would settle it

Take two target workloads whose pooled embedding sets yield identical SD_F, SD_M, and SD_SW relative to the same source, but whose true execution accuracies differ by more than 10 points — for instance, one set of simple single-table questions and an identically phrased set over a schema with many semantically overlapping columns. If the regressor outputs the same estimate for both, the descriptor set cannot be carrying the accuracy-relevant information, and the central claim collapses.

Watch this falsifier. Get emailed when new claim-graph text bears on it.

If this is right

  • Pre-deployment checks become possible without labels: teams can estimate whether a Text2SQL model is ready for a new database before any gold SQL is written.
  • Continuous monitoring of evolving schemas is feasible: as a database changes, shift descriptors can be recomputed and an accuracy drop flagged before users report failures.
  • The estimator is model-agnostic: it works with five large neural systems and with classic non-neural Text2SQL engines, so one evaluation tool can cover an organization's whole model fleet.
  • New Text2SQL model families can be handled after a few gradient steps, because the meta-learned initialization adapts rapidly without target labels.
  • Because the descriptor pipeline is a matrix multiplication plus a small MLP, evaluation is faster than LLM-as-a-judge baselines, making routine readiness checks practical.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same shift-descriptor recipe (global + tail + shape statistics on pooled embeddings) could plausibly transfer to other structured prediction tasks — code generation, semantic parsing — where accuracy is defined by execution and labels are scarce; the paper does not test this.
  • The paper's meta-adaptation for a previously unseen model requires computing the model's accuracy on known sample sets, which presupposes labels for those samples; the 'fully label-free' framing strictly holds for models whose behavior is already covered by the meta-training pool.
  • The stated objective of calibrated prediction intervals is not reported in the experiments; a testable extension is to check coverage of the reported MAE intervals on held-out transfers.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes FusionSQL, a method to estimate the execution accuracy of a fixed Text2SQL model on an unseen, unlabeled workload without gold SQL. It computes shift descriptors (Fréchet, Mahalanobis, sliced Wasserstein distances) between pooled embeddings of the model's training workload and the target workload, and trains a small MLP regressor on a large synthetic corpus (FusionDataset, 3.3M examples) to map these descriptors to accuracy. Experiments on five public benchmark transfers report MAE around 4–5 percentage points, outperforming confidence-based and LLM-judge baselines. A meta-learning extension (FusionSQL-ML) is proposed for unseen models, and ablations address scalability and non-neural systems.

Significance. If the transfer result is robust, FusionSQL addresses a real deployment gap and is one of the first dataset-level label-free evaluators for Text2SQL. Strengths include a clearly stated problem formulation, a large synthetic training corpus, a lightweight and interpretable descriptor set, and an efficiency analysis. The code release is a positive step. However, the paper's headline claims are weakened by an unspecified MAE unit, a labeled auxiliary set in the unseen-model protocol, an unevaluated uncertainty objective, and a transfer evidence base that is finite and descriptive rather than a stress test of out-of-support shifts. These issues need to be addressed before the results can be taken at face value.

major comments (4)
  1. [§V-C, Table III, Eq. (15)] The unit over which MAE is computed is not defined. For a transfer such as Spider→BIRD there is a single target workload, so Eq. (15)'s N cannot be one unless you split the target into multiple datasets (per database or per subset). The paper reports MAE with 95% CIs averaged over five seeds, but this does not explain how a dataset-level MAE is formed. Please specify what N is, how shift descriptors are computed for each unit, and report per-unit results. Without this, the central numerical claim in Table III is not interpretable.
  2. [§IV-C, Table IV] The generalization-to-unseen-models protocol is not label-free. The text says 'At deployment, for a previously unseen model m_new, we adapt the evaluator using S_i. Since S_i is known, we can compute the model's accuracy and form the loss L_{m_new,S_i}(θ).' Computing accuracy on S_i requires gold SQL and execution. Thus FusionSQL-ML uses reference labels on an auxiliary set, contradicting the abstract's 'without reference labels.' The paper should either scope the claim to 'no labels on the target workload' or provide a label-free adaptation procedure, and Table IV should be re-described accordingly.
  3. [§III-C, §V] The problem formulation promises calibrated uncertainty estimates: a prediction interval [M̂−δ_α, M̂+δ_α] with P(M* ∈ ...) ≥ 1−α. No experiment in §V reports interval half-widths, empirical coverage, or calibration plots. Since this is listed as a required property of the evaluator, the claim is unsupported. Add uncertainty-calibration experiments or remove/soften the property.
  4. [§IV-A, §V-A, Fig. 10b] The central transfer mechanism trains g_θ on synthetic shifts from FusionDataset and applies it to real benchmarks. The evidence is limited to five selected transfers, and the coverage analysis (t-SNE, radar) is descriptive. Fig. 10b shows the shift–accuracy relationship is not monotonic, so extrapolation to shift directions outside FusionDataset's support is fragile. Please add an explicit out-of-support test (e.g., a deployment workload whose schema/question style is deliberately absent from FusionDataset) or a leave-one-benchmark-out protocol, and report MAE and per-transfer errors. This would make the 'unseen workload' claim more than a finite-set demonstration.
minor comments (5)
  1. [Eq. (14)–(15)] The symbols ŷ_i and y_i are used for SQL queries in Eq. (14) and then for predicted/true accuracies in Eq. (15); use different notation to avoid ambiguity.
  2. [Table IV] Most 'unseen' models are size variants of families already in the meta-training pool (CodeLlama, StarCoder2, Phi, DeepSeek-Coder); only Mistral-7B is a genuinely new family. Please clarify how much of the reported gain is due to cross-family generalization.
  3. [§V-C, Fig. 10a] Pearson r=0.53–0.70 is described as 'strong'; this is more accurately moderate correlation. Please qualify the wording.
  4. [§IV-A] FusionDataset is not released. Since the training distribution is central to the method, please include a data-release plan or a detailed generation script in the repository.
  5. [§V-C] The sentence 'FusionSQL is trained purely from shift descriptors ... zero-label transfer setting' conflicts with the labeled adaptation described in §IV-C for unseen models. Clarify the scope.

Circularity Check

0 steps flagged

No construction-level circularity: the evaluator is trained on synthetic shift→accuracy pairs and applied to real unlabeled targets; the main risks are external validity and evaluation hygiene, not definitional or self-citational circularity.

full rationale

The central chain is: (1) construct FusionDataset of synthetic workloads with gold SQL; (2) for each (model, synthetic subset) pair compute shift descriptors Δ = h(φ_src, φ_syn) and the true synthetic accuracy M*_syn; (3) train g_θ by minimizing E[(g_θ(Δ)−M*)^2]; (4) at inference compute Δ_test from the real training workload and the unlabeled target workload, then output M̂_test = g_θ(Δ_test) (Eqs. 7–10). Nothing in this chain defines Δ in terms of M* on the target, nor fits any parameter to target labels. The public-benchmark transfers in Table III use target gold only to measure MAE after prediction, not as training or adaptation signal, so the reported numbers are external validation of a learned mapping rather than a forced reconstruction of the input. The adaptation protocol for unseen models (§IV-C, Eq. 13) does use labeled subsets S_i drawn from FusionDataset to adapt g_θ, but S_i is a synthetic training/adaptation set, not the unlabeled deployment target; the final target prediction remains label-free and is not statistically forced by a fit to target accuracy. Self-citations are present but not load-bearing: the sliced Wasserstein distance is cited to co-author prior work [51]–[53], but SWD is a standard, externally defined metric, and the meta-learning steps cite Reptile [54] and MAML [55], both external algorithms. The t-SNE and radar coverage evidence (§V-A) is descriptive support for the synthetic-to-real transfer assumption, not a derivation step; if the conditional Δ→accuracy relationship does not hold on genuinely new deployment workloads, that is a generalization-failure risk, not circularity. The paper does not state that early stopping or hyperparameters were selected on the Table III targets, so I do not treat that as an established circular step. Overall, the core result is an empirical regression-transfer claim with independent content; no step reduces by construction to its own inputs.

Axiom & Free-Parameter Ledger

5 free parameters · 5 axioms · 0 invented entities

The central claim rests on a learned mapping from embedding-distribution statistics to accuracy; the free parameters are tunable evaluation and data-construction choices, and the axioms are domain assumptions about transferability and label quality. No invented entities (particles, forces, dimensions) are needed; SD_F/SD_M/SD_SW are computed descriptors, and FusionDataset is a corpus artifact rather than a postulated entity.

free parameters (5)
  • Meta-set size n = 30,000
    Chosen by cost–error tradeoff in Fig. 6; larger n improves MLP until saturation, so 30K is a hand-selected operating point rather than derived.
  • Sample-set size cap |S_i| = ≤10,000
    Budget-limited; Fig. 5 shows MAE decreases with |S_i|, but the cap is set by the $1,000 generation budget and per-database caps.
  • Hybrid SWD slices (k,R) = k=8–10, R=16–24
    Selected from ablation in §V-D; fewer slices reduce latency while preserving MAE, so the reported accuracy depends on this configuration choice.
  • MLP architecture/hyperparameters = hidden 256/128/64, lr=1e-4, dropout=0.2, AdamW, 20 epochs
    Standard tuning choices; early stopping is done on validation MAE from the benchmark suite used for reporting, so they are fit on the same targets.
  • Generation budget costs c_gen,c_val,c_exec = 0.00012, 0.00003, 0.0004 USD/sample
    Ad hoc unit costs in Eq. 16 used to keep total under B=$1,000; they determine how many validation/execution checks are performed and thus label quality.
axioms (5)
  • domain assumption Dataset-level accuracy of a Text2SQL model on a target workload is a learnable function of first/second-order and sliced-Wasserstein statistics of pooled embeddings of source vs target workloads.
    This is the core inductive premise of §IV-B: no proof is offered that these descriptors capture accuracy-relevant shift; the MLP g_θ is trained to approximate it.
  • domain assumption Synthetic FusionDataset shift conditions are representative of real-world Text2SQL distribution shifts, so g_θ transfers to real unseen benchmarks.
    Section V-A argues coverage via t-SNE/radar, but transferability from synthetic to real shifts is assumed; no formal guarantee or distribution-free bound is provided.
  • domain assumption Execution checks and GPT-5 judging in FusionDataset produce correct gold SQL and accuracy labels.
    Section IV-A says generated SQLs are filtered by execution and 'independent judges such as GPT-5'; if these are noisy, the training labels for g_θ are noisy.
  • domain assumption The labeled sample sets S_i used for meta-adaptation are available to the deployer for a new model.
    In §IV-C, adapting to m_new requires computing its true accuracy on labeled S_i; 'target evaluation remains label-free' only after this labeled adaptation.
  • domain assumption Last-layer pooled embeddings from different base Text2SQL models are sufficiently aligned for meta-training across model families.
    Section IV-C meta-trains across 20 models with 'shared backbone families', assuming cross-model representation alignment without quantitative alignment analysis.

pith-pipeline@v1.3.0-alltime-deepseek · 24060 in / 19349 out tokens · 174116 ms · 2026-08-02T18:34:35.280869+00:00 · methodology

0 comments
read the original abstract

Recent advances in large language models have strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL.

Figures

Figures reproduced from arXiv: 2603.07841 by Hongzhi Yin, Quoc Viet Hung Nguyen, Thanh Tam Nguyen, Trinh Pham, Viet Huynh.

Figure 1
Figure 1. Figure 1: Top: Existing Text2SQL evaluations rely on ground￾truth labels, which are often unavailable as databases evolve. Bottom: FusionSQL estimates model accuracy directly from unlabeled inputs without requiring ground-truth SQL labels. faster than annotation cycles, and manual labeling (drafting SQL, executing, debugging, and verifying) is both costly and time-consuming. Consequently, practitioners are forced in… view at source ↗
Figure 2
Figure 2. Figure 2: FusionSQL framework. Training: A frozen Text2SQL model encodes training and FusionDataset samples into embeddings to compute shift descriptors (SDF , SDM, SDSW ) for training the FusionSQL evaluator. Inference: For unseen, unlabeled workloads, the same descriptors are computed to estimate accuracy without labels or retraining. where h(·) denotes a deterministic function that extracts a fixed-length represe… view at source ↗
Figure 3
Figure 3. Figure 3: t-SNE coverage. Comparing 50K samples, Fusion￾Dataset bridges clusters of existing benchmarks in both do￾main (a) and question space (b), reflecting broader semantic and structural diversity of real-world Text2SQL variability. monotonic performance decline across tiers demonstrates that FusionDataset accurately captures a realistic gradient of rea￾soning complexity. Semantic Coverage [PITH_FULL_IMAGE:figu… view at source ↗
Figure 5
Figure 5. Figure 5: Sample-set size. Impact of sample-set size |Si | used to compute distribution shifts for an instance (Dtrain, Si). Error decreases as |Si | grows. The per-sample costs are c gen = 0.00012, c val = 0.00003, and c exec = 0.0004, and we budget for expected to￾tals P d∈D n gen d = 1.05N, P d∈D n val P d = 1.05N, and d∈D n exec d = 0.10N over N = 3,373,204 accepted pairs, yielding C ≈ N(1.05(c gen + c val) + 0.… view at source ↗
Figure 6
Figure 6. Figure 6: Meta-set size. FusionSQL’s MLP attains the lowest error and benefits most from larger meta-sets, while costs rise sharply beyond 30,000 with marginal gains. with larger meta-sets for all methods, but the MLP benefits most and continues to improve up to n=30K, whereas simpler models saturate earlier. Training cost increases gradually with n but rises sharply beyond n=30K, after which performance [PITH_FULL… view at source ↗
Figure 8
Figure 8. Figure 8: Latency–accuracy trade-off. Bars report average evaluation latency, while the red curve shows MAE (lower is better). FusionSQL offers the best trade-off, whereas judge￾based methods incur high latency without much MAE gains. Estimation Calibration. MAE remains the primary evalu￾ation metric because it directly measures the error between estimated and gold EX. However, EX curves reveal whether an estimator … view at source ↗
Figure 7
Figure 7. Figure 7: EX across transfers. Across six source–target transfers and five base models, FusionSQL closely follows the ground￾truth (GT) trend, while FusionSQL-TL exhibits mild bias on harder targets due to limited labeled adaptation. Evaluator Benchmark. As this setting has not been directly addressed before, we adapt representative label-free baselines, including confidence-based methods ATC, including its two vari… view at source ↗
Figure 9
Figure 9. Figure 9: Latency–accuracy trade-off on unseen Text2SQL models. Bars denote relative latency, while lines show MAE. FusionSQL-ML attains the lowest MAE with low latency. Latency [PITH_FULL_IMAGE:figures/full_fig_p009_9.png] view at source ↗
Figure 10
Figure 10. Figure 10: FusionSQL reliability and shift sensitivity. (a) FusionSQL accurately predicts EX. (b) Shift impacts EX on hard targets but only mildly on distant-but-easy transfers, demonstrating robustness. decoding and confidence statistics beyond the base Text2SQL generation. FusionSQL-TL and PseAutoEval incur extra la￾tency from acquiring a small set of human labels for transfer￾style calibration. FusionSQL-TL remai… view at source ↗
Figure 11
Figure 11. Figure 11: Latency + MAE vs. number of slices L. Latency scales linearly with L, while MAE stabilizes beyond L≈32. Hybrid SWD (red points) achieves low MAE with reduced L, optimizing efficiency without sacrificing fidelity. Scalability. We evaluate FusionSQL’s scalability by varying the number of SWD slices and measuring inference latency (seconds), memory usage, and accuracy [PITH_FULL_IMAGE:figures/full_fig_p011_… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 2 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

    cs.LG 2026-05 unverdicted novelty 7.0

    MetaEvaluator meta-learns an initialization from reference models to enable accurate, label-free performance estimation for unseen models across architectures and modalities.

  2. Learning to Evaluate: Cost-Effective Model Evaluation on Unlabeled Data with Meta-Learning

    cs.LG 2026-05 unverdicted novelty 6.0

    MetaEvaluator applies meta-learning over reference models to deliver label-free performance estimates for unseen models across architectures and modalities on unlabeled datasets.

Reference graph

Works this paper leans on

75 extracted references · 17 linked inside Pith · cited by 1 Pith paper

  1. [1]

    The dawn of natural language to sql: Are we fully ready?

    B. Li, Y . Luo, C. Chai, G. Li, and N. Tang, “The dawn of natural language to sql: Are we fully ready?”PVLDB, vol. 17, 2024

    2024

  2. [2]

    Seq2sql: Generating structured queries from natural language using reinforcement learning,

    V . Zhong, C. Xiong, and R. Socher, “Seq2sql: Generating structured queries from natural language using reinforcement learning,”arXiv preprint arXiv:1709.00103, 2017

    Pith/arXiv arXiv 2017

  3. [3]

    Sqlnet: Generating structured queries from natural language without reinforcement learning,

    X. Xu, C. Liu, and D. Song, “Sqlnet: Generating structured queries from natural language without reinforcement learning,”arXiv preprint arXiv:1711.04436, 2017

    Pith/arXiv arXiv 2017

  4. [4]

    TypeSQL: Knowledge- based type-aware neural text-to-SQL generation,

    T. Yu, Z. Li, Z. Zhang, R. Zhang, and D. Radev, “TypeSQL: Knowledge- based type-aware neural text-to-SQL generation,” inNAACL, 2018

    2018

  5. [5]

    SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task,

    T. Yu, M. Yasunaga, K. Yang, R. Zhang, D. Wang, Z. Li, and D. Radev, “SyntaxSQLNet: Syntax tree networks for complex and cross-domain text-to-SQL task,” inEMNLP, 2018

    2018

  6. [6]

    Editing-based SQL query generation for cross-domain context-dependent questions,

    R. Zhang, T. Yu, H. Er, S. Shim, E. Xue, X. V . Lin, T. Shi, C. Xiong, R. Socher, and D. Radev, “Editing-based SQL query generation for cross-domain context-dependent questions,” inEMNLP, 2019

    2019

  7. [7]

    To- wards complex text-to-SQL in cross-domain database with intermediate representation,

    J. Guo, Z. Zhan, Y . Gao, Y . Xiao, J.-G. Lou, T. Liu, and D. Zhang, “To- wards complex text-to-SQL in cross-domain database with intermediate representation,” inACL, 2019

    2019

  8. [8]

    RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers,

    B. Wang, R. Shin, X. Liu, O. Polozov, and M. Richardson, “RAT-SQL: Relation-aware schema encoding and linking for text-to-SQL parsers,” inACL, 2020

    2020

  9. [9]

    Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing,

    X. V . Lin, R. Socher, and C. Xiong, “Bridging textual and tabular data for cross-domain text-to-SQL semantic parsing,” inEMNLP, 2020

    2020

  10. [10]

    SmBoP: Semi-autoregressive bottom-up se- mantic parsing,

    O. Rubin and J. Berant, “SmBoP: Semi-autoregressive bottom-up se- mantic parsing,” inNAACL, 2021

    2021

  11. [11]

    LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations,

    R. Cao, L. Chen, Z. Chen, Y . Zhao, S. Zhu, and K. Yu, “LGESQL: Line graph enhanced text-to-SQL model with mixed local and non-local relations,” inACL, 2021

    2021

  12. [12]

    Gra{pp}a: Grammar-augmented pre-training for table semantic parsing,

    T. Yu, C.-S. Wu, X. V . Lin, B. Wang, Y . C. Tan, X. Yang, D. Radev, R. Socher, and C. Xiong, “Gra{pp}a: Grammar-augmented pre-training for table semantic parsing,” inICLR, 2021

    2021

  13. [13]

    PICARD: Parsing in- crementally for constrained auto-regressive decoding from language models,

    T. Scholak, N. Schucher, and D. Bahdanau, “PICARD: Parsing in- crementally for constrained auto-regressive decoding from language models,” inEMNLP, 2021

    2021

  14. [14]

    DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction,

    M. Pourreza and D. Rafiei, “DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction,” inNeurIPS, 2023

    2023

  15. [15]

    Text- to-sql empowered by large language models: A benchmark evaluation,

    D. Gao, H. Wang, Y . Li, X. Sun, Y . Qian, B. Ding, and J. Zhou, “Text- to-sql empowered by large language models: A benchmark evaluation,” PVLDB, vol. 17, 2024

    2024

  16. [16]

    Chess: Contextual harnessing for efficient sql synthesis,

    S. Talaei, M. Pourreza, Y .-C. Chang, A. Mirhoseini, and A. Saberi, “Chess: Contextual harnessing for efficient sql synthesis,”arXiv preprint arXiv:2405.16755, 2024

    Pith/arXiv arXiv 2024

  17. [17]

    How to answer secure and private SQL queries?

    Q. Luo, Q. Xu, and C. Yang, “How to answer secure and private SQL queries?” inICDE, 2025

    2025

  18. [18]

    Lampat: Low-rank adaption for multilingual paraphrasing using adversarial training,

    K. M. Le, T. Pham, T. Quan, and A. T. Luu, “Lampat: Low-rank adaption for multilingual paraphrasing using adversarial training,”AAAI, 2024

    2024

  19. [19]

    UniBridge: A unified approach to cross- lingual transfer learning for low-resource languages,

    T. Pham, K. Le, and A. T. Luu, “UniBridge: A unified approach to cross- lingual transfer learning for low-resource languages,” inACL, 2024

    2024

  20. [20]

    Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql,

    H. Li, J. Zhang, C. Li, and H. Chen, “Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql,” inAAAI, vol. 37, 2023

    2023

  21. [21]

    CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL,

    M. Pourreza, H. Li, R. Sun, Y . Chung, S. Talaei, G. T. Kakkar, Y . Gan, A. Saberi, F. Ozcan, and S. O. Arik, “CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL,” inICLR, 2025

    2025

  22. [22]

    Csc-sql: Corrective self-consistency in text-to- sql via reinforcement learning,

    L. Sheng and S.-S. Xu, “Csc-sql: Corrective self-consistency in text-to- sql via reinforcement learning,” inIJCNLP, 2025

    2025

  23. [23]

    Alpha- SQL: Zero-shot text-to-SQL using monte carlo tree search,

    B. Li, J. Zhang, J. Fan, Y . Xu, C. Chen, N. Tang, and Y . Luo, “Alpha- SQL: Zero-shot text-to-SQL using monte carlo tree search,” inICML, 2025

    2025

  24. [24]

    Grounding natural language to SQL translation with data-based self-explanations,

    Y . Fan, T. Ren, C. Huang, Z. He, and X. S. Wang, “Grounding natural language to SQL translation with data-based self-explanations,” inICDE, 2025

    2025

  25. [25]

    Machine learning inference pipeline execution using pure SQL based on operator fusion,

    Q. Pan, J. Zhi, C. Zhang, C. Xu, Z. Zhang, A. Shao, G. Bao, Q. Cui, X. Chen, and A. Zhou, “Machine learning inference pipeline execution using pure SQL based on operator fusion,” inICDE, 2025

    2025

  26. [26]

    AID-SQL: adaptive in-context learning of text-to-sql with difficulty-aware instruction and retrieval-augmented generation,

    X. Li, Q. Cai, Y . Shu, C. Guo, and B. Yang, “AID-SQL: adaptive in-context learning of text-to-sql with difficulty-aware instruction and retrieval-augmented generation,” inICDE, 2025

    2025

  27. [27]

    Agenticdata: An agentic data analytics system for heterogeneous data,

    J. Sun, G. Li, P. Zhou, Y . Ma, J. Xu, and Y . Li, “Agenticdata: An agentic data analytics system for heterogeneous data,”arXiv preprint arXiv:2508.05002, 2025

    Pith/arXiv arXiv 2025

  28. [28]

    RefoRCE: A text-to-SQL agent with self-refinement, format restriction, and column exploration,

    M. Deng, A. Ramachandran, C. Xu, L. Hu, Z. Yao, A. Datta, and H. Zhang, “RefoRCE: A text-to-SQL agent with self-refinement, format restriction, and column exploration,” inICLR, 2025

    2025

  29. [29]

    Omnisql: Synthesizing high-quality text-to-sql data at scale,

    H. Li, S. Wu, X. Zhang, X. Huang, J. Zhang, F. Jiang, S. Wang, T. Zhang, J. Chen, R. Shi, H. Chen, and C. Li, “Omnisql: Synthesizing high-quality text-to-sql data at scale,”PVLDB, vol. 18, 2025

    2025

  30. [30]

    Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents,

    K. T. Pham, T. H. Nguyen, J. Jo, Q. V . H. Nguyen, and T. T. Nguyen, “Multilingual text-to-sql: Benchmarking the limits of language models with collaborative language agents,” inADC, 2025

    2025

  31. [31]

    Jacobsson

    P. Jacobsson. Getting ai to write good SQL: Text-to-SQL techniques explained. [Online]. Available: https://cloud.google.com/blog/products/ databases/techniques-for-improving-text-to-sql

  32. [32]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,

    T. Yu, R. Zhang, K. Yang, M. Yasunaga, D. Wang, Z. Li, J. Ma, I. Li, Q. Yao, S. Roman, Z. Zhang, and D. Radev, “Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task,” inEMNLP, 2018

    2018

  33. [33]

    Sparc: Cross-domain semantic parsing in context,

    T. Yu, R. Zhang, M. Yasunaga, Y . C. T. Li, S. Er, E. Li, B. Li, B. Pang, X. V . Lin, Z. Y . Tanet al., “Sparc: Cross-domain semantic parsing in context,” inACL, 2019

    2019

  34. [34]

    Cosql: A conversational text- to-sql challenge towards cross-domain natural language interfaces to databases,

    T. Yu, R. Zhang, M. Yasunaga, B. P. Tan, X. V . Lin, S. Er, Y . C. Li, E. Li, B. Li, W.-t. Y . Chenet al., “Cosql: A conversational text- to-sql challenge towards cross-domain natural language interfaces to databases,” inEMNLP, 2019

    2019

  35. [35]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,

    J. Li, B. Hui, G. QU, J. Yang, B. Li, B. Li, B. Wang, B. Qin, R. Geng, N. Huo, X. Zhou, C. Ma, G. Li, K. Chang, F. Huang, R. Cheng, and Y . Li, “Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls,” inNeurIPS, 2023

    2023

  36. [36]

    Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows,

    F. Lei, J. Chen, Y . Ye, R. Cao, D. Shin, H. SU, Z. SUO, H. Gao, W. Hu, P. Yin, V . Zhong, C. Xiong, R. Sun, Q. Liu, S. Wang, and T. Yu, “Spider 2.0: Evaluating language models on real-world enterprise text-to-SQL workflows,” inICLR, 2025

    2025

  37. [37]

    Estimating large language model capabilities without labeled test data,

    Y . Fuet al., “Estimating large language model capabilities without labeled test data,” inEMNLP, 2023

    2023

  38. [38]

    SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,

    P. Manakul, A. Liusie, and M. Gales, “SelfCheckGPT: Zero-resource black-box hallucination detection for generative large language models,” inEMNLP, 2023

    2023

  39. [39]

    Detecting hallucinations in large language models using semantic entropy,

    S. Farquhar, J. Kossen, L. Kuhn, and Y . Gal, “Detecting hallucinations in large language models using semantic entropy,”Nature, 2024

    2024

  40. [40]

    Autoeval done right: Using synthetic data for model evaluation,

    P. Boyeau, A. N. Angelopoulos, T. Li, N. Yosef, J. Malik, and M. I. Jordan, “Autoeval done right: Using synthetic data for model evaluation,” inICML, 2025

    2025

  41. [41]

    Prediction-powered inference,

    A. N. Angelopoulos, S. Bates, C. Fannjiang, M. I. Jordan, and T. Zrnic, “Prediction-powered inference,”Science, 2023

    2023

  42. [42]

    Leveraging unlabeled data to predict out-of-distribution performance,

    S. Garg, S. Balakrishnan, Z. C. Lipton, B. Neyshabur, and H. Sedghi, “Leveraging unlabeled data to predict out-of-distribution performance,” inNeurIPS, 2021

    2021

  43. [43]

    Predicting with confidence on unseen distributions,

    D. Guillory, V . Shankar, S. Ebrahimi, T. Darrell, and L. Schmidt, “Predicting with confidence on unseen distributions,” inICCV, 2021

    2021

  44. [44]

    Nl2sql-bugs: A benchmark for detecting semantic errors in nl2sql translation,

    X. Liu, S. Shen, B. Li, N. Tang, and Y . Luo, “Nl2sql-bugs: A benchmark for detecting semantic errors in nl2sql translation,” inKDD, 2025

    2025

  45. [45]

    Judging llm-as-a-judge with mt-bench and chatbot arena,

    L. Zheng, W.-L. Chiang, Y . Sheng, S. Zhuang, Z. Wu, Y . Zhuang, Z. Lin, Z. Li, D. Li, E. P. Xing, H. Zhang, J. E. Gonzalez, and I. Stoica, “Judging llm-as-a-judge with mt-bench and chatbot arena,” inNeurIPS, 2023

    2023

  46. [46]

    Tablib: A dataset of 627m tables with context,

    G. Eggert, K. Huo, M. Biven, and J. Waugh, “Tablib: A dataset of 627m tables with context,”arXiv preprint arXiv:2310.07875, 2023

    Pith/arXiv arXiv 2023

  47. [47]

    KaggleDBQA: Realistic evaluation of text-to-SQL parsers,

    C.-H. Lee, O. Polozov, and M. Richardson, “KaggleDBQA: Realistic evaluation of text-to-SQL parsers,” inACL, 2021

    2021

  48. [48]

    SQLForge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs,

    Y . Guo, D. Jin, S. Ye, S. Chen, J. Yang, and X. Tan, “SQLForge: Synthesizing reliable and diverse data to enhance text-to-SQL reasoning in LLMs,” inACL, 2025

    2025

  49. [49]

    Parsql: Enhancing text-to-sql through sql parsing and reasoning,

    Y . Dai, H. Yang, M. Hao, and P. Chao, “Parsql: Enhancing text-to-sql through sql parsing and reasoning,” inACL, 2025

    2025

  50. [50]

    Llm-based data synthesis and distillation for high-quality text-to-sql training,

    S. Cui, K. Wen, B. Sang, T. Li, Y . Zhang, and H. Gao, “Llm-based data synthesis and distillation for high-quality text-to-sql training,” inICIC, 2025

    2025

  51. [51]

    On efficient multilevel clustering via wasserstein distances,

    V . Huynh, N. Ho, N. Dam, X. Nguyen, M. Yurochkin, H. Bui, and D. Phung, “On efficient multilevel clustering via wasserstein distances,” Journal of Machine Learning Research, 2021

    2021

  52. [52]

    Probabilistic multilevel clustering via composite transportation distance

    N. Ho, V . Huynh, D. Phung, and M. Jordan, “Probabilistic multilevel clustering via composite transportation distance.” PMLR, 2019

    2019

  53. [53]

    Multilevel clustering via Wasserstein means,

    N. Ho, X. Nguyen, M. Yurochkin, H. H. Bui, V . Huynh, and D. Phung, “Multilevel clustering via Wasserstein means,” inICML, 2017

    2017

  54. [54]

    On first-order meta-learning algorithms,

    A. Nichol, J. Achiam, and J. Schulman, “On first-order meta-learning algorithms,”arXiv preprint arXiv:1803.02999, 2018

    Pith/arXiv arXiv 2018

  55. [55]

    Model-agnostic meta-learning for fast adaptation of deep networks,

    C. Finn, P. Abbeel, and S. Levine, “Model-agnostic meta-learning for fast adaptation of deep networks,” inICML, 2017

    2017

  56. [56]

    Sciencebenchmark: A complex real-world bench- mark for evaluating natural language to sql systems,

    Y . Zhang, J. Deriu, G. Katsogiannis-Meimarakis, C. Kosten, G. Koutrika, and K. Stockinger, “Sciencebenchmark: A complex real-world bench- mark for evaluating natural language to sql systems,”PVLDB, vol. 17, 2024

    2024

  57. [57]

    Ehrsql: A practical text-to-sql benchmark for electronic health records,

    G. Lee, H. Hwang, S. Bae, Y . Kwon, W. Shin, S. Yang, M. Seo, J.- Y . Kim, and E. Choi, “Ehrsql: A practical text-to-sql benchmark for electronic health records,” inNeurIPS, vol. 35, 2022

    2022

  58. [58]

    Gnnevaluator: Evaluating gnn performance on unseen graphs without labels,

    X. Zheng, M. Zhang, C. Chen, S. Molaei, C. Zhou, and S. Pan, “Gnnevaluator: Evaluating gnn performance on unseen graphs without labels,” inNeurIPS, vol. 36, 2023

    2023

  59. [59]

    Qwen3 technical report,

    A. Yang, A. Li, B. Yang, B. Zhang, B. Hui, B. Zheng, B. Yu, C. Gao, C. Huang, C. Lvet al., “Qwen3 technical report,”arXiv preprint arXiv:2505.09388, 2025

    Pith/arXiv arXiv 2025

  60. [60]

    The llama 3 herd of models,

    A. Grattafiori, A. Dubey, A. Jauhri, A. Pandey, A. Kadian, A. Al-Dahle, A. Letman, A. Mathur, A. Schelten, A. Vaughanet al., “The llama 3 herd of models,”arXiv preprint arXiv:2407.21783, 2024

    Pith/arXiv arXiv 2024

  61. [61]

    Deepseek-coder: When the large language model meets programming–the rise of code intelligence,

    D. Guo, Q. Zhu, D. Yang, Z. Xie, K. Dong, W. Zhang, G. Chen, X. Bi, Y . Wu, Y . Liet al., “Deepseek-coder: When the large language model meets programming–the rise of code intelligence,”arXiv preprint arXiv:2401.14196, 2024

    Pith/arXiv arXiv 2024

  62. [62]

    Xiyan-sql: A novel multi-generator framework for text- to-sql,

    Y . Liu, Y . Zhu, Y . Gao, Z. Luo, X. Li, X. Shi, Y . Hong, J. Gao, Y . Li, B. Dinget al., “Xiyan-sql: A novel multi-generator framework for text- to-sql,”arXiv preprint arXiv:2507.04701, 2025

    Pith/arXiv arXiv 2025

  63. [63]

    Mixtral of experts,

    A. Q. Jiang, A. Sablayrolles, A. Roux, A. Mensch, B. Savary, C. Bam- ford, D. S. Chaplot, D. d. l. Casas, E. B. Hanna, F. Bressandet al., “Mixtral of experts,”arXiv preprint arXiv:2401.04088, 2024

    Pith/arXiv arXiv 2024

  64. [64]

    Starcoder 2 and the stack v2: The next generation,

    A. Lozhkov, R. Li, L. B. Allal, F. Cassano, J. Lamy-Poirier, N. Tazi, A. Tang, D. Pykhtar, J. Liu, Y . Weiet al., “Starcoder 2 and the stack v2: The next generation,”arXiv preprint arXiv:2402.19173, 2024

    Pith/arXiv arXiv 2024

  65. [65]

    Code llama: Open foundation models for code,

    B. Roziere, J. Gehring, F. Gloeckle, S. Sootla, I. Gat, X. E. Tan, Y . Adi, J. Liu, R. Sauvestre, T. Remezet al., “Code llama: Open foundation models for code,”arXiv preprint arXiv:2308.12950, 2023

    Pith/arXiv arXiv 2023

  66. [66]

    Phi- 4 technical report,

    M. Abdin, J. Aneja, H. Behl, S. Bubeck, R. Eldan, S. Gunasekar, M. Harrison, R. J. Hewett, M. Javaheripi, P. Kauffmannet al., “Phi- 4 technical report,”arXiv preprint arXiv:2412.08905, 2024

    Pith/arXiv arXiv 2024

  67. [67]

    Yi: Open foundation models by 01. ai,

    A. Young, B. Chen, C. Li, C. Huang, G. Zhang, G. Zhang, G. Wang, H. Li, J. Zhu, J. Chenet al., “Yi: Open foundation models by 01. ai,” arXiv preprint arXiv:2403.04652, 2024

    Pith/arXiv arXiv 2024

  68. [68]

    Gemma: Open models based on gemini research and technology,

    G. Team, T. Mesnard, C. Hardin, R. Dadashi, S. Bhupatiraju, S. Pathak, L. Sifre, M. Rivi `ere, M. S. Kale, J. Loveet al., “Gemma: Open models based on gemini research and technology,”arXiv preprint arXiv:2403.08295, 2024

    Pith/arXiv arXiv 2024

  69. [69]

    Internlm2 technical report,

    Z. Cai, M. Cao, H. Chen, K. Chen, K. Chen, X. Chen, X. Chen, Z. Chen, Z. Chen, P. Chuet al., “Internlm2 technical report,”arXiv preprint arXiv:2403.17297, 2024

    Pith/arXiv arXiv 2024

  70. [70]

    Athena: An ontology-driven system for natural language querying over relational data stores,

    D. Saha, A. Floratou, K. Sankaranarayanan, U. F. Minhas, A. R. Mittal, and F. Ozcan, “Athena: An ontology-driven system for natural language querying over relational data stores,”PVLDB, vol. 9, 2016

    2016

  71. [71]

    Athena++: Natural language querying for complex nested SQL queries,

    J. Sen, C. Lei, A. Quamar, F. Ozcan, V . Efthymiou, A. Dalmia, G. Stager, A. Mittal, D. Saha, and K. Sankaranarayanan, “Athena++: Natural language querying for complex nested SQL queries,”PVLDB, vol. 13, 2020

    2020

  72. [72]

    Sqlizer: Query synthesis from natural language,

    N. Yaghmazadeh, Y . Wang, I. Dillig, and T. Dillig, “Sqlizer: Query synthesis from natural language,”Proc. ACM Program. Lang., vol. 1, 2017

    2017

  73. [73]

    Don’t judge a language model by its last layer: Contrastive learning with layer-wise attention pooling,

    D. Oh, Y . Kim, H. Lee, H. H. Huang, and H. Lim, “Don’t judge a language model by its last layer: Contrastive learning with layer-wise attention pooling,” inCOLING, 2022

    2022

  74. [74]

    SimCSE: Simple contrastive learning of sentence embeddings,

    T. Gao, X. Yao, and D. Chen, “SimCSE: Simple contrastive learning of sentence embeddings,” inEMNLP, 2021

    2021

  75. [75]

    PromptBERT: Improving BERT sentence embeddings with prompts,

    T. Jiang, J. Jiao, S. Huang, Z. Zhang, D. Wang, F. Zhuang, F. Wei, H. Huang, D. Deng, and Q. Zhang, “PromptBERT: Improving BERT sentence embeddings with prompts,” inEMNLP, 2022

    2022