Pith. sign in

REVIEW 5 major objections 5 minor 28 references

Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL

T0 review · 5 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read An LLM-based pipeline can judge SQL semantic equivalence with high accuracy, lifting equivalent-pair detection from 61.25% to 95% on a synthetic test set.

desk verdict Useful practitioner taxonomy, but the headline accuracy claim is not established because the datasets and prompts are too tightly coupled to the authors' own definitions. read the letter →

arxiv 2506.09359 v1 pith:5VUGRL5X submitted 2025-06-11 cs.CL

Qingyun Zeng , Simin Ma , Arash Niknafs , Ashish Basran , Carol Szabo This is my paper
classification cs.CL
keywords SQLsemanticequivalencetext-to-SQLevaluationLLM-as-judgeexecutionaccuracypromptengineeringqueryrewritingMiniature&Mullnatural-language-to-SQL
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Deciding whether two SQL queries are semantically equivalent is a hard, often undecidable problem, and common metrics like execution accuracy misjudge many cases due to sparse test data or cosmetic differences. This paper argues that large language models, especially GPT-4, can judge SQL equivalence accurately when wrapped in a carefully structured pipeline. It characterizes common patterns of equivalent and inequivalent SQL, builds three test sets, and shows that adding subquery rewriting and a Miniature & Mull prompting strategy raises the passing rate on equivalent synthetic pairs from 61.25% to 95%, with a modest drop on inequivalent pairs. If correct, this gives text-to-SQL systems a practical LLM-based evaluator that can complement or replace execution-based checks.

What carries the argument

The load-bearing mechanism is a staged evaluation pipeline. First, both queries are preprocessed to standardize formatting; exact match and exact set match string checks then settle unambiguous cases cheaply. Remaining pairs go to an LLM judge (GPT-4) prompted with a task definition, the database context, optional schema and execution results, and few-shot Chain-of-Thought examples, run multiple times with majority voting to stabilize judgments. The improved pipeline adds a module that rewrites subqueries into LEFT JOINs and uses the Miniature & Mull prompting strategy, which asks the model to simulate executing both queries on a small example database, then alter the database and re-execute to search for a counterexample.

What would settle it

Have a panel of SQL experts who have not seen the paper's taxonomy independently label a fresh set of equivalent and inequivalent query pairs drawn from real text-to-SQL outputs, then run the improved pipeline on that set; if its accuracy falls well below the reported 95% and 83.75%, the claim that the pipeline achieves high accuracy on SQL equivalence in general is refuted.

Watch

Extended reading notes

Core claim

The central claim is that LLMs, particularly GPT-4, can achieve high accuracy in assessing SQL semantic and weak-semantic equivalence. On a synthetic dataset built around the paper's own taxonomy of equivalence patterns, the improved pipeline pushes the equivalent-pair passing rate from 61.25% to 95%, while the inequivalent-pair rate falls slightly from 90% to 83.75%, a trade-off the authors attribute to the aggressiveness of query rewriting. On the real-world business dataset, the basic pipeline records high precision and recall on equivalent pairs (0.95 precision, 0.89 recall) and weaker precision on inequivalent pairs (0.64), which the authors consider acceptable because incorrectly generated SQL is rarer in production use.

Load-bearing premise

The ground-truth labels for the Dataverse and synthetic datasets were assigned by the authors using their own equivalence criteria, and the LLM prompts encode the same criteria, so the reported accuracy measures agreement with those criteria rather than with an independently verified notion of semantic equivalence.

Editorial extensions

If this is right

  • An LLM-based judge with preprocessing, string checks, and multi-run voting can serve as a practical alternative to execution accuracy for text-to-SQL evaluation, reducing false positives from sparse test data and false negatives from cosmetic differences.
  • The 95% equivalent-pair passing rate indicates that with prompt engineering and query rewriting, GPT-4 can reliably recognize diverse SQL formulations such as join versus subquery, distinct versus group by, and EXISTS versus JOIN.
  • The drop on inequivalent pairs from 90% to 83.75% shows that automatic query rewriting can hide real differences, so such transformations should be applied selectively or with a fallback.
  • The paper's catalog of equivalent and inequivalent SQL patterns gives builders a reusable checklist for creating targeted evaluation sets and for debugging why a judge fails on specific constructs.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the synthetic pairs were generated from the same pattern taxonomy that the prompts encode, the reported 95% figure may overstate accuracy on pairs drawn from unseen patterns or real production failures; an independent test set would be needed to verify generalization.
  • The five-category output (equivalent, minor difference, significant difference, not equivalent, undetermined) could be developed into a graded equivalence score, which would fit the paper's weak-equivalence notion better than a binary label.
  • A public, expert-labeled SQL equivalence benchmark with hard distractors would let the community compare LLM judges, prompting strategies, and rewriting methods in a controlled way.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. The paper proposes an LLM-based pipeline for assessing semantic and 'weak' semantic equivalence of SQL query pairs, motivated by the difficulties of evaluating Text-to-SQL outputs. The authors characterize a taxonomy of equivalent and inequivalent SQL patterns (Section 4, Appendices B and C), build three datasets (a manually labeled Dataverse set of 77 pairs, a 14-pair development set of failure cases, and a synthetic dataset of 160 pairs), and design a pipeline combining preprocessing, string-based matching, and multi-run GPT-4 prompting. The headline quantitative result is that an improved pipeline with subquery rewriting and the Miniature & Mull prompting strategy raises the equivalent-pair passing rate on the synthetic dataset from 61.25% to 95%, with a drop on inequivalent pairs from 90% to 83.75% (Table 4). The paper concludes that LLMs, particularly GPT-4, can achieve high accuracy in assessing SQL equivalence (Section 9).

Significance. If the reported results were externally validated, the paper would make a practical contribution to NL2SQL evaluation, particularly for business applications where relaxed equivalence is important. The structured taxonomy of equivalence/inequivalence patterns in Appendix B and the modular pipeline design (preprocessing, string-based short-circuiting, multi-run majority voting) are reusable and will be of interest to practitioners. The paper also correctly identifies known limitations of execution accuracy as an evaluation proxy. However, the central empirical claim is not currently established: the synthetic ground truth is generated from the authors' own taxonomy, the prompts encode the same criteria used to create that taxonomy, and all reported numbers are point estimates on small datasets with no confidence intervals or significance tests, so the evidence primarily demonstrates self-consistency rather than independent correctness.

major comments (5)
  1. [§6.3, §8.2.3, Appendix D.2] The synthetic dataset in §6.3 is constructed from the authors' own pattern taxonomy (Section 4, Appendix B), and the improved prompt template in Appendix D.2 feeds the LLM the 'detailed criteria of equivalence' that were used to define the ground-truth labels. Table 4 therefore measures how well the pipeline follows the authors' criteria, not how accurately it judges SQL equivalence against an independent standard. The claim in Section 9 that LLMs 'can achieve high accuracy in assessing SQL equivalence' is not supported without an external reference (e.g., independent human annotations on a held-out set, or formal verification such as Cosette on a subset of pairs).
  2. [§8.2.2] The dev set (Section 6.2) was explicitly built from failure cases of the pipeline, and the improved criteria were tuned on it. Reporting 92.9% accuracy on the same dev set is not a valid evaluation. The sentence stating that the pipeline achieved '100% precision and recall on the original testing data' is ambiguous and, if the Dataverse pairs were used during iterative refinement, makes Table 2 optimistically biased. The paper must separate tuning, validation, and test sets and report results on a truly held-out test set.
  3. [Table 4] Results are point estimates on 80 equivalent and 80 inequivalent pairs with no confidence intervals or significance tests. The regression for inequivalent pairs (90% to 83.75%) is a difference of 5 pairs out of 80, which is plausibly within sampling noise. Please report confidence intervals (e.g., Wilson intervals) and apply a paired test (e.g., McNemar) when comparing the initial and new pipelines.
  4. [Definition 3.5] 'Weak equivalence' is operationalized so loosely ('most likely produce the same results given the database in practical use, or if minor, trivial edits... according to user intent') that the ground-truth labels in §6.1 and §6.3 are not reproducibly checkable. The paper should provide a concrete annotation protocol, decision rules, and inter-annotator agreement (e.g., Cohen's kappa) to show that the labels are not just the authors' subjective judgments.
  5. [Algorithm 2 / §8.2.3] The improved pipeline rewrites subqueries to LEFT JOINs, but the paper does not report how often this rewrite preserves semantics in the tested dialect. If the rewrite is unsound for some patterns, it could explain both the improvement on equivalent pairs and the regression on inequivalent pairs, confounding the comparison of the two pipelines. Please report rewrite fidelity per pattern (e.g., the fraction of rewritten queries verified equivalent by an external mechanism).
minor comments (5)
  1. [Throughout] There are numerous typos and grammatical errors (e.g., 'chanllenge', 'notioced', 'acutally', 'acheive', 'accessment') that should be corrected before publication.
  2. [Section 7 / Table 4] Section 7 mentions GPT-4-0314 and gpt-4-32k-0613, while Table 4's caption says 'both use GPT-4o'; please clarify which model(s) produced each reported result.
  3. [§6.3 / Table 4] Table 4 does not report the breakdown of the synthetic datasets (the 60/20 split mentioned in §6.3); please report per-subset performance or explain why the split is irrelevant.
  4. [References] The reference list contains duplicates: Pourreza et al. 2024a and 2024b have the same title 'Chase-sql: Multi-path reasoning...'; please merge or correct.
  5. [Appendix A.2.1] The false-negative example assumes that EX implicitly orders results; in many standard implementations EX sorts the results before comparison, so this example may be misleading without a precise definition of the EX variant used.

Circularity Check

2 steps flagged · score 6.0 of 10

Synthetic-dataset improvement is fit-to-test and shares its taxonomy with the prompts; Dataverse results give partial independent support.

  1. fitted input called prediction [Section 8.2.3, Table 4]
    "We initially evaluated an improved evaluation pipeline (Algorithm 1) on the synthetic data. Based on the failure cases from this initial evaluation, we observed that many issues arose from subqueries... This led us to experiment with a query-rewrite module, which rewrites subqueries (if they appear) into queries with left joins. ... The combined effect of these enhancements (query-rewrite and Miniature & Mull prompting, as described in Algorithm 2) on the synthetic dataset is shown in Table 4."

    The failure cases that motivated the query-rewrite and Miniature & Mull additions come from the same synthetic dataset whose Table 4 numbers are then reported as the improvement (61.25% to 95%). This is fitting the pipeline to the test set and then presenting the result on that same set as a performance gain. The development set (Section 6.2) was likewise built from failure cases the pipeline 'cannot get it right' and was used to tune 'more clear definition of equivalence criteria' and 'more detailed criteria on the grading' (Section 8.2.2). The reported improvement is therefore a measure of fit to the test set and its failure taxonomy, not an out-of-sample prediction.

  2. self definitional [Sections 4.3, 6.3, Appendix D.2]
    "The characterization of equivalent and inequivalent SQL patterns detailed in Sections 4.1 and 4.2 directly informs the construction of targeted evaluation datasets. ... The criteria of equivalence of two SQL queries are defined as follows: [detailed criteria of equivalence]"

    The synthetic pairs are constructed from the paper's own equivalence taxonomy, and the same taxonomy is inserted into the LLM prompt as the 'detailed criteria of equivalence.' The ground-truth labels and the judging rubric therefore have a single author-defined source; high accuracy on the synthetic set demonstrates that GPT-4 can apply the authors' criteria to pairs generated from those criteria, rather than validating the criteria against an independent standard of SQL equivalence.

full rationale

The paper's strongest quantitative result—the synthetic-dataset improvement from 61.25% to 95% equivalent-pair passing rate (Table 4)—is not an independent, out-of-sample measurement. The new pipeline components (query rewriting, Miniature & Mull) were introduced after examining failure cases on the same synthetic dataset (Section 8.2.3), and the development set was built from failure cases and used to tune the equivalence criteria and grading (Sections 6.2, 8.2.2). In addition, the synthetic ground truth was constructed from the paper's own equivalence taxonomy (Sections 4.3, 6.3), and the refined prompts embed the same taxonomy as 'detailed criteria of equivalence' (Appendix D.2). The reported 95% therefore largely measures self-consistency: can GPT-4 apply the authors' criteria to pairs generated from those criteria? This is partial circularity, not total: the manually labeled Dataverse dataset (Section 6.1) provides some independent content, since the LLM was not given those labels, and the qualitative error analysis is informative. However, the central headline improvement reduces to test-set fitting and shared definition, so the claim that 'LLMs, particularly GPT-4, can achieve high accuracy in assessing SQL equivalence' is established only relative to the authors' own criteria and failure taxonomy. Definition 3.5's weak-equivalence notion is also deliberately vague, which makes the author-defined labels hard to audit against an external standard.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper introduces no physical or conceptual entities, and it uses no fitted numeric constants from data. Its load-bearing choices are the hand-defined equivalence criteria, the multi-run settings, and the assumption that its own labels are correct.

free parameters (2)
  • Multi-run count and majority vote threshold = 3 and 5 runs
    Default runs=3, max runs=5, plus majority voting for unstable cases. These are hand-chosen hyperparameters that affect the stability and accuracy results.
  • Equivalence criteria definitions = Custom
    The 'detailed criteria of equivalence' used for labeling ground truth and for prompting are hand-authored. They define what counts as equivalent and directly shape all reported accuracies.
assumptions (3)
  • ad hoc to paper Manual ground-truth labels in Section 6.1 are correct, and the synthetic labels in Section 6.3 are correct and complete representations of SQL equivalence patterns.
    The evaluation rests on this assumption. If the labels are biased, all accuracy numbers are off.
  • domain assumption LLM judgments can be treated as proxies for semantic equivalence under arbitrary database states.
    The paper assumes that prompt-based reasoning (including imagined database instances in Miniature and Mull) is a reliable way to decide an undecidable property.
  • ad hoc to paper Pattern taxonomy is exhaustive and dialect-appropriate.
    The synthetic dataset is created from the taxonomy, so if the taxonomy misses common patterns, the benchmark overstates generality.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL." pith.science (2026). https://pith.science/paper/5VUGRL5X

@misc{pith2026250609359,
  author       = {Pith},
  title        = {Pith review of: Taming SQL Complexity: LLM-Based Equivalence Evaluation for Text-to-SQL},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/5VUGRL5X}},
  note         = {Machine review of arXiv:2506.09359}
}
read the original abstract

The rise of Large Language Models (LLMs) has significantly advanced Text-to-SQL (NL2SQL) systems, yet evaluating the semantic equivalence of generated SQL remains a challenge, especially given ambiguous user queries and multiple valid SQL interpretations. This paper explores using LLMs to assess both semantic and a more practical "weak" semantic equivalence. We analyze common patterns of SQL equivalence and inequivalence, discuss challenges in LLM-based evaluation.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

28 extracted references · 13 canonical work pages

  1. [1]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION format.date year duplicate empty "emp...

  2. [2]

    Open-sourcing sql eval: Making benchmarking easier for sql generation models, November 2024

    AI, D. Open-sourcing sql eval: Making benchmarking easier for sql generation models, November 2024. URL https://defog.ai/blog/open-sourcing-sqleval. Accessed: 2024-12-03

    work page 2024

  3. [3]

    G., Kandikonda, Y

    Ascoli, B. G., Kandikonda, Y. S. R., and Choi, J. D. Etm: Modern insights into perspective on text-to-sql evaluation in the age of large language models, 2025. URL https://arxiv.org/abs/2407.07313

    arXiv 2025

  4. [4]

    and Lee, H.-y

    Chiang, C.-H. and Lee, H.-y. Can large language models be an alternative to human evaluations? In Proceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pp.\ 15607--15631, Toronto, Canada, July 2023. Association for Computational Linguistics. URL https://aclanthology.org/2023.acl-long.870

    work page 2023

  5. [5]

    Cosette: An automated prover for sql

    Chu, S., Wang, C., Weitz, K., and Cheung, A. Cosette: An automated prover for sql. In Conference on Innovative Data Systems Research, 2017. URL https://api.semanticscholar.org/CorpusID:12408033

    work page 2017

  6. [6]

    Axiomatic foundations and algorithms for deciding semantic equivalences of sql queries

    Chu, S., Cheung, A., and Suciu, D. Axiomatic foundations and algorithms for deciding semantic equivalences of sql queries. Proc. VLDB Endow., 11: 0 1482--1495, 2018. URL https://api.semanticscholar.org/CorpusID:44183892

    work page 2018

  7. [7]

    sql-eval, n.d

    defog ai. sql-eval, n.d. URL https://github.com/defog-ai/sql-eval. Accessed: 2024-12-03

    work page 2024

  8. [8]

    and Singh, M

    Dhinakaran, A. and Singh, M. Text-to-SQL : evaluating SQL generation with LLM as a judge, 2024. URL https://arize.com/blog/text-to-sql-evaluating-sql-generation-with-llm-as-a-judge/. Accessed: 2024-12-03

    work page 2024

Show all 28 references
  1. [9]

    Proving query equivalence using linear integer arithmetic

    Ding, H., Wang, Z., Yang, Y., Zhang, D., Xu, Z., Chen, H., Piskac, R., and Li, J. Proving query equivalence using linear integer arithmetic. Proc. ACM Manag. Data, 1 0 (4), December 2023. doi:10.1145/3626768. URL https://doi.org/10.1145/3626768

    2023 doi

  2. [10]

    A survey on llm-as-a-judge, 2025

    Gu, J., Jiang, X., Shi, Z., Tan, H., Zhai, X., Xu, C., Li, W., Shen, Y., Ma, S., Liu, H., Wang, S., Zhang, K., Wang, Y., Gao, W., Ni, L., and Guo, J. A survey on llm-as-a-judge, 2025. URL https://arxiv.org/abs/2411.15594

    2025 arXiv

  3. [11]

    FLEX : Expert-level false-less EX ecution metric for text-to- SQL benchmark

    Kim, H., Taeyang, J., Choi, S., Choi, S., and Cho, H. FLEX : Expert-level false-less EX ecution metric for text-to- SQL benchmark. In Chiruzzo, L., Ritter, A., and Wang, L. (eds.), Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for...

    2025 doi

  4. [12]

    Quantifying semantic query similarity for automated linear sql grading: A graph-based approach, 2024

    Köberlein, L., Probst, D., and Lenz, R. Quantifying semantic query similarity for automated linear sql grading: A graph-based approach, 2024. URL https://arxiv.org/abs/2403.14441

    2024 arXiv

  5. [13]

    Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls

    Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Geng, R., Huo, N., et al. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36, 2024

    2024

  6. [14]

    D e T riever: Decoder-representation-based retriever for improving NL 2 SQL in-context learning

    Li, R., Feng, Y., Fan, Z., Carenini, G., Zhang, W., Pourreza, M., and Zhang, Y. D e T riever: Decoder-representation-based retriever for improving NL 2 SQL in-context learning. In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Schockaert, S. (eds.)...

    2025

  7. [15]

    Calibrating LLM -based evaluator

    Liu, Y., Yang, T., Huang, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., and Zhang, Q. Calibrating LLM -based evaluator. In Calzolari, N., Kan, M.-Y., Hoste, V., Lenci, A., Sakti, S., and Xue, N. (eds.), Proceedings of the 2024 Joint International Conference on Computat...

    2024

  8. [16]

    HD -eval: Aligning large language model evaluators through hierarchical criteria decomposition

    Liu, Y., Yang, T., Huang, S., Zhang, Z., Huang, H., Wei, F., Deng, W., Sun, F., and Zhang, Q. HD -eval: Aligning large language model evaluators through hierarchical criteria decomposition. In Ku, L.-W., Martins, A., and Srikumar, V. (eds.), Proceedings of the 62nd Annual Meet...

    2024 doi

  9. [17]

    Hybrid ranking network for text-to-sql, 2020

    Lyu, Q., Chakrabarti, K., Hathi, S., Kundu, S., Zhang, J., and Chen, Z. Hybrid ranking network for text-to-sql, 2020. URL https://arxiv.org/abs/2008.04759

    2020 arXiv

  10. [18]

    and Rafiei, D

    Pourreza, M. and Rafiei, D. Din-sql: decomposed in-context learning of text-to-sql with self-correction. In Proceedings of the 37th International Conference on Neural Information Processing Systems, NIPS '23, Red Hook, NY, USA, 2024 a . Curran Associates Inc

    2024

  11. [19]

    and Rafiei, D

    Pourreza, M. and Rafiei, D. DTS - SQL : Decomposed text-to- SQL with small large language models. In Al-Onaizan, Y., Bansal, M., and Chen, Y.-N. (eds.), Findings of the Association for Computational Linguistics: EMNLP 2024, pp.\ 8212--8220, Miami, Florida, USA, November 2024 b...

    2024 doi

  12. [21]

    T., Gan, Y., Saberi, A., Ozcan, F., and Arik, S

    Pourreza, M., Li, H., Sun, R., Chung, Y., Talaei, S., Kakkar, G. T., Gan, Y., Saberi, A., Ozcan, F., and Arik, S. O. Chase-sql: Multi-path reasoning and preference optimized candidate selection in text-to-sql, 2024 b . URL https://arxiv.org/abs/2410.01943

    2024 arXiv

  13. [22]

    Sql-encoder: Improving nl2sql in-context learning through a context-aware encoder, 2024 c

    Pourreza, M., Rafiei, D., Feng, Y., Li, R., Fan, Z., and Zhang, W. Sql-encoder: Improving nl2sql in-context learning through a context-aware encoder, 2024 c . URL https://arxiv.org/abs/2403.16204

    2024 arXiv

  14. [23]

    Pourreza, M., Sun, R., Li, H., Miculicich, L., Pfister, T., and Arik, S. O. Sql-gen: Bridging the dialect gap for text-to-sql via synthetic data and model merging, 2024 d . URL https://arxiv.org/abs/2408.12733

    2024 arXiv

  15. [24]

    Chess: Contextual harnessing for efficient sql synthesis, 2024

    Talaei, S., Pourreza, M., Chang, Y.-C., Mirhoseini, A., and Saberi, A. Chess: Contextual harnessing for efficient sql synthesis, 2024. URL https://arxiv.org/abs/2405.16755

    2024 arXiv

  16. [25]

    Is C hat GPT a good NLG evaluator? a preliminary study

    Wang, J., Liang, Y., Meng, F., Sun, Z., Shi, H., Li, Z., Xu, J., Qu, J., and Zhou, J. Is C hat GPT a good NLG evaluator? a preliminary study. In Dong, Y., Xiao, W., Wang, L., Liu, F., and Carenini, G. (eds.), Proceedings of the 4th New Frontiers in Summarization Workshop, pp.\...

    2023 doi

  17. [26]

    S pider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to- SQL task

    Yu, T., Zhang, R., Yang, K., Yasunaga, M., Wang, D., Li, Z., Ma, J., Li, I., Yao, Q., Roman, S., Zhang, Z., and Radev, D. S pider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to- SQL task. In Riloff, E., Chiang, D., Hockenmaier, J...

    2018

  18. [27]

    Towards database-free text-to- SQL evaluation: A graph-based metric for functional correctness

    Zhan, Y., Cui, L., Weng, H., Wang, G., Tian, Y., Liu, B., Yang, Y., Yin, X., Xie, J., and Sun, Y. Towards database-free text-to- SQL evaluation: A graph-based metric for functional correctness. In Rambow, O., Wanner, L., Apidianaki, M., Al-Khalifa, H., Eugenio, B. D., and Scho...

    2025

  19. [28]

    Self-tuning: Instructing llms to effectively acquire new knowledge through self-teaching, 2024

    Zhang, X., Peng, B., Tian, Y., Zhou, J., Zhang, Y., Mi, H., and Meng, H. Self-tuning: Instructing llms to effectively acquire new knowledge through self-teaching, 2024. URL https://arxiv.org/abs/2406.06326

    2024 arXiv

  20. [29]

    Zhao, F., Lim, L., Ahmad, I., Agrawal, D., and Abbadi, A. E. Llm-sql-solver: Can llms determine sql equivalence?, 2024. URL https://arxiv.org/abs/2312.10321

    2024 arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.