Pith. sign in

REVIEW 4 major objections 5 minor 38 references

Effectiveness of Prompt Optimization in NL2SQL Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper argues that NL2SQL systems can replace per-query retrieval with a static, optimized prompt of exemplars, and that an iterative prompt optimizer achieves the best tested accuracy on the BIRD dev benchmark while also enabling…

desk verdict A credible preliminary study of static prompt optimization for NL2SQL, but the paper's production-oriented claim that static exemplars can replace retrieval-based selection is untested because no retrieval baseline appears anywhere in the experiments. read the letter →

arxiv 2505.20591 v1 pith:OFY23FVD submitted 2025-05-26 cs.CL cs.DB

Sairam Gurajada , Eser Kandogan , Sajjadur Rahman This is my paper
classification cs.CLcs.DB
keywords NL2SQLpromptoptimizationiterativeexemplarselectionin-contextlearningmulti-objectiveSQLexecutionlatencyBIRDbenchmark
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Most NL2SQL systems pick context per query by retrieving similar examples, which costs inference time and can overfit to query similarity. The authors argue that production systems instead need a fixed, carefully chosen exemplar set that captures the domain's query patterns, SQL constructs, and execution latencies. They test this by treating exemplar selection and instruction text as prompt-optimization problems, and by adding a multi-objective variant that also rewards fast SQL execution. Their experiments on the BIRD dev set suggest that an iterative, feedback-driven optimizer (IPO) reaches higher execution accuracy than random selection, tuned random selection, and a non-iterative instruction–exemplar optimizer, while producing a much shorter prompt. They also report that optimizing for latency as a second objective nearly eliminates worst-case slowdowns with only a small accuracy drop.

What carries the argument

The carrying mechanism is an iterative LLM-as-optimizer loop. A Proposer agent maintains the current best prompt — an instruction plus a small exemplar set — and rewrites it using feedback from a SQL Generator agent, which executes candidate SQL on a validation sample and reports accuracy, wrong examples, and correct examples. The same loop can be extended to a multi-objective score that blends execution accuracy with measured SQL latency, and the prompt templates carry a fixed schema-and-evidence format. The short prompt length IPO produces (6,495 tokens versus roughly 23,000–26,000 for the baselines) is evidence that the optimizer is implicitly pruning schema, a by-product rather than a stated design goal.

What would settle it

Re-run the four methods (random exemplars, tuned random exemplars, non-iterative optimizer, and IPO) on the same BIRD dev split with multiple random seeds and matched optimization budgets, and record per-seed accuracy; if the IPO advantage over the next-best baseline falls within the run-to-run spread, the paper's ranking is not established. A second check: hold out a separate query log or database and see whether the static optimized exemplar set still beats retrieval-based selection without re-optimization.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that exemplar selection for NL2SQL need not be retrieval-based: a static set of exemplars produced by an optimization process can match or exceed the accuracy of similarity-based selection. The proposed IPO uses two cooperating agents — a Proposer that writes an instruction and picks diverse exemplars, and a SQL Generator that evaluates the prompt on a sampled validation set and returns correct and wrong cases as feedback. Over five iterations this loop discovers prompts that are not only more accurate but also far more compact, because the optimizer prunes irrelevant schema from exemplars. With accuracy-only optimization, IPO reaches 59.24 percent execution accuracy on BIRD dev versus 57.89 percent for the strongest non-iterative baseline; adding latency to the objective keeps accuracy at 58.98 percent while reducing the maximum query execution time from about 18.5 to 10.2 seconds, close to the ground-truth bound.

Load-bearing premise

The paper's conclusions rest on treating the one-run accuracy gaps it reports — such as 59.24 versus 57.89 percent on BIRD dev — as meaningful differences rather than noise, given no repeated runs, variance, or significance tests.

Editorial extensions

If this is right

  • A production NL2SQL system can serve a fixed prompt per database, eliminating per-query retrieval indices and lookup costs.
  • The optimization objective can fold in execution latency alongside accuracy, yielding near-ground-truth worst-case timings at a small accuracy cost on BIRD dev.
  • The optimizer's automatic schema pruning shows that prompt-size reduction and inference cost savings can emerge without explicit pruning logic.
  • Joint instruction–exemplar optimization outperforms optimizing only the exemplar count, suggesting that feedback on wrong examples is an active ingredient in the improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the static-exemplar result generalizes, NL2SQL deployment becomes a one-time compile step per database rather than per-query retrieval, changing the latency budget of the serving stack.
  • The same Proposer–SQL-generator feedback loop could be applied to other retrieval-heavy in-context learning tasks, such as summarization or tool use, where a fixed optimized prompt may replace dynamic retrieval.
  • Because the baselines were run with different optimization budgets and without repeated seeds, a fair head-to-head with equal trials and variance reporting would be the natural next experiment; the paper itself does not supply that comparison.
  • The BIRD-MULTI benchmark opens a direction beyond the paper's results: optimizing not just what SQL says but how it runs on a specific engine, which is closer to what production systems actually optimize.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper argues that for production NL2SQL systems, a carefully selected static set of exemplars is more important than retrieval-based exemplar selection, and that prompt optimization can find such a set while also enabling joint accuracy and latency optimization. The authors propose Iterative Prompt Optimization (IPO), a two-agent (Proposer and SQL Generator) iterative refinement method, and compare it against Random Exemplar Selection (RES), Optimized Random Exemplar Selection (ORES), and MIPROv2 on the BIRD dev set. They report that IPO achieves the best execution accuracy (59.24%), produces the shortest prompts due to automatic schema pruning, and, when extended to multi-objective optimization on a newly introduced BIRD-MULTI benchmark, can reduce query latency variance while maintaining accuracy.

Significance. If the central claim were fully established, the paper would make a useful practical contribution: eliminating inference-time exemplar retrieval could lower latency and cost in production NL2SQL pipelines, and the joint accuracy-latency optimization addresses a real deployment concern. The introduction of a latency-augmented benchmark is also potentially valuable. The paper is clearly positioned as preliminary, but the empirical evidence currently falls short of supporting the broad comparative claims, because the evaluation lacks a retrieval-based exemplar-selection baseline and reports no variance or significance information.

major comments (4)
  1. [§1, §5.1, Tables 1-3] The central claim that a static optimized exemplar set can replace retrieval-based exemplar selection is untested. The abstract and Section 1 argue that static exemplars play a more crucial role than similarity-based selection, and the conclusion states that prompt optimization is effective for NL2SQL, but every evaluated method (RES, ORES, MIPROv2, IPO) is a static selection strategy. No retrieval-based exemplar baseline (e.g., MageSQL [21] or a simple similarity retriever over the training set) appears in Tables 1-3. The Schema Retrieval module is fixed across all methods and does not address exemplar retrieval. To support the central claim, the authors must add a retrieval-based exemplar-selection baseline and compare against it under matched optimization budgets.
  2. [§5.1, Table 1] The reported accuracy differences are not supported by variance or significance measures. RES reports the best of 10 random samples, while ORES and MIPROv2 use 20 trials and IPO uses 5 iterations, so the methods are evaluated with different optimization budgets. Without multiple seeds, standard deviations, confidence intervals, or significance tests, the observed ranking (IPO 59.24% versus MIPROv2 57.89% on the total) could be within noise. The paper should report means and variances over multiple independent runs, or otherwise justify that the differences are statistically meaningful.
  3. [§4, Benchmark Creation] The BIRD-MULTI benchmark construction is underdescribed and rests on an unverified assumption. The paper states that two SQL variants per query were generated using OpenAI O3, but it does not describe how semantic equivalence to the ground truth was ensured, whether the variants were executed and matched, how many queries are included, what databases are covered, or whether the benchmark and generation scripts will be released. Since the multi-objective experiments in Section 5.2 rely on the correctness of the O3-generated variants and their latency measurements, the authors should provide validation details and make the benchmark artifacts available.
  4. [§5.2, Table 3] The multi-objective results are reported without any measure of variability, and the comparison is not controlled. The table compares accuracy-only IPO with joint accuracy-latency IPO, but no standard deviations across runs, no significance tests, and no separate comparison against a non-optimized baseline are provided. Additionally, the experimental setup for latency measurement (e.g., hardware, database engine, number of executions, caching) is not specified, which makes the latency improvements difficult to interpret or reproduce.
minor comments (5)
  1. [Figure 2] In the Proposer prompt template inside Figure 2, the line 'Correct Examples: {wrong_exemplars}' appears to be a copy-paste error; it should be '{correct_exemplars}'.
  2. [Table 1] The caption 'Ex. Accuracies of Prompt optimization Methods' is unclear; consider spelling out 'Execution Accuracies' and capitalizing 'Optimization' consistently.
  3. [§3.2] The acronym 'LCMs' is introduced as 'Long-context LLMs', but the more standard term 'long-context LLMs' is used elsewhere; consider using one consistent acronym or spelling it out.
  4. [§5.1, Table 2] The table reports ORES with 75 exemplars, but the method description in Section 5 says ORES tunes the hyperparameter k without specifying the upper bound; please clarify how k=75 was obtained and why such a large prompt was selected by the optimizer.
  5. [Appendix A.1] The title 'Random and Optimizes Exemplar Search (RES & ORES)' contains a grammatical error; it should be 'Random and Optimized Exemplar Search'.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; optimization is tuned on a validation split and evaluated on held-out BIRD dev, with only a contextual non-load-bearing self-citation.

full rationale

The paper's derivation chain is empirically grounded rather than circular. Prompt optimization methods (RES, ORES, MIPROv2, IPO) are tuned using a validation set sampled from the training data, as described in Algorithm 1 ('D_train, D_valid <- Split(D)') and Section 3.2 ('evaluates the generated prompt on a validation set (sampled iteratively from the training data)'), and then reported on the BIRD dev set, which is external to the optimization loop. No fitted parameter is renamed as a prediction: the reported accuracies are measured on BIRD dev, not on the validation set used for selection. The only self-citation is reference [21] (MageSQL), which shares two authors with this paper, but it is used only as contextual related work describing retrieval-based exemplar selection; it is not load-bearing for the paper's conclusion that IPO is effective, and no uniqueness theorem or ansatz is imported from it. Concerns that the central claim comparing static optimized exemplars against retrieval-based exemplar selection is untested because no retrieval-based baseline appears in Tables 1-3, and that accuracy differences lack significance tests, are validity concerns rather than circularity: they do not show that any result reduces by construction to its own input. Therefore no circular step meets the evidentiary bar, and the score is 0.

Assumptions & free parameters 5 free parameters · 5 assumptions · 1 invented entities

The paper's central empirical claims rest on several experimental assumptions: BIRD dev as a proxy, SQLite3 latency as a proxy, the validity of O3-generated SQL variants, and the stability of the optimization. The methods also require several hyperparameters (k, trials, iterations) that are set by hand or tuned on validation data. The only new artifact, BIRD-MULTI, has no external validation.

free parameters (5)
  • k (number of exemplars) in ORES = 75
    Selected by Bayesian optimization over 20 trials on a validation split; yields 84,519-token prompts (Table 2).
  • k (number of exemplars) in RES = 10
    Fixed baseline hyperparameter (Section 5).
  • Number of optimization trials (ORES, MIPROv2) = 20
    Chosen by the authors; no sensitivity analysis is reported.
  • IPO iterations = 5
    Chosen by the authors; no sensitivity analysis is reported.
  • Minimum exemplars per IPO iteration = 5
    Instructed in the Proposer prompt (Section 5).
assumptions (5)
  • domain assumption BIRD dev is a representative benchmark for NL2SQL evaluation.
    All accuracy and latency numbers are reported on BIRD dev, and the paper generalizes to production scenarios from these results (Section 5).
  • domain assumption Wall-clock execution time on SQLite3 is a valid proxy for query latency.
    The multi-objective benchmark measures execution times in SQLite3 without discussing hardware or concurrency effects (Section 4).
  • ad hoc to paper OpenAI O3 generated SQL variants for BIRD-MULTI are correct and semantically equivalent to ground truth.
    The variants were generated by a reasoning model and not independently validated; the benchmark is not released (Section 4).
  • domain assumption The train/validation split used for optimization prevents overfitting to the reported BIRD dev results.
    The paper does not describe the split size or confirm that dev was not used for prompt selection (Section 5).
  • domain assumption GPT-4o is a suitable representative LLM for the findings.
    All experiments use GPT-4o; no other models are tested (Section 5).
invented entities (1)
  • BIRD-MULTI benchmark
    purpose: Augment BIRD with latency-annotated SQL variants for multi-objective prompt optimization.
    Introduced in Section 4 but not released, and the correctness of O3-generated variants is unverified.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Effectiveness of Prompt Optimization in NL2SQL Systems." pith.science (2026). https://pith.science/paper/OFY23FVD

@misc{pith2026250520591,
  author       = {Pith},
  title        = {Pith review of: Effectiveness of Prompt Optimization in NL2SQL Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/OFY23FVD}},
  note         = {Machine review of arXiv:2505.20591}
}
read the original abstract

NL2SQL approaches have greatly benefited from the impressive capabilities of large language models (LLMs). In particular, bootstrapping an NL2SQL system for a specific domain can be as simple as instructing an LLM with sufficient contextual information, such as schema details and translation demonstrations. However, building an accurate system still requires the rigorous task of selecting the right context for each query-including identifying relevant schema elements, cell values, and suitable exemplars that help the LLM understand domain-specific nuances. Retrieval-based methods have become the go-to approach for identifying such context. While effective, these methods introduce additional inference-time costs due to the retrieval process. In this paper, we argue that production scenarios demand high-precision, high-performance NL2SQL systems, rather than simply high-quality SQL generation, which is the focus of most current NL2SQL approaches. In such scenarios, the careful selection of a static set of exemplars-capturing the intricacies of the query log, target database, SQL constructs, and execution latencies-plays a more crucial role than exemplar selection based solely on similarity. The key challenge, however, lies in identifying a representative set of exemplars for a given production setting. To this end, we propose a prompt optimization framework that not only addresses the high-precision requirement but also optimizes the performance of the generated SQL through multi-objective optimization. Preliminary empirical analysis demonstrates the effectiveness of the proposed framework.

Figures

Figures reproduced from arXiv: 2505.20591 by the authors.

Figure 1
Figure 1. NL2SQL Pipeline Algorithm 1: Optimization of Random Exemplar Selection Input: D, 𝑛𝑢𝑚𝑇 𝑟𝑖𝑎𝑙𝑠, P𝑏𝑎𝑠𝑒 , LLM Output: P𝑜𝑝𝑡 1 D𝑡𝑟𝑎𝑖𝑛, D𝑣𝑎𝑙𝑖𝑑 ← Split(D) 2 𝑏𝑒𝑠𝑡_𝑠𝑐𝑜𝑟𝑒 ← 0 3 𝑃𝑜𝑝𝑡 ← P𝑏𝑎𝑠𝑒 4 while 𝑖 ≤ 𝑛𝑢𝑚𝑇 𝑟𝑖𝑎𝑙𝑠 do 5 𝑘 ← 𝑡𝑟𝑖𝑎𝑙.suggest_int( ′k ′ , 0, K) /* choose 𝑘 */ 6 𝐸𝑖 ← Random(D𝑡𝑟𝑎𝑖𝑛, 𝑘) /* 𝑘 exemplars */ 7 P𝑖 ← P𝑏𝑎𝑠𝑒 + 𝐸𝑥𝑖 8 𝑠𝑐𝑜𝑟𝑒 ← Evaluate(LLM, P𝑖 , D𝑣𝑎𝑙𝑖𝑑 ) 9 if 𝑠𝑐𝑜𝑟𝑒 > 𝑏𝑒𝑠𝑡_𝑠𝑐𝑜𝑟𝑒 then 10 𝑃𝑜𝑝𝑡 ← P𝑖 11 𝑏𝑒𝑠𝑡_𝑠𝑐𝑜𝑟𝑒 ← 𝑠𝑐𝑜𝑟… view at source ↗
Figure 3
Figure 3. IPO generated exemplar with automatic schema [PITH_FULL_IMAGE:figures/full_fig_p003_3.png] view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 15 canonical work pages

  1. [21]

    Krista Opsahl-Ong, Michael J Ryan, Josh Purtell, David Broman, Christopher Potts, Matei Zaharia, and Omar Khattab. 2024. Optimizing Instructions and Demonstrations for Multi-Stage Language Model Programs. InProceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, Yaser Al-Onaizan, Mohit Bansal, and Yun-Nung Chen (Eds.). Asso...

    doi:10.18653/v1/2024.emnlp- 2024

  2. [1]

    James Bergstra, Rémi Bardenet, Yoshua Bengio, and Balázs Kégl. 2011. Algorithms for Hyper-Parameter Optimization. InAdvances in Neural Information Processing Systems, J. Shawe-Taylor, R. Zemel, P. Bartlett, F. Pereira, and K.Q. Weinberger (Eds.), Vol. 24. Curran Associates, Inc. https://proceedings.neurips.cc/paper_ files/paper/2011/file/86e8f7ab32cfd1257...

    work page 2011

  3. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, Sandhini Agarwal, Ariel Herbert-Voss, Gretchen Krueger, Tom Henighan, Rewon Child, Aditya Ramesh, Daniel Ziegler, Jeffrey Wu, Clemens Winter, Chris Hesse, Mark Chen, Eric Sigler, Mateusz Litwin, Scott Gr...

    2020

  4. [3]

    Hasan Alp Caferoğlu and Özgür Ulusoy. 2025. E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL. arXiv:2409.16751 [cs.CL] https://arxiv. org/abs/2409.16751

    arXiv 2025

  5. [4]

    Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan

    Yeounoh Chung, Gaurav T. Kakkar, Yu Gan, Brenton Milne, and Fatma Ozcan

    work page

  6. [5]

    Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, lu Chen, Jin- shu Lin, and Dongfang Lou. 2023. C3: Zero-shot Text-to-SQL with ChatGPT. arXiv:2307.07306 [cs.CL] https://arxiv.org/abs/2307.07306

    arXiv 2023

  7. [6]

    Venkatesh Emani, Vivek Pandit, Victor Shnayder, Wenjing Wang, and Carlo Curino

    Avrilia Floratou, Fotis Psallidas, Fuheng Zhao, Shaleen Deep, Gunther Hagleither, Wangda Tan, Joyce Cahoon, Rana Alotaibi, Jordan Henkel, Abhik Singla, Alex Van Grootel, Brandon Chow, Kai Deng, Katherine Lin, Marcos Campos, K. Venkatesh Emani, Vivek Pandit, Victor Shnayder, Wenjing Wang, and Carlo Curino. 2024. NL2SQL is a solved problem... Not!. InCIDR. ...

    work page 2024

  8. [7]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2023. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. arXiv:2308.15363 [cs.DB] https://arxiv.org/abs/2308. 15363

    arXiv 2023

Show all 38 references
  1. [8]

    Yingqi Gao, Yifu Liu, Xiaoxia Li, Xiaorong Shi, Yin Zhu, Yiming Wang, Shiqi Li, Wei Li, Yuntao Hong, Zhiling Luo, Jinyang Gao, Liyu Mou, and Yu Li. 2025. A Preview of XiYan-SQL: A Multi-Generator Ensemble Framework for Text-to-SQL. arXiv:2411.08599 [cs.AI] https://arxiv.org/ab...

    2025 arXiv

  2. [9]

    Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

    Omar Khattab, Arnav Singhvi, Paridhi Maheshwari, Zhiyuan Zhang, Keshav Santhanam, Sri Vardhamanan A, Saiful Haq, Ashutosh Sharma, Thomas T. Joshi, Hanna Moazam, Heather Miller, Matei Zaharia, and Christopher Potts

  3. [10]

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, et al. 2024. Spider 2.0: Evaluating language models on real-world enterprise text-to-sql workflows. arXiv preprint arXiv:2411.07763(2024)

    2024 arXiv

  4. [11]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al . 2024. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Sy...

    2024

  5. [12]

    Tianle Li, Ge Zhang, Quy Duc Do, Xiang Yue, and Wenhu Chen. 2025. Long- context LLMs Struggle with Long In-context Learning.Transactions on Machine Learning Research(2025). https://openreview.net/forum?id=Cw2xlg0e46

    2025

  6. [13]

    Chunwei Liu, Matthew Russo, Michael Cafarella, Lei Cao, Peter Baille Chen, Zui Chen, Michael Franklin, Tim Kraska, Samuel Madden, and Gerardo Vitagliano. 2024. A Declarative System for Optimizing AI Workloads. arXiv:2405.14696 [cs.CL] https://arxiv.org/abs/2405.14696

    2024 arXiv

  7. [14]

    Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang

    Nelson F. Liu, Kevin Lin, John Hewitt, Ashwin Paranjape, Michele Bevilacqua, Fabio Petroni, and Percy Liang. 2024. Lost in the Middle: How Language Models Use Long Contexts.Transactions of the Association for Computational Linguistics 12 (2024), 157–173. doi:10.1162/tacl_a_00638

    2024 doi

  8. [15]

    Karime Maamari, Fadhil Abubaker, Daniel Jaroslawicz, and Amine Mhedhbi

  9. [16]

    Linyong Nan, Yilun Zhao, Weijin Zou, Narutatsu Ri, Jaesung Tae, Ellen Zhang, Arman Cohan, and Dragomir Radev. 2023. Enhancing Text-to-SQL Capabilities of Large Language Models: A Study on Prompt Design Strategies. InFindings of the Association for Computational Linguistics: EM...

    2023 doi

  10. [17]

    OpenAI. 2025. Introducing o3 and o4-mini. https://openai.com/index/ introducing-o3-and-o4-mini/ Accessed: 2025-05-07

    2025

  11. [18]

    arXiv:2408.07702 [cs.CL] https://arxiv.org/abs/2408.07702

    The Death of Schema Linking? Text-to-SQL in the Age of Well-Reasoned Language Models. arXiv:2408.07702 [cs.CL] https://arxiv.org/abs/2408.07702

    arXiv

  12. [19]

    Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O Arik

  13. [20]

    Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction. InThirty-seventh Con- ference on Neural Information Processing Systems. https://openreview.net/forum? id=p53QDxSIc5

    2023

  14. [22]

    Logan IV, Eric Wallace, and Sameer Singh

    Taylor Shin, Yasaman Razeghi, Robert L. Logan IV, Eric Wallace, and Sameer Singh. 2020. AutoPrompt: Eliciting Knowledge from Language Models with Auto- matically Generated Prompts. InProceedings of the 2020 Conference on Empirical Methods in Natural Language Processing (EMNLP)...

    2020 doi

  15. [23]

    InThe Thirteenth International Conference on Learning Representations

    CHASE-SQL: Multi-Path Reasoning and Preference Optimized Candidate Selection in Text-to-SQL. InThe Thirteenth International Conference on Learning Representations. https://openreview.net/forum?id=CvGqMD5OtX

  16. [24]

    Gemini Team. 2024. Gemini: A Family of Highly Capable Multimodal Models. arXiv:2312.11805 [cs.CL] https://arxiv.org/abs/2312.11805

    2024 arXiv

  17. [25]

    Chen Shen, Jin Wang, Sajjadur Rahman, and Eser Kandogan. 2025. MageSQL: En- hancing In-context Learning for Text-to-SQL Applications with Large Language Models. arXiv:2504.02055 [cs.DB] https://arxiv.org/abs/2504.02055

    2025 arXiv

  18. [26]

    Yuxin Wen, Neel Jain, John Kirchenbauer, Micah Goldblum, Jonas Geiping, and Tom Goldstein. 2023. Hard Prompts Made Easy: Gradient-Based Discrete Opti- mization for Prompt Tuning and Discovery. InThirty-seventh Conference on Neural Information Processing Systems. https://openre...

    2023

  19. [27]

    Nan Tang, Chenyu Yang, Ju Fan, Lei Cao, Yuyu Luo, and Alon Y. Halevy. 2024. VerifAI: Verified Generative AI. InCIDR. https://www.cidrdb.org/cidr2024/ papers/p5-tang.pdf

    2024

  20. [29]

    OpenAI Team. 2024. GPT-4o System Card. arXiv:2410.21276 [cs.CL] https: //arxiv.org/abs/2410.21276

    2024 arXiv

  21. [31]

    Chengrun Yang, Xuezhi Wang, Yifeng Lu, Hanxiao Liu, Quoc V Le, Denny Zhou, and Xinyun Chen. 2024. Large Language Models as Optimizers. InThe Twelfth International Conference on Learning Representations. https://openreview.net/ forum?id=Bb4VGOWELI A Appendix A.1 Prompt Template...

    2024

  22. [51]

    The best prompt generated so far

  23. [62]

    The accuracy of the best prompt

  24. [84]

    The accuracy of the current prompt

  25. [95]

    A set of exemplars where the current prompt 10incorrectly generated the SQL query

  26. [116]

    A set of exemplars where the current prompt 12correctly generated the SQL query 13 14#Goal: 15Think step by step to generate a prompt comprising 16of two parts in JSON format:

  27. [171]

    Instruction for the LLM to generate SQL query 18for sqlite3 database

  28. [192]

    A set of diverse exemplars to assist the LLM in 20generating the SQL query. 21 22#Best Prompt: 23{best_prompt} 24 25#Best Accuracy: 26{best_accuracy} 27 28#Current Prompt: 29{current_prompt} 30 31#Current Prompt Accuracy: 32{current_accuracy} 33 34#Wrong Exemplars: 35{wrong_ex...

  29. [2024]

    InThe Twelfth International Conference on Learning Representations

    DSPy: Compiling Declarative Language Model Calls into State-of-the-Art Pipelines. InThe Twelfth International Conference on Learning Representations. https://openreview.net/forum?id=sY5N0zY5Od

  30. [2025]

    arXiv:2501.12372 [cs.DB] https://arxiv.org/abs/2501.12372

    Is Long Context All You Need? Leveraging LLM’s Extended Context for NL2SQL. arXiv:2501.12372 [cs.DB] https://arxiv.org/abs/2501.12372

    arXiv

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.