Pith. sign in

REVIEW 2 major objections 4 minor 53 references

Lightweight models that watch the running consistency of repeated Text-to-SQL runs can predict when that consistency has settled and stop sampling early.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 22:27 UTC pith:NOOB2WFC

load-bearing objection Solid applied systems paper: learned 1-D detectors beat a tuned Beta-Bernoulli ASC baseline on a hand-chosen convergence target for Text-to-SQL, with real call savings and noise robustness, but small N and no (W,K) ablation keep the edge provisional. the 2 major comments →

arxiv 2607.03991 v1 pith:NOOB2WFC submitted 2026-07-04 cs.LG cs.CL

Knowing When to Stop: Predicting Execution-Consistency Convergence in Text-to-SQL

Yaron Anavi , Mor Aisenberg , Nadav Nesher , Elena Khabibullina , Isabella Cattinelli This is my paper
classification cs.LG cs.CL
keywords Text-to-SQLexecution consistencyadaptive stoppingLLM inference costconvergence predictionself-consistencyBeta-Bernoulli stopping
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When a Text-to-SQL pipeline runs the same question many times, the fraction of runs that agree becomes a trust signal, but each extra run costs latency and money. This paper asks when that consistency has already converged so further calls are wasted. It treats the running consistency sequence as a one-dimensional signal and trains simple classifiers and detectors that decide, after every run, whether the next thirty runs will stay within five percentage points of the current value. On a public finance benchmark and two production customer sets, the learned stoppers adapt to each question, halt sooner when agreement arrives early, continue longer when it arrives late, beat a tuned Beta-Bernoulli statistical rule and a fixed-run baseline, and still work when the correctness labels are deliberately flipped to mimic a noisy production judge. The practical payoff is fewer LLM calls without giving up a reliable consistency estimate.

Core claim

The authors show that a family of lightweight models (logistic regression, XGBoost, and a small 1-D temporal convolutional network) that read only the running consistency trajectory can predict both whether consistency has converged and the first run at which it does so, outperforming a principled Beta-Bernoulli Adaptive-Consistency stopping rule and a fixed-budget baseline on three datasets while using roughly 35 percent fewer total LLM calls and remaining accurate under 5–15 percent label noise.

What carries the argument

The operational convergence label: consistency after run n is declared converged if it stays within K = 0.05 of its current value over the next W = 30 runs. Models consume either hand-crafted features of that trajectory (run index, current consistency, local variances) or the raw sequence itself and are trained to classify the label and to fire as close as possible to the first true converged run.

Load-bearing premise

The paper treats “stays within five percentage points for the next thirty runs” as the ground-truth definition of convergence that every production system should optimize; if a different stability window is what operators actually need, the detectors are solving the wrong problem.

What would settle it

On a fresh set of questions each given at least 100 independent runs, measure the first run each detector fires; if that run’s consistency later drifts more than five points inside a thirty-run window more often than the Beta-Bernoulli baseline, or if the root-mean-square error to the true first-converged run exceeds the baseline’s, the central claim is false.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Production Text-to-SQL systems can replace a fixed run budget with an adaptive stopper that spends fewer calls on easy questions and more on hard ones.
  • The same stopper can sit above any binary execution judge, including imperfect learned judges, because performance degrades only gradually under random label flips.
  • Weak serial correlation between successive runs justifies random order permutation as a cheap training-data augmentation controlled by a single shuffle weight.
  • The method returns both a stop decision and the consistency value at that stop, giving an immediate trust score without a second estimation step.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same trajectory-based detectors could be tried on majority-vote self-consistency for free-form reasoning tasks, not only Text-to-SQL execution matches.
  • If real judge errors are structured (for example, systematically missing certain join failures) rather than random flips, the noise-injection results may overstate robustness; testing actual judge error profiles is the natural next measurement.
  • Datasets whose questions show wider spread in convergence times should yield larger average call savings than the relatively high-consistency collections studied here.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 4 minor

Summary. The paper formulates adaptive stopping for Text-to-SQL reliability estimation as a convergence-prediction problem. From repeated LLM runs of a question, each producing a binary True/False execution-correctness label (via ground-truth table comparison), it builds the running majority consistency C(n) (Eq. 1) and defines convergence at run n by a fixed window criterion: |C(t)−C(n)| ≤ K over the next W runs, with W=30 and K=0.05 (Eq. 2). Lightweight models (logistic regression and XGBoost on hand-crafted features of run index, C(n), and local variances; a small 1-D TCN on the raw trajectory) are trained under question-level 5-fold CV to classify whether a prefix has converged and to detect the first such run. On the BIRD finance subset (83 questions) and two small production sets (25 and 19 questions), each with 100 runs, the best detectors achieve lower RMSE to the first converged run than a tuned Beta-Bernoulli Adaptive-Consistency baseline or a runs-only baseline (Table 4: logistic 8.18 vs ASC 12.82), use ~35% fewer total LLM calls, and degrade gracefully under 5–15% random label flips that simulate an imperfect judge (Tables 3–4). Weak lag-k autocorrelations (Table 1) justify order-permutation augmentation controlled by β_shuffle.

Significance. If the result holds under the chosen operational definition of convergence, the work supplies a practical, drop-in stopping layer that can sit above any binary execution judge and that adapts the sample budget per question rather than using a fixed run count. The comparison against a principled Beta-Bernoulli ASC baseline, the noise-injection stress test, the explicit autocorrelation justification for shuffling, and the consistent behavior across a public benchmark and two customer domains are concrete strengths. The contribution is engineering-oriented rather than foundational, but it addresses a real production cost–latency trade-off in Text-to-SQL pipelines and is immediately usable once a judge is available.

major comments (2)
  1. The detection claim (Table 4: RMSE 8.18 vs ASC 12.82, ~35% call savings) is defined entirely with respect to the fixed target Converged(n, W=30, K=0.05) of Eq. 2. Section 3 justifies this pair only by the post-hoc observation that once the window holds, consistency stays inside K for the remainder of the 100-run trajectory with empirical probability 0.995. No sensitivity analysis over alternative (W, K) pairs is reported, nor is ASC re-tuned against alternative ground-truth labels. Because labels can be formed only for n ≤ 70, the supervised objective is locked to this horizon. If operators care about a shorter window or tighter tolerance, both absolute RMSE and the ranking versus ASC could change. A modest ablation over a small grid of (W, K) would make the headline numbers transferable.
  2. Customer datasets A and B contain only 25 and 19 questions (Table 2). With question-level 5-fold CV this yields test folds of roughly 4–5 questions. Tables 3–4 report only point averages of AUC/RMSE with no standard errors, confidence intervals, or per-fold ranges. Given that the central comparison is a few RMSE points against ASC, the absence of uncertainty quantification makes it hard to judge whether the reported edge is stable. At minimum, fold-level standard deviations or bootstrap intervals on the detection RMSE should be added.
minor comments (4)
  1. All generations use a single model (Azure OpenAI GPT-4.1). The Limitations section notes this, but a short remark on expected transfer to other generators or temperatures would help readers gauge scope.
  2. The random-flip noise model (5%/15%) is a convenient proxy for an imperfect judge; a sentence acknowledging that real judge errors may be structured (e.g., systematic on certain SQL patterns) would set expectations more accurately.
  3. Figure 3 and the labeling pipeline (Figure 2) are clear; adding a short caption note that consistency can be 100% even when every run is False (as stated in §3) would prevent misreading of the y-axis.
  4. Hyperparameter grids and the exact TCN architecture are relegated to Appendix A; a one-sentence pointer in §5.2 would improve reproducibility for readers who stop at the main text.

Circularity Check

0 steps flagged

No circularity: supervised predictors of an externally defined future-window label, compared to independent baselines.

full rationale

The paper defines Consistency(n) from binary True/False execution outcomes (Eq. 1) and labels a prefix as Converged(n, W=30, K=0.05) only when the next W held-out runs stay inside the K band (Eq. 2). Models receive only the prefix trajectory (or hand-crafted features of it) and are trained to predict that future-window label under question-level CV; the label is therefore an external target, not a quantity recovered by construction from the inputs. The Beta-Bernoulli ASC baseline and the runs-only baseline are independent comparators whose parameters are tuned on validation folds, not derived from the learned models. Weak lag-k autocorrelations (Table 1) justify permutation augmentation but do not force any reported metric. No self-citation supplies a uniqueness theorem or ansatz that the central claim rests on; W and K are free operational choices justified by a post-hoc empirical probability (0.995), not a circular derivation. The detection RMSE and call-saving claims are therefore ordinary supervised-learning results against an independently defined ground truth, not tautologies.

Axiom & Free-Parameter Ledger

6 free parameters · 5 axioms · 0 invented entities

The central claim rests on a hand-chosen operational definition of convergence, the approximation that runs are nearly i.i.d., standard supervised-learning machinery, and small per-dataset models trained on limited question counts. No new physical or mathematical entities are postulated; free parameters are the usual ML and stopping hyperparameters plus the (W,K) convergence pair.

free parameters (6)
  • convergence window W = 30
    Fixed at 30 runs; chosen so that once the K-band holds over W, it holds for the remainder with empirical probability 0.995 (Section 3).
  • convergence tolerance K = 0.05
    Fixed at 0.05 (5 percentage points); co-chosen with W to define the binary Converged label (Eq. 2).
  • β_shuffle = tuned per dataset (grid)
    Weights original run order versus permuted copies during training; tuned per dataset by validation grid search (Section 4.2).
  • detection decision threshold = validation-tuned
    Tuned on validation away from 0.5 to bias against early stopping for the RMSE detection objective (Section 5.4).
  • model hyperparameters (C, l1_ratio, XGB depth/lr/n_estimators, TCN dropout/lr/batch) = grid-searched per fold
    Selected by per-fold grid search on validation AUC or RMSE (Appendix A).
  • ASC credible-interval level and width threshold = validation-tuned
    Tuned on validation for the Beta-Bernoulli baseline, same as learned models (Section 5.2).
axioms (5)
  • domain assumption Repeated Text-to-SQL runs of the same prompt are approximately i.i.d. Bernoulli, justified by lag-1..3 autocorrelations near zero (Table 1).
    Used to motivate both the variance feature and permutation augmentation; authors note serving nondeterminism can break strict i.i.d. (Section 3).
  • ad hoc to paper Consistency is the majority fraction max(N_True, N_False)/n (Eq. 1), including the case of consistently wrong SQL.
    Defines the 1-D signal the models observe; measures behavioral stability, not correctness.
  • domain assumption A run is True iff the generated result table matches the gold table under the paper’s subset-column equality rule (Section 5.1).
    Ground-truth labeling for all training and evaluation; production would replace this with a learned judge.
  • ad hoc to paper Random independent flips of True/False labels at rate 5% or 15% adequately simulate an imperfect production judge.
    Noise-injection experiment (Section 4 intro, Tables 3–4); authors note real judge errors need not be random.
  • standard math Standard supervised classification/detection with ROC AUC and RMSE is the right evaluation of stopping quality.
    Used throughout Section 5; no conformal or decision-theoretic loss is optimized.

pith-pipeline@v1.1.0-grok45 · 16969 in / 3369 out tokens · 29705 ms · 2026-07-11T22:27:25.631622+00:00 · methodology

0 comments
read the original abstract

Repeated LLM calls are the standard way to estimate how trustworthy a Text-to-SQL result is: run the pipeline multiple times, judge each SQL execution, and use the consistency of the verdicts as a confidence signal. The open question is when to stop, when the consistency has converged. We formulate this as a convergence-prediction problem and train a family of lightweight 1-D models that observe the running consistency trajectory and decide, at each step, whether further runs are unlikely to shift it materially, and we benchmark them against a principled Beta-Bernoulli stopping rule and a learned run-count baseline. On the BIRD benchmark and two production customer datasets, our method adapts its stopping point to each user question, halting sooner when consistency converges early and continuing longer when it converges late. We further show that the weak serial correlation between runs lets us permute their order as a training augmentation, controlled by a tunable shuffling weight. Performance stays consistent across the three datasets, and to mimic an imperfect production judge we inject noise into the correct/incorrect verdicts obtained by comparing the generated and ground-truth SQL results, showing that the method still predicts convergence reliably.

Figures

Figures reproduced from arXiv: 2607.03991 by Elena Khabibullina, Isabella Cattinelli, Mor Aisenberg, Nadav Nesher, Yaron Anavi.

Figure 1
Figure 1. Figure 1: Stopping-criterion pipeline. A user question is [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Ground-truth labeling pipeline. For each run, the table executed from the generated SQL is compared [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: Consistency as a function of the number of [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

53 extracted references · 10 linked inside Pith

  1. [1]

    Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with

    Aggarwal, Pranjal and Madaan, Aman and Yang, Yiming and Mausam , booktitle =. Let's Sample Step by Step: Adaptive-Consistency for Efficient Reasoning and Coding with

  2. [2]

    Askari, Arian and Poelitz, Christian and Tang, Xinye , journal =

  3. [5]

    Chen, Lingjiao and Zaharia, Matei and Zou, James , journal =

  4. [7]

    Chen, Jikai and Gan, Leilei and Zhao, Ziyu and Wang, Zechuan and Wang, Dong and Zhuang, Chenyi , journal =

  5. [8]

    Gao, Dawei and Wang, Haibin and Li, Yaliang and Sun, Xiuyu and Qian, Yichen and Ding, Bolin and Zhou, Jingren , journal =

  6. [9]

    Gou, Zhibin and Shao, Zhihong and Gong, Yeyun and Shen, Yelong and Yang, Yujiu and Duan, Nan and Chen, Weizhu , booktitle =

  7. [10]

    Defeating Nondeterminism in

    He, Horace and. Defeating Nondeterminism in. 2025 , howpublished =

    2025

  8. [11]

    Kim, Heegyu and Taeyang, Jeon and Choi, SeungHwan and Choi, Seungtaek and Cho, Hyunsouk , booktitle =

  9. [12]

    ICLR , year =

    Semantic Uncertainty: Linguistic Invariances for Uncertainty Estimation in Natural Language Generation , author =. ICLR , year =

  10. [13]

    Spider 2.0: Evaluating Language Models on Real-World Enterprise

    Lei, Fangyu and Chen, Jixuan and Ye, Yuxiao and Cao, Ruisheng and Shin, Dongchan and Su, Hongjin and Suo, Zhaoqing and Gao, Hongcheng and Hu, Wenjing and Yin, Pengcheng and Zhong, Victor and Xiong, Caiming and Sun, Ruoxi and Liu, Qian and Wang, Sida and Yu, Tao , booktitle =. Spider 2.0: Evaluating Language Models on Real-World Enterprise. 2025 , note =

    2025

  11. [14]

    and Huang, Fei and Cheng, Reynold and Li, Yongbin , booktitle =

    Li, Jinyang and Hui, Binyuan and Qu, Ge and Yang, Jiaxi and Li, Binhua and Li, Bowen and Wang, Bailin and Qin, Bowen and Geng, Ruiying and Huo, Nan and Zhou, Xuanhe and Ma, Chenhao and Li, Guoliang and Chang, Kevin C.C. and Huang, Fei and Cheng, Reynold and Li, Yongbin , booktitle =. Can

  12. [15]

    ICLR , year =

    Escape Sky-high Cost: Early-stopping Self-Consistency for Multi-step Reasoning , author =. ICLR , year =

  13. [16]

    Liu, Yang and Iter, Dan and Xu, Yichong and Wang, Shuohang and Xu, Ruochen and Zhu, Chenguang , booktitle =

  14. [17]

    Advances in Neural Information Processing Systems (NeurIPS) , year =

    A Unified Approach to Interpreting Model Predictions , author =. Advances in Neural Information Processing Systems (NeurIPS) , year =

  15. [18]

    ICLR , year =

    A Time Series is Worth 64 Words: Long-term Forecasting with Transformers , author =. ICLR , year =

  16. [19]

    2025 , howpublished =

    2025

  17. [20]

    Pourreza, Mohammadreza and Rafiei, Davood , booktitle =

  18. [21]

    , booktitle =

    Pourreza, Mohammadreza and Li, Hailong and Sun, Ruoxi and Chung, Yeounoh and Talaei, Shayan and Kakkar, Gaurav Tarlok and Gan, Yu and Saberi, Amin and Ozcan, Fatma and Arik, Sercan O. , booktitle =

  19. [22]

    Efficient

    Qu, Huaizhi and Choi, Inyoung and Tan, Zhen and Wang, Song and Yun, Sukwon and Long, Qi and Siddiqui, Faizan and Lee, Kwonjoon and Chen, Tianlong , journal =. Efficient

  20. [23]

    ICLR , year =

    Self-Consistency Improves Chain of Thought Reasoning in Language Models , author =. ICLR , year =

  21. [24]

    Xiong, Miao and Hu, Zhiyuan and Lu, Xinyang and Li, Yifei and Fu, Jie and He, Junxian and Hooi, Bryan , booktitle =. Can

  22. [25]

    Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and

    Yu, Tao and Zhang, Rui and Yang, Kai and Yasunaga, Michihiro and Wang, Dongxu and Li, Zifan and Ma, James and Li, Irene and Yao, Qingning and Roman, Shanelle and Zhang, Zilin and Radev, Dragomir , booktitle =. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and

  23. [26]

    and Zhang, Hao and Gonzalez, Joseph E

    Zheng, Lianmin and Chiang, Wei-Lin and Sheng, Ying and Zhuang, Siyuan and Wu, Zhanghao and Zhuang, Yonghao and Lin, Zi and Li, Zhuohan and Li, Dacheng and Xing, Eric P. and Zhang, Hao and Gonzalez, Joseph E. and Stoica, Ion , booktitle =. Judging

  24. [27]

    An Actor-Critic Approach to Boosting

    Zheng, Ziyang and Jing, Haipeng and Rui, Canyu and Hamdulla, Askar and Wang, Dong , journal =. An Actor-Critic Approach to Boosting

  25. [28]

    Zhong, Victor and Xiong, Caiming and Socher, Richard , journal =

  26. [29]

    Pranjal Aggarwal, Aman Madaan, Yiming Yang, and Mausam. 2023. Let's sample step by step: Adaptive-consistency for efficient reasoning and coding with LLMs . In Proceedings of EMNLP

    2023

  27. [30]

    Arian Askari, Christian Poelitz, and Xinye Tang. 2024. MAGIC : Generating self-correction guideline for in-context Text-to-SQL . arXiv preprint arXiv:2406.12692

    Pith/arXiv arXiv 2024

  28. [31]

    Zico Kolter, and Vladlen Koltun

    Shaojie Bai, J. Zico Kolter, and Vladlen Koltun. 2018. An empirical evaluation of generic convolutional and recurrent networks for sequence modeling. arXiv preprint arXiv:1803.01271

    Pith/arXiv arXiv 2018

  29. [32]

    Le, Christopher R \'e , and Azalia Mirhoseini

    Bradley Brown, Jordan Juravsky, Ryan Ehrlich, Ronald Clark, Quoc V. Le, Christopher R \'e , and Azalia Mirhoseini. 2024. Large language monkeys: Scaling inference compute with repeated sampling. arXiv preprint arXiv:2407.21787

    Pith/arXiv arXiv 2024

  30. [33]

    Jikai Chen, Leilei Gan, Ziyu Zhao, Zechuan Wang, Dong Wang, and Chenyi Zhuang. 2025. SQLCritic : Correcting Text-to-SQL generation via clause-wise critic. arXiv preprint arXiv:2503.07996

    Pith/arXiv arXiv 2025

  31. [34]

    Lingjiao Chen, Matei Zaharia, and James Zou. 2023 a . FrugalGPT : How to use large language models while reducing cost and improving performance. arXiv preprint arXiv:2305.05176

    Pith/arXiv arXiv 2023

  32. [35]

    Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023 b . Universal self-consistency for large language model generation. arXiv preprint arXiv:2311.17311

    Pith/arXiv arXiv 2023

  33. [36]

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. 2024. Text-to-SQL empowered by large language models: A benchmark evaluation. Proceedings of the VLDB Endowment

    2024

  34. [37]

    Zhibin Gou, Zhihong Shao, Yeyun Gong, Yelong Shen, Yujiu Yang, Nan Duan, and Weizhu Chen. 2024. CRITIC : Large language models can self-correct with tool-interactive critiquing. In ICLR

    2024

  35. [38]

    Horace He and Thinking Machines Lab . 2025. Defeating nondeterminism in LLM inference. https://thinkingmachines.ai/blog/defeating-nondeterminism-in-llm-inference/. Thinking Machines Lab: Connectionism

    2025

  36. [39]

    Heegyu Kim, Jeon Taeyang, SeungHwan Choi, Seungtaek Choi, and Hyunsouk Cho. 2025. FLEX : Expert-level false-less EX ecution metric for Text-to-SQL benchmark. In Proceedings of the 2025 Conference of the Nations of the Americas Chapter of the Association for Computational Linguistics: Human Language Technologies (Volume 1: Long Papers), pages 4448--4475

    2025

  37. [40]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2023. Semantic uncertainty: Linguistic invariances for uncertainty estimation in natural language generation. In ICLR

    2023

  38. [41]

    Fangyu Lei, Jixuan Chen, Yuxiao Ye, Ruisheng Cao, Dongchan Shin, Hongjin Su, Zhaoqing Suo, Hongcheng Gao, Wenjing Hu, Pengcheng Yin, Victor Zhong, Caiming Xiong, Ruoxi Sun, Qian Liu, Sida Wang, and Tao Yu. 2025. Spider 2.0: Evaluating language models on real-world enterprise Text-to-SQL workflows. In International Conference on Learning Representations (I...

    Pith/arXiv arXiv 2025

  39. [42]

    Chang, Fei Huang, Reynold Cheng, and Yongbin Li

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C.C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM already serve as a database interface? a big bench for large-scale database grounded Text-to-SQLs ( BIRD ). In Advances in Neural Inf...

    2023

  40. [43]

    Yiwei Li, Peiwen Yuan, Shaoxiong Feng, Boyuan Pan, Xinglin Wang, Bin Sun, Heda Wang, and Kan Li. 2024. Escape sky-high cost: Early-stopping self-consistency for multi-step reasoning. In ICLR

    2024

  41. [44]

    Yang Liu, Dan Iter, Yichong Xu, Shuohang Wang, Ruochen Xu, and Chenguang Zhu. 2023. G-Eval : NLG evaluation using GPT-4 with better human alignment. In Proceedings of EMNLP

    2023

  42. [45]

    Lundberg and Su-In Lee

    Scott M. Lundberg and Su-In Lee. 2017. A unified approach to interpreting model predictions. In Advances in Neural Information Processing Systems (NeurIPS)

    2017

  43. [46]

    Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam

    Yuqi Nie, Nam H. Nguyen, Phanwadee Sinthong, and Jayant Kalagnanam. 2023. A time series is worth 64 words: Long-term forecasting with transformers. In ICLR

    2023

  44. [47]

    OpenAI . 2025. GPT-4.1 . https://openai.com/index/gpt-4-1/

    2025

  45. [48]

    Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O. Arik. 2025. CHASE-SQL : Multi-path reasoning and preference-optimized candidate selection in Text-to-SQL . In ICLR

    2025

  46. [49]

    Mohammadreza Pourreza and Davood Rafiei. 2023. DIN-SQL : Decomposed in-context learning of Text-to-SQL with self-correction. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  47. [50]

    Huaizhi Qu, Inyoung Choi, Zhen Tan, Song Wang, Sukwon Yun, Qi Long, Faizan Siddiqui, Kwonjoon Lee, and Tianlong Chen. 2025. Efficient MAP estimation of LLM judgment performance with prior transfer. arXiv preprint arXiv:2504.12589

    Pith/arXiv arXiv 2025

  48. [51]

    Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-consistency improves chain of thought reasoning in language models. In ICLR

    2023

  49. [52]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi. 2024. Can LLMs express their uncertainty? an empirical evaluation of confidence elicitation in LLMs . In ICLR

    2024

  50. [53]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2018. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and Text-to-SQL task. In Proceedings of EMNLP

    2018

  51. [54]

    Xing, Hao Zhang, Joseph E

    Lianmin Zheng, Wei-Lin Chiang, Ying Sheng, Siyuan Zhuang, Zhanghao Wu, Yonghao Zhuang, Zi Lin, Zhuohan Li, Dacheng Li, Eric P. Xing, Hao Zhang, Joseph E. Gonzalez, and Ion Stoica. 2023. Judging LLM-as-a-Judge with MT-Bench and chatbot arena. In Advances in Neural Information Processing Systems (NeurIPS)

    2023

  52. [55]

    Ziyang Zheng, Haipeng Jing, Canyu Rui, Askar Hamdulla, and Dong Wang. 2024. An actor-critic approach to boosting Text-to-SQL large language model. arXiv preprint arXiv:2410.22082

    Pith/arXiv arXiv 2024

  53. [56]

    Victor Zhong, Caiming Xiong, and Richard Socher. 2017. Seq2SQL : Generating structured queries from natural language using reinforcement learning. arXiv preprint arXiv:1709.00103

    Pith/arXiv arXiv 2017