Pith. sign in

REVIEW 4 major objections 5 minor 54 references

Confidence Estimation for Error Detection in Text-to-SQL Systems

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that entropy-based confidence scores let a selective classifier detect erroneous SQL generations, with T5 models better calibrated than GPT-4 or Llama 3.

desk verdict Careful empirical study of reject-option Text-to-SQL that ships code and uses an honest Dknown/Dunk protocol, but the abstract's T5-beats-GPT-4 claim is confounded by using different uncertainty estimators across model families. read the letter →

arxiv 2501.09527 v1 pith:FY6IVO6P submitted 2025-01-16 cs.LG cs.CL

Oleg Somov , Elena Tutubalina This is my paper · ORCID
classification cs.LGcs.CL
keywords text-to-SQLselectiveclassificationuncertaintyestimationcalibrationentropydistributionshifterrordetectionlargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper asks whether a Text-to-SQL system can be made safer by letting it abstain from answering when it is not confident, and whether that confidence can be trusted. It shows that an entropy-based score—the uncertainty of the model's weakest predicted token—fed into a Gaussian mixture selective classifier can detect erroneous SQL generations under distribution shift, cutting risk by about 80 percent on average across the tested models and splits. The paper's headline claim is that encoder-decoder T5 models are better calibrated than the in-context-learning GPT-4 system and decoder-only Llama 3, so the same selective classifier works better on T5 outputs. It also finds that the detector is more likely to catch queries generated in response to irrelevant or unanswerable questions than to catch incorrect queries for relevant questions. The practical stake is that a reject option, with proper calibration, could reduce the false positives that make text-to-SQL unreliable in real deployments.

What carries the argument

The engine is the maximum-token-entropy uncertainty estimate: for an autoregressive model, each predicted token carries a softmax distribution, and the sequence's confidence is set by the worst token, $u = \max_l H(p_l)$, where $H$ is Shannon entropy of the token distribution (for the closed GPT-4 system the paper substitutes the normalized sequence log-probability). This single score is the input to a Gaussian mixture selective classifier fit by expectation-maximization on a known out-of-distribution subset; the classifier treats the score as a draw from one of two components, correct or erroneous, and abstains when the erroneous component is more likely. Post-hoc calibration (Platt scaling and isotonic regression) maps the raw scores to probabilities, with isotonic regression giving the best Brier scores.

What would settle it

Rerun the calibration and selective-classification experiments using the same uncertainty estimator for every model family—for example, sequence log-probability for T5 and Llama, or token-level probabilities from GPT-4 if they can be obtained—and check whether the T5 advantage in Brier score and selective AUC persists; if it disappears, the paper's headline claim fails.

Watch

Extended reading notes

Core claim

The central claim is that a selective classifier built on a token-entropy confidence estimate can identify erroneous SQL generations in out-of-distribution settings, and that the reliability of this rejection mechanism tracks the model's calibration. Across four distribution-shift splits from PAUQ (cross-database, template, and target-length shifts) and EHRSQL (covariate shift with unanswerable questions), the entropy-based Gaussian mixture classifier reduces risk by an average of 80 percent, but at the cost of high false-discovery rates in full-shift settings. Under covariate shift alone, however, selective text-to-SQL keeps false discovery low—around 10 percent for the best models—and improves the net execution match. The paper's comparative conclusion is that encoder-decoder T5 models (large and 3B) are better calibrated after post-hoc calibration than the GPT-4-based DAIL-SQL and Llama 3 variants, and the Gaussian mixture selective classifier performs better on them; it is also more likely to flag generations for irrelevant questions than incorrect generations for relevant ones.

Load-bearing premise

The comparison across model families uses different uncertainty scores—maximum token entropy for T5 and Llama, but normalized sequence log-probability for GPT-4—so the conclusion that T5 is better calibrated could be an artifact of the score choice rather than a property of the architecture.

Editorial extensions

If this is right

  • If the central claim holds, a text-to-SQL deployment can attach a reject option to any generation model and abstain on the least-confident outputs, trading a bounded loss of coverage for a large drop in wrong SQL delivered to users.
  • The result that unanswerable questions are easier to catch than incorrect generations implies that production systems should treat OOD/unanswerable detection and in-scope error detection as separate channels with different thresholds.
  • Calibration under distribution shift does not restore reliability: even after isotonic regression, full-shift splits leave models uncalibrated, so a selective classifier trained on one OOD sample may not transfer to another.
  • The better calibration of encoder-decoder T5 suggests that for safety-critical text-to-SQL, architecture choice matters as much as accuracy when selecting a model.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper's uncertainty score for fine-tuned models is not truly an entropy of the whole sequence but the maximum over tokens, which ignores token position and dependence; a sequence-level product or mutual-information-based score might change the ranking between architectures.
  • A natural extension is to apply the same pipeline to benchmarks with dirty values and external-knowledge requirements to see if the covariate-shift finding generalizes beyond the clinical domain.
  • Because the Gaussian mixture assumes exactly two normal components, the selective classifier may be miscalibrated for multimodal error score distributions; fitting a density-ratio or a nonparametric threshold could improve FDR on the hardest TSL SSP split.
  • The paper stops short of a decision-theoretic analysis of abstention cost; in practice the threshold should be set by a user-specified cost ratio between a wrong SQL and a refused answer, which the reported recall/FDR tables make possible.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper studies error detection and calibration for Text-to-SQL systems under distribution shift. It defines an uncertainty score for generated SQL queries, fits three selective classifiers (threshold, logistic regression, and Gaussian mixture) on a known OOD sample Dknown, and evaluates on a held-out OOD sample Dunk. Experiments compare T5-large, T5-3B, DAIL-SQL with ChatGPT-4, and Llama-3-8B in SFT and LoRA variants on PAUQ (cross-database, template, and length splits) and EHRSQL. The main findings are that Gaussian Mixture gives the best coverage/risk trade-off, that unanswerable questions are detected more reliably than incorrect generations, that post-hoc calibration (especially isotonic regression) improves Brier scores, and that T5 models are better calibrated than the other families, leading to better selective classification.

Significance. If the architecture-level calibration claim held, the paper would be a useful practical contribution: it identifies an entropy-based reject option as a workable safety mechanism for Text-to-SQL, shows which calibration method is most effective, and provides new shifted splits for PAUQ. The paper has notable strengths: an honest Dknown/Dunk protocol for fitting calibration and selective classifiers, multiple random seeds, three selective classifiers reported (two in the appendix), and released code for splits and models. The central limitation is that the comparison across model families is confounded with the uncertainty estimator, so the headline conclusion needs additional evidence or rephrasing.

major comments (4)
  1. [3 (Uncertainty Estimate), Eqs. (1)-(2); Tables 1 and 3] The headline comparison of model families is confounded with the choice of uncertainty estimator. T5 and Llama use max-token entropy (Eq. 1), while DAIL-SQL/GPT-4 uses normalized sequence log-probability (Eq. 2). These are not interchangeable scores: one is the entropy of the weakest token, the other is an average log-likelihood over the generated sequence, and they have different scales, length sensitivities, and behavior under distribution shift. Because model family and estimator are perfectly aligned, the Brier-score advantage of T5 in Table 3 and the selective-classifier advantage in Table 1 may reflect the estimator rather than the architecture. Please re-run the comparison with at least one shared estimator across all models (e.g., normalized sequence log-probability for T5/Llama, and max-token entropy for DAIL-SQL if token log-probabilities are accessible), or explicitly restate the conclusion as a system-level comparison such as 'T5 with max-token entropy is better calibrated than DAIL-SQL with normalized sequence log-probability' rather than an architecture-level claim.
  2. [4 (Case Study #1), Tables 1-2] The metric 'Result EX = EM - FDR' is not a well-defined selective-classification performance measure. It subtracts the false discovery rate of the rejection decision from the execution match of the base generator, which conflates different populations and does not account for the reduction in coverage caused by abstention. A classifier that abstains on almost all correct queries can have a low FDR and therefore a high 'Result EX' even though the system is practically useless. The comparison of Gaussian Mixture versus Logistic Regression and Threshold in Table 2 is based on this metric, so the RQ1/RQ2 conclusion is not supported by the reported numbers. Please report standard risk-coverage curves or evaluate at fixed coverage levels, and use an aggregate such as the area under the risk-coverage curve.
  3. [5 (Case Study #2), Table 3] The central calibration claim rests on Brier scores averaged across splits with no confidence intervals, per-split breakdowns, or significance tests. The differences between T5-3B (0.106) and Llama3-8B SFT (0.175) or DIAL-SQL (0.152) could be real or could be within seed and split variance. Please provide standard deviations or bootstrap intervals and show per-split values; otherwise the 'better calibrated' claim in the abstract is not quantified with uncertainty.
  4. [3 (Selective Prediction Setting)] The prediction rule is stated as 'the system predicts the query if u >= gamma; otherwise it abstains,' but Eq. (1) defines u as max-token entropy, for which higher values mean lower confidence. Under the stated rule the system would predict exactly the high-entropy generations it should abstain on. This is the sign convention for a confidence score, not an uncertainty score. Please clarify the intended direction of the threshold or correct the inequality, since the threshold-based selective classifier is otherwise not reproducible.
minor comments (5)
  1. [Throughout] There are typos: 'descent ability' (Sec. 3), 'utulize' (Sec. 3), 'Platts' (Sec. 5), and 'ERHSQL' (Sec. 4) should be corrected.
  2. [4, Table 1 caption] The caption calls recall 'the coverage of our selective classifier,' but recall is the fraction of detected errors, while coverage is the fraction of all test examples that are predicted; these concepts should be separated.
  3. [4, Fig. 2] Gaussian Mixture appears to be selected based on the same evaluation data used to report final numbers; since the other classifiers are reported in the appendix, this is acceptable as an exploratory comparison, but it should be stated explicitly that this involves model selection on the evaluation set.
  4. [5, Eq. (6)] The notation lists theta_0, ..., theta_M while the objective uses theta_m; the index ranges should be aligned.
  5. [3 (Text-to-SQL Models)] The term 'inverted execution match metric (EX)' is confusing; execution match is normally denoted EM, and calling it 'inverted' obscures the standard definition.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity; the held-out Dknown/Dunk protocol keeps the central empirical claims self-contained.

full rationale

The paper's central claims are empirical and are supported by a standard honest evaluation protocol: selective classifiers and calibrators are fit on Dknown and evaluated on Dunk (Sec. 3, 'Selective Prediction Setting'), with results reported per split in Tables 1-3. The Gaussian mixture, logistic regression, Platt, and isotonic parameters are all fitted on the known OOD sample and assessed on the unknown OOD sample, so no fitted parameter is renamed as a prediction. Equation 1 (max token entropy) and Eq. 2 (normalized sequence log-probability) are different uncertainty scores used for different model families; this is a legitimate experimental-design concern about comparing calibration across architectures, but it is not a circular reduction because the Brier scores and selective-classification metrics are computed from those scores rather than being defined as those scores. Self-citations to PAUQ (Bakshandaeva et al. 2022) and Shifted PAUQ (Somov and Tutubalina 2023) provide datasets and split constructions, not the theoretical content of the calibration or selective-classification claims, and the cited splits are applied as evaluation benchmarks rather than as evidence for the derivation. No uniqueness theorem or ansatz is imported from prior work to force the conclusions; the paper explicitly follows Yang et al. (2024) for the max-entropy estimate, which is a stated modeling choice rather than a hidden circular premise. The 'Result EX = EM - FDR' metric is nonstandard but is presented as a defined summary, not as a derivation from the model's confidence estimates. Overall, the derivation chain from uncertainty estimates to selective performance and calibration does not reduce to its inputs by construction.

Assumptions & free parameters 4 free parameters · 5 assumptions · 0 invented entities

The paper rests on standard autoregressive modeling and on two task-specific assumptions: max token entropy captures sequence risk, and execution-match equality labels correctness. A further comparability assumption (max entropy vs. normalized sequence log probability) is load-bearing for the cross-family comparison but is not defended. No new entities are introduced.

free parameters (4)
  • Gaussian mixture cluster parameters (means, standard deviations, weights for error and correct clusters) = Fitted per split on Dknown; values not reported in paper
    Used in Eq. 5 to classify each uncertainty score as error or correct; determines all selective rejection decisions.
  • Platt calibration logistic coefficients (theta_0, theta_1) = Fitted on Dknown via maximum likelihood; values not reported
    Used in Eq. 4 to map uncertainty scores to calibrated probabilities; central to the calibration comparison in Sec. 5.
  • Isotonic regression step heights and interval boundaries = Fitted on Dknown; values not reported
    Piecewise constant calibration function from Eq. 6; used to produce best Brier scores in Table 3.
  • Logistic regression selective classifier coefficients (theta_0, theta_1) = Fitted per split; values not reported
    Alternative selective classifier in Eq. 4; comparison baseline in Tables 4 and 5.
assumptions (5)
  • standard math Autoregressive factorization of sequence probability holds for the generated SQL (P(y|x,theta) = product over token predictions).
    Used in Sec. 3 to define token-level probabilities and entropy; standard for the T5 and Llama model classes.
  • domain assumption Maximum token entropy over the generated sequence is a valid proxy for sequence-level uncertainty.
    The paper states 'a sequence is only as good as its weakest token prediction' (Sec. 3) without proof; all fine-tuned model results depend on this heuristic.
  • domain assumption Execution match equality (EX(gi)==EX(pi)) is a valid label for query correctness.
    Eq. 3 defines the positive error class as mismatched execution results; SQL queries that differ semantically but return the same result are treated as correct.
  • ad hoc to paper Max token entropy (Eq. 1) and normalized sequence log probability (Eq. 2) are comparable confidence scores across model families.
    The comparison of calibration and selective performance across T5/Llama and GPT-4/DAIL-SQL relies on this comparability, which the paper does not justify.
  • domain assumption The designed template and length splits of PAUQ isolate compositional generalization.
    The paper assumes template shift and target-length shift test systematicity and productivity, following the taxonomy of Hupkes et al., but does not validate that these splits avoid other confounds.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Confidence Estimation for Error Detection in Text-to-SQL Systems." pith.science (2026). https://pith.science/paper/FY6IVO6P

@misc{pith2026250109527,
  author       = {Pith},
  title        = {Pith review of: Confidence Estimation for Error Detection in Text-to-SQL Systems},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/FY6IVO6P}},
  note         = {Machine review of arXiv:2501.09527}
}
read the original abstract

Text-to-SQL enables users to interact with databases through natural language, simplifying the retrieval and synthesis of information. Despite the success of large language models (LLMs) in converting natural language questions into SQL queries, their broader adoption is limited by two main challenges: achieving robust generalization across diverse queries and ensuring interpretative confidence in their predictions. To tackle these issues, our research investigates the integration of selective classifiers into Text-to-SQL systems. We analyse the trade-off between coverage and risk using entropy based confidence estimation with selective classifiers and assess its impact on the overall performance of Text-to-SQL models. Additionally, we explore the models' initial calibration and improve it with calibration techniques for better model alignment between confidence and accuracy. Our experimental results show that encoder-decoder T5 is better calibrated than in-context-learning GPT 4 and decoder-only Llama 3, thus the designated external entropy-based selective classifier has better performance. The study also reveal that, in terms of error detection, selective classifier with a higher probability detects errors associated with irrelevant questions rather than incorrect query generations.

Figures

Figures reproduced from arXiv: 2501.09527 by the authors.

Figure 1
Figure 1. The interaction scenario with Text-to-SQL system. [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. Heatmaps of Fβ=1 per split and model for every selective classifier (Logistic Regression, Gaussian Mixture, and Threshold). Distribution Shift in Text-to-SQL We evaluate the uncertainty estimates of Text-to-SQL mod￾els in distribution shift settings, mimicking various types of shifts: domain shift, compositional shift, and covariate shift. Domain and compositional shifts are full shift exam￾ples where p(xtst) ̸= p(x… view at source ↗
Figure 3
Figure 3. Left: The system risk decrease with a Gaussian Mixture for every split averaged between all SQL generation models. Right: The system coverage decrease with the presence of an Gaussian Mixture external classifier for every split averaged between all SQL generation models. Recall FDR Result EX Gaussian Mixture 0.798 0.364 0.251 Logistic Regression 0.873 0.469 0.145 Threshold 0.872 0.471 0.143 [PITH_FULL_IMAGE:figures… view at source ↗
Figures from the paper (7 more)
Figure 4
Figure 4. Figure 4: The calibration effect on T5-3B on PAUQ XSP (cross-database setting) and EHRSQL (single clinical database) compared across MinMax, Platts, and Isotonic cal￾ibration (BS stands for Brier score). In this section, we will investigate the following research question (RQ5):…
Figure 5
Figure 5. Figure 5: Trade-off plots between execution match and calibration for selected Text-to-SQL models (T5-large, T5-3B, Llama 3 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png]
Figure 6
Figure 6. Figure 6: Top: Fβ scores for β ∈ [0; 1) – favoring precision over recall. Bottom: Fβ scores for β ∈ [1; 5]. – favoring recall over precision. D Error Detection Tables [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]
Figure 7
Figure 7. Figure 7: ROC curves for selected Text-to-SQL models across splits with calculated AUC-ROC score for Logistic Regression [PITH_FULL_IMAGE:figures/full_fig_p013_7.png]
Figure 8
Figure 8. Figure 8: The bar plot for the EHRSQL dataset shows the detection rates of unanswerable queries (NULL bar) and low gener [PITH_FULL_IMAGE:figures/full_fig_p014_8.png]
Figure 9
Figure 9. Figure 9: Calibration results using isotonic regression for each model across different splits. [PITH_FULL_IMAGE:figures/full_fig_p014_9.png]
Figure 10
Figure 10. Figure 10: Four scatter plots comparing different splits: PAUQ XSP, Template SSP split, TSL SSP split, and EHRSQL sampled [PITH_FULL_IMAGE:figures/full_fig_p015_10.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

54 extracted references · 36 canonical work pages

  1. [1]

    , " * write output.state after.block = add.period write newline

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint howpublished institution isbn journal key month note number organization pages publisher school series title type volume year label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block FUNCTION init.state.consts #0 'before.a...

  2. [2]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  3. [3]

    URL: " 'urlintro :=

    ENTRY address author booktitle chapter edition editor howpublished institution journal key month note number organization pages publisher school series title type volume year eprint doi pubmed url lastchecked label extra.label sort.label short.list INTEGERS output.state before.all mid.sentence after.sentence after.block STRINGS urlintro eprinturl eprintpr...

  4. [4]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 global.max substring 't := if while FUNCTION word.in bbl.in capitalize " " * FUNCT...

  5. [5]

    L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al

    Achiam, J.; Adler, S.; Agarwal, S.; Ahmad, L.; Akkaya, I.; Aleman, F. L.; Almeida, D.; Altenschmidt, J.; Altman, S.; Anadkat, S.; et al. 2023. Gpt-4 technical report. arXiv preprint arXiv:2303.08774

    arXiv 2023

  6. [6]

    Ashukha, A.; Lyzhov, A.; Molchanov, D.; and Vetrov, D. 2020. Pitfalls of in-domain uncertainty estimation and ensembling in deep learning. arXiv preprint arXiv:2002.06470

    arXiv 2020

  7. [7]

    Bakshandaeva, D.; Somov, O.; Dmitrieva, E.; Davydova, V.; and Tutubalina, E. 2022. PAUQ : Text-to- SQL in R ussian. In Findings of the Association for Computational Linguistics: EMNLP 2022, 2355--2376. Abu Dhabi, United Arab Emirates: Association for Computational Linguistics

    work page 2022

  8. [8]

    H.; Lan, W.; Zhang, S.; Jiang, J.; Lilien, J.; et al

    Chang, S.; Wang, J.; Dong, M.; Pan, L.; Zhu, H.; Li, A. H.; Lan, W.; Zhang, S.; Jiang, J.; Lilien, J.; et al. 2023. Dr. Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness. In The Eleventh International Conference on Learning Representations

    work page 2023

Show all 54 references
  1. [9]

    Chen, S.; Chen, Z.; Sun, H.; and Su, Y. 2023. Error detection for text-to-sql semantic parsing. In Findings of the Association for Computational Linguistics: EMNLP 2023, 11730--11743

    2023

  2. [10]

    Chow, C. 1970. On optimum recognition error and reject tradeoff. IEEE Transactions on information theory, 16(1): 41--46

    1970

  3. [11]

    Chow, C. K. 1957. An optimum character recognition system using decision functions. IRE Transactions on Electronic Computers, EC-6(4): 247--254

    1957

  4. [12]

    Dong, L.; Quirk, C.; and Lapata, M. 2018. Confidence Modeling for Neural Semantic Parsing. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 743--753. Melbourne, Australia: Association for Computational Linguistics

    2018

  5. [13]

    El-Yaniv, R.; and Wiener, Y. 2010. On the Foundations of Noise-free Selective Classification. Journal of Machine Learning Research, 11(53): 1605--1641

    2010

  6. [14]

    Fadeeva, E.; Vashurin, R.; Tsvigun, A.; Vazhentsev, A.; Petrakov, S.; Fedyanin, K.; Vasilev, D.; Goncharova, E.; Panchenko, A.; Panov, M.; Baldwin, T.; and Shelmanov, A. 2023. LM -Polygraph: Uncertainty Estimation for Language Models. In Proceedings of the 2023 Conference on E...

    2023

  7. [15]

    K.; Zhang, L.; Ramanathan, K.; Sadasivam, S.; Zhang, R.; and Radev, D

    Finegan-Dollak, C.; Kummerfeld, J. K.; Zhang, L.; Ramanathan, K.; Sadasivam, S.; Zhang, R.; and Radev, D. 2018 a . Improving Text-to- SQL Evaluation Methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers),...

    2018

  8. [16]

    K.; Zhang, L.; Ramanathan, K.; Sadasivam, S.; Zhang, R.; and Radev, D

    Finegan-Dollak, C.; Kummerfeld, J. K.; Zhang, L.; Ramanathan, K.; Sadasivam, S.; Zhang, R.; and Radev, D. 2018 b . Improving Text-to-SQL Evaluation Methodology. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), 351--360

    2018

  9. [17]

    Gan, Y.; Chen, X.; and Purver, M. 2021. Exploring Underexplored Limitations of Cross-Domain Text-to-SQL Generalization. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, 8926--8931

    2021

  10. [18]

    Gao, D.; Wang, H.; Li, Y.; Sun, X.; Qian, Y.; Ding, B.; and Zhou, J. 2024. Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation. Proceedings of the VLDB Endowment, 17(5): 1132--1145

    2024

  11. [19]

    Guo, C.; Pleiss, G.; Sun, Y.; and Weinberger, K. Q. 2017. On calibration of modern neural networks. In International conference on machine learning, 1321--1330. PMLR

    2017

  12. [20]

    Hendrycks, D.; and Gimpel, K. 2017. A Baseline for Detecting Misclassified and Out-of-Distribution Examples in Neural Networks. Proceedings of International Conference on Learning Representations

    2017

  13. [21]

    J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W

    Hu, E. J.; Shen, Y.; Wallis, P.; Allen-Zhu, Z.; Li, Y.; Wang, S.; Wang, L.; and Chen, W. 2022. Lo RA : Low-Rank Adaptation of Large Language Models. In International Conference on Learning Representations

    2022

  14. [22]

    Hupkes, D.; Dankers, V.; Mul, M.; and Bruni, E. 2020. Compositionality Decomposed: How do Neural Networks Generalise? (Extended Abstract). In Proceedings of the Twenty-Ninth International Joint Conference on Artificial Intelligence, IJCAI-20 , 5065--5069. International Joint C...

    2020

  15. [23]

    Hupkes, D.; Giulianelli, M.; Dankers, V.; Artetxe, M.; Elazar, Y.; Pimentel, T.; Christodoulopoulos, C.; Lasri, K.; Saphra, N.; Sinclair, A.; Ulmer, D.; Schottmann, F.; Batsuren, K.; Sun, K.; Sinha, K.; Khalatbari, L.; Ryskina, M.; Frieske, R.; Cotterell, R.; and Jin, Z. 2023....

    2023 arXiv

  16. [24]

    Kadavath, S.; Conerly, T.; Askell, A.; Henighan, T.; Drain, D.; Perez, E.; Schiefer, N.; Hatfield-Dodds, Z.; DasSarma, N.; Tran-Johnson, E.; Johnston, S.; El-Showk, S.; Jones, A.; Elhage, N.; Hume, T.; Chen, A.; Bai, Y.; Bowman, S.; Fort, S.; Ganguli, D.; Hernandez, D.; Jacobs...

    2022 arXiv

  17. [25]

    Kim, S.; Han, D.; and Kim, S. 2024. P rob G ate at EHRSQL 2024: Enhancing SQL Query Generation Accuracy through Probabilistic Threshold Filtering and Error Handling. In Proceedings of the 6th Clinical Natural Language Processing Workshop, 687--696. Mexico City, Mexico: Associa...

    2024

  18. [26]

    Lake, B.; and Baroni, M. 2018. Generalization without systematicity: On the compositional skills of sequence-to-sequence recurrent networks. In International conference on machine learning, 2873--2882. PMLR

    2018

  19. [27]

    Lakshminarayanan, B.; Pritzel, A.; and Blundell, C. 2017. Simple and scalable predictive uncertainty estimation using deep ensembles. Advances in neural information processing systems, 30

    2017

  20. [28]

    Lee, C.-H.; Polozov, O.; and Richardson, M. 2021. KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers. In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing (Vol...

    2021

  21. [29]

    Lee, G.; Chay, W.; Cho, S.; and Choi, E. 2024 a . TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring. arXiv:2403.15879

    2024 arXiv

  22. [30]

    Lee, G.; Hwang, H.; Bae, S.; Kwon, Y.; Shin, W.; Yang, S.; Seo, M.; Kim, J.-Y.; and Choi, E. 2022. Ehrsql: A practical text-to-sql benchmark for electronic health records. Advances in Neural Information Processing Systems, 35: 15589--15601

    2022

  23. [31]

    Lee, G.; Kweon, S.; Bae, S.; and Choi, E. 2024 b . Overview of the EHRSQL 2024 Shared Task on Reliable Text-to- SQL Modeling on Electronic Health Records. In Proceedings of the 6th Clinical Natural Language Processing Workshop, 644--654. Association for Computational Linguistics

    2024

  24. [32]

    Li, J.; Hui, B.; Qu, G.; Yang, J.; Li, B.; Li, B.; Wang, B.; Qin, B.; Geng, R.; Huo, N.; et al. 2024. Can llm already serve as a database interface? a big bench for large-scale database grounded text-to-sqls. Advances in Neural Information Processing Systems, 36

    2024

  25. [33]

    Malinin, A.; and Gales, M. 2021. Uncertainty Estimation in Autoregressive Structured Prediction. arXiv:2002.07650

    2021 arXiv

  26. [34]

    Meta, A. 2024. Introducing meta llama 3: The most capable openly available llm to date. Meta AI

    2024

  27. [35]

    Platt, J.; et al. 1999. Probabilistic outputs for support vector machines and comparisons to regularized likelihood methods. Advances in large margin classifiers, 10(3): 61--74

    1999

  28. [36]

    Qin, B.; Wang, L.; Hui, B.; Li, B.; Wei, X.; Li, B.; Huang, F.; Si, L.; Yang, M.; and Li, Y. 2022. SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers. arXiv:2209.06442

    2022 arXiv

  29. [37]

    Raffel, C.; Shazeer, N.; Roberts, A.; Lee, K.; Narang, S.; Matena, M.; Zhou, Y.; Li, W.; and Liu, P. J. 2020. Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer. Journal of Machine Learning Research, 21(140): 1--67

    2020

  30. [38]

    Ren, J.; Luo, J.; Zhao, Y.; Krishna, K.; Saleh, M.; Lakshminarayanan, B.; and Liu, P. J. 2023. Out-of-Distribution Detection and Selective Generation for Conditional Language Models. arXiv:2209.15558

    2023 arXiv

  31. [39]

    Shaw, P.; Chang, M.-W.; Pasupat, P.; and Toutanova, K. 2021. Compositional Generalization and Natural Language Variation: Can a Semantic Parsing Approach Handle Both? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th Intern...

    2021

  32. [40]

    Shazeer, N.; and Stern, M. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. In International Conference on Machine Learning, 4596--4604. PMLR

    2018

  33. [41]

    Somov, O.; and Tutubalina, E. 2023. Shifted PAUQ : Distribution shift in text-to- SQL . In Proceedings of the 1st GenBench Workshop on (Benchmarking) Generalisation in NLP, 214--220. Singapore: Association for Computational Linguistics

    2023

  34. [42]

    Stengel-Eskin, E.; and Van Durme, B. 2023. Calibrated Interpretation: Confidence Estimation in Semantic Parsing. Transactions of the Association for Computational Linguistics, 11: 1213--1231

    2023

  35. [43]

    Suhr, A.; Chang, M.-W.; Shaw, P.; and Lee, K. 2020. Exploring Unexplored Generalization Challenges for Cross-Database Semantic Parsing. In Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics, 8372--8388. Online: Association for Computational...

    2020

  36. [44]

    Sun, K.; Williams, A.; and Hupkes, D. 2023. A replication study of compositional generalization works on semantic parsing. In ML Reproducibility Challenge 2022

    2023

  37. [45]

    Ueffing, N.; and Ney, H. 2005. Word-Level Confidence Estimation for Machine Translation using Phrase-Based Translation Models. In Proceedings of Human Language Technology Conference and Conference on Empirical Methods in Natural Language Processing, 763--770. Vancouver, Britis...

    2005

  38. [46]

    Ueffing, N.; and Ney, H. 2007. Word-level confidence estimation for machine translation. Computational Linguistics, 33(1): 9--40

    2007

  39. [47]

    van der Poel, L.; Cotterell, R.; and Meister, C. 2022. Mutual Information Alleviates Hallucinations in Abstractive Summarization. In Proceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, 5956--5965. Association for Computational Linguistics

    2022

  40. [48]

    Vazhentsev, A.; Tsvigun, A.; Vashurin, R.; Petrakov, S.; Vasilev, D.; Panov, M.; Panchenko, A.; and Shelmanov, A. 2023. Efficient Out-of-Domain Detection for Sequence to Sequence Models. In Findings of the Association for Computational Linguistics: ACL 2023, 1430--1454. Toront...

    2023

  41. [49]

    G.; Biswas, R.; Ahmadi, A.; and Holmstr \"o m, O

    Wretblad, N.; Riseby, F. G.; Biswas, R.; Ahmadi, A.; and Holmstr \"o m, O. 2024. Understanding the Effects of Noise in Text-to-SQL: An Examination of the BIRD-Bench Benchmark. arXiv preprint arXiv:2402.12243

    2024 arXiv

  42. [50]

    Yang, Y.; Kim, S.; Kim, S.; Lee, G.; Yun, S.-Y.; and Choi, E. 2024. Towards Unbiased Evaluation of Detecting Unanswerable Questions in EHRSQL. arXiv preprint arXiv:2405.01588

    2024 arXiv

  43. [51]

    Yu, T.; Zhang, R.; Yang, K.; Yasunaga, M.; Wang, D.; Li, Z.; Ma, J.; Li, I.; Yao, Q.; Roman, S.; et al. 2018. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. In Proceedings of the 2018 Conference on Empirical Meth...

    2018

  44. [52]

    Zadrozny, B.; and Elkan, C. 2002. Transforming classifier scores into accurate multiclass probability estimates. In Proceedings of the eighth ACM SIGKDD international conference on Knowledge discovery and data mining, 694--699

    2002

  45. [53]

    M.; and Mooney, R

    Zelle, J. M.; and Mooney, R. J. 1996. Learning to parse database queries using inductive logic programming. In Proceedings of the national conference on artificial intelligence, 1050--1055

    1996

  46. [54]

    S.; and Collins, M

    Zettlemoyer, L. S.; and Collins, M. 2005. Learning to map sentences to logical form: structured classification with probabilistic categorial grammars. In Proceedings of the Twenty-First Conference on Uncertainty in Artificial Intelligence, UAI'05, 658–666. AUAI Press. ISBN 0974903914

    2005

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.