Pith. sign in

REVIEW 4 major objections 6 minor 38 references

ODIN: A NL2SQL Recommender to Handle Schema Ambiguity

T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read ODIN's central claim is that a generate-select-personalize pipeline yields a candidate SQL set that contains the correct query 1.5–2x more often than sampling or forced-diversity baselines, while showing 2–3x fewer queries.

desk verdict A credible Generate-Select-Personalize pipeline for ambiguous NL2SQL, but the headline gain is inflated by an unequal LLM budget and a self-created benchmark. read the letter →

arxiv 2505.19302 v1 pith:GVNGIX42 submitted 2025-05-25 cs.DB cs.CL

Kapil Vaidya , Abishek Sankararaman , Jialin Ding , Chuan Lei , Xiao Qin , Balakrishnan Narayanaswamy , Tim Kraska This is my paper
classification cs.DBcs.CL
keywords NL2SQLschemaambiguitySQLqueryrecommendationmaskingconformalpredictionpersonalizationLLMexecutionmatchaccuracy
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

ODIN is a recommendation engine for natural-language-to-SQL (NL2SQL) that tries to solve schema ambiguity—the situation in which one question can be answered by multiple tables or columns with similar names. Instead of returning a single SQL query, it returns a small set of candidate queries, sized dynamically by how ambiguous the question is, and learns from the user's choice to personalize later suggestions. The paper claims this generate-select-personalize design raises the chance that the correct query is in the result set by 1.5–2× compared with two baselines (high-temperature sampling and forced diversity) while making the result set 2–3× smaller. The authors argue this matters because enterprise schemas are large and ambiguous, and a single-query NL2SQL system can silently pick the wrong table or column.

What carries the argument

The load-bearing mechanism is the masked-schema generator: after the LLM produces a query for a schema, the columns used in that query are removed one at a time to create child schemas, and the LLM is called again on each; because the old columns are invisible, it must pick alternatives such as origin instead of birthplace. Exploration is guided by a priority queue scored with Cal_Score, the minimum over question entities of their best entity-column similarity, and duplicate schemas are discarded to stay within a fixed LLM-call budget. The selector wraps this in conformal prediction, using a calibration set to set a threshold on an LLM-logit or SBERT-based scoring function so that, with probability at least 1−α, the selected set still contains a query whose execution matches the ground truth. The personalizer completes the loop by schema-linking entities in the question to columns in the chosen SQL and emitting textual hints to bias the next generation.

What would settle it

Construct a workload where the same entity phrase maps to different correct schema components depending on the question (e.g., 'sales' sometimes means gross_sales, sometimes net_sales) and measure ODIN's AvgAcc at K=1 and K=10 against forced diversity at matched result-set sizes; if the personalization lifts of 30%, 8%, and 30% disappear or ODIN no longer beats forced diversity, the central claim fails.

Watch

Extended reading notes

Core claim

The paper's central discovery claim is that schema masking—removing specific columns or tables that appeared in previously generated SQL from the prompt—forces an LLM to explore genuinely different schema components, producing SQL queries that differ in execution result rather than only in syntax. A greedy, budget-limited tree search with duplicate-schema detection and a relevance score keeps generation efficient; a conformal-prediction selector then prunes candidates while guaranteeing that the correct query is retained with probability at least 1−α; and a personalizer converts the user's selection into textual hints that bias future generation and scoring. On the modified AmbiQT benchmark, ODIN at a 10-call LLM budget reaches 71.6%, 50.3%, and 73.6% execution-match accuracy for join, table, and column ambiguity, versus 33.2%, 38.0%, and 58.8% for forced diversity, while showing about half the candidates.

Load-bearing premise

The end-to-end evaluation assumes that a phrase in a user's question maps consistently to the same schema component across questions, so that preferences learned from one selection generalize to future questions; if user preferences are context-dependent or drift, the personalization gains may shrink.

Editorial extensions

If this is right

  • At a budget of 10 LLM calls, ODIN reaches 71.6%, 50.3%, and 73.6% execution-match accuracy for join, table, and column ambiguity, versus 33.2%, 38.0%, and 58.8% for forced diversity.
  • The conformal selector cuts the average number of SQL queries shown from 10 to roughly 4–5 while lowering accuracy by only 1–1.6 percentage points.
  • Personalization alone lifts K=1 accuracy by 30%, 8%, and 30% for join, table, and column ambiguity, showing that learned preferences can get the right query on the first suggestion.
  • Even without personalization, ODIN still outperforms forced diversity by 27%, 10%, and 8% at 10 LLM calls once the generator and selector are active.
  • The mask-based generator covers both correct alternatives on AmbiQT up to 1.9x, 1.2x, and 1.5x more often than Logical Beam for join, table-synonym, and column-synonym ambiguity.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Extension: a sliding window over recent user feedback, mentioned but not implemented in the paper, could be added to handle preference drift and would make the personalization claim testable over longer horizons.
  • Extension: the same mask-a-used-component generation loop applies to any structured output space with schema-like alternatives, such as API calls or configuration queries, not just SQL.
  • Extension: the sized candidate set could double as an ambiguity signal; when it stays large, a clarification question may be cheaper than showing the user many SQL options.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 6 minor

Summary. ODIN is a generate-select-personalize NL2SQL system for ambiguous database schemas. Given a natural-language question, the Generator uses budget-limited greedy schema masking to produce a diverse set of candidate SQL queries; the Selector applies conformal prediction with an LLM-based or SBERT-based scoring function to prune candidates while preserving a conditional recall guarantee; and the Personalizer converts user feedback into textual hints that bias future generation and selection. The paper evaluates the Generator on the AmbiQT benchmark and the full system on a newly constructed Mod-AmbiQT benchmark, reporting that ODIN improves execution-match accuracy by 1.5–2× over sampling and forced-diversity baselines at the same generator LLM budget K while returning smaller result sets.

Significance. If the results hold, ODIN is a valuable step toward handling schema ambiguity and preference learning in NL2SQL. The schema-masking generator addresses a known failure mode of high-temperature sampling, the conformal selector provides a principled way to trade result-set size against recall, and the personalization mechanism is a practical use of user feedback. The paper also introduces Mod-AmbiQT, a benchmark for evaluating multi-candidate generation with a single designated correct query. However, the headline empirical claim is currently weakened by an unequal total LLM budget between ODIN and the baselines, by reliance on a self-constructed benchmark with fixed entity-schema mappings, and by an abstract-level overstatement relative to the per-ambiguity numbers. These issues are addressable but require additional experiments and reporting before the central claim can be accepted as stated.

major comments (4)
  1. [Section 8.2, Figure 5] The headline comparison at fixed generator budget K is not an equal-resource comparison. Section 8.2 states that "K does not include LLM calls made by the Selector and Personalizer." The Selector's default scoring function is LLM-based (Section 6.3.1) and requires a logit-probability evaluation for every candidate query, so at K=10 it adds at least 10 LLM evaluations beyond the Generator. In addition, the Generator's Cal_Score calls Extract_Entities, which "identifies entities in the user's question with an LLM call" (Section 5.3), adding more calls during search. Sampling and Forced Diversity consume exactly K LLM calls. The reported 1.5–2× accuracy advantage and 2–3× smaller set size are therefore confounded with total inference budget. Please report total LLM call counts and wall-clock time, or give the baselines an equivalent budget (e.g., additional samples or reranking) so the contributions of schema masking, conformal selection, and personalization are isolated.
  2. [Section 8.2, Figure 5] The abstract's "1.5–2×" claim is not uniformly supported by the reported per-ambiguity numbers. At K=10, the execution-match accuracy ratios of ODIN over Forced Diversity are 71.6/33.2 ≈ 2.16 for join ambiguity, 50.3/38.0 ≈ 1.32 for table ambiguity, and 73.6/58.8 ≈ 1.25 for column ambiguity. Only the join-ambiguity case reaches the 1.5× bound; table and column ambiguity fall below it. The phrase "up to twice" in Section 8.2 is accurate, but the abstract and Section 1 should be revised to either state the per-type results or report a workload-averaged ratio consistent with the data.
  3. [Section 8.1 (Mod-AmbiQT) and Section 7.3] The end-to-end evaluation, including the personalization results, is measured on Mod-AmbiQT, a benchmark constructed by the authors in which "the entity singers consistently maps to artist across all questions in the database" (Section 8.1). That fixed entity-schema mapping is exactly what allows the Personalizer to transfer feedback from one question to the next. Section 7.3 acknowledges that user preferences can drift or be context-dependent; in such cases the K=1 gains (30%, 8%, and 30% for join, table, and column ambiguity) may not transfer. The paper should evaluate end-to-end accuracy on the original AmbiQT benchmark, or add a Mod-AmbiQT variant with inconsistent or drifting entity-schema mappings, to support the generality of the headline claim.
  4. [Section 8.2 and Figure 5] No variance information is reported for any of the accuracy or result-size numbers. The Generator, Selector, and baselines all involve stochastic LLM sampling (e.g., temperature 1.0 for the Sampling baseline), so single-run point estimates are insufficient to establish the stated advantages, particularly for the smaller gaps in table and column ambiguity. Please report means and standard deviations over at least several seeds, or otherwise justify that the comparisons are deterministic.
minor comments (6)
  1. [Section 1, footnote 1] Footnote 1 says the evaluation uses a modified version of AmbiQT, but the text immediately before it says the evaluation is "on the AmbiQT benchmark"; clarify that the headline end-to-end result is on Mod-AmbiQT.
  2. [Section 8.2] "Improvements of 38%, 12%, and 15% in accuracy" should distinguish percentage-point gains from relative improvements; for join ambiguity, 71.6 minus 33.2 is 38.4 percentage points, not a 38% relative improvement.
  3. [Section 8.2 and abstract] The abstract and introduction say the result set is "2–2.5× smaller," while Section 8.2 says "2–3× fewer SQL queries"; align these statements.
  4. [Section 6.1 and Section 6.2] The conformal coverage guarantee is conditional on the generator containing a correct query (Eq. 1). Section 6.1's Theorem 6.1 states a marginal guarantee; the mapping in Section 6.2 should explicitly state that the exchangeability argument is applied to the filtered subpopulation where the correct query is present, so readers do not infer an unconditional recall guarantee.
  5. [Algorithm 3] In the caption of Algorithm 3, "entity ans schema components" should read "entity and schema components."
  6. [Section 8.1] Specify the exact version of Claude 3 Haiku and the sampling hyperparameters used for all methods to support reproducibility.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: ODIN's generator, conformal selector, and personalizer are self-contained and evaluated against externally defined ground-truth SQL labels.

full rationale

The paper's derivation chain is not circular. The Generator's schema-masking algorithm scores candidate masked schemas using an LLM-based entity extractor and SBERT similarity (Algorithms 1 and 2), which are external, pre-trained components not defined in terms of the target metric. The Selector uses the conformal prediction framework, with a threshold calibrated on a calibration set of (question, generated-query-set) pairs; the coverage guarantee in Theorem 6.1 is a standard mathematical result and does not assume the target outcome. The Personalizer learns textual hints from explicit user feedback via a separate SBERT-based schema-linking subroutine (Algorithm 3), and these hints are then supplied to the Generator and Selector; this is a learning loop, not a definitional identity. The evaluation uses AmbiQT [2], an external benchmark, and Mod-AmbiQT, a modified benchmark in which the authors designate one of several plausibly correct queries as correct. While the consistency assumption in Mod-AmbiQT ('The entity singers consistently maps to artist across all questions') is a synthetic condition favorable to personalization, the system still must learn that mapping from feedback; the accuracy numbers are not forced by the label-assignment procedure. No load-bearing self-citations appear: reference [2] is by different authors and provides the benchmark and baselines, not a uniqueness theorem or ansatz. The concern that ODIN's reported K excludes Selector and Personalizer LLM calls is a resource-accounting limitation of the comparison, not a circular reduction of the claimed result to its inputs. Accordingly, no circular step meeting the evidence threshold is present.

Assumptions & free parameters 2 free parameters · 4 assumptions · 1 invented entities

The central claim rests on semantic similarity measures, the conformal calibration setup, and the synthetic Mod-AmbiQT evaluation. No physical entities or forces are introduced.

free parameters (2)
  • LLM call budget K = 1, 2, 3, 5, 7, 10
    Chosen by hand and swept in the evaluation. Directly controls the number of generated candidates and therefore the accuracy versus result-set-size trade-off.
  • Conformal significance level alpha = 0.01 to 0.1
    Chosen by hand and swept in the Selector ablation. Higher alpha discards more candidates, and the coverage guarantee depends on this parameter.
assumptions (4)
  • domain assumption There is a single ground-truth SQL query capturing user intent for each ambiguous question.
    Stated in Section 3. Drives the accuracy metric and the conformal formulation, and may fail when multiple queries are equally acceptable to the user.
  • domain assumption Calibration and test questions are exchangeable, conditional on the generator having produced a correct query.
    Required for the conformal coverage guarantee in Section 6.2. Not validated on real workloads or across different schemas.
  • domain assumption SBERT and LLM similarity scores reflect whether a schema element or SQL column correctly represents an entity in the question.
    Used by Cal_Score in Section 5.3 and by both Selector scoring functions in Section 6.3. If the similarity signal is wrong, the masking, ranking, and pruning decisions fail.
  • ad hoc to paper Masking columns used in prior SQL queries eventually exposes all plausible alternative columns and tables for ambiguous entities.
    This is the core heuristic behind the Generator in Section 5.2. The paper provides empirical evidence on two benchmarks but no completeness guarantee.
invented entities (1)
  • Mod-AmbiQT benchmark
    purpose: Synthetic evaluation benchmark with duplicate tables and columns, designated correct mappings, and 2 to 8 plausible SQL queries per question. Used for the main end-to-end evaluation in Section 8.2.
    Not released, and correctness labels are assigned by the authors to enforce consistent entity-to-schema mappings. It cannot serve as an external check of the personalization claims.

how reviews work

0 comments
Cite this review

Pith. "Pith review of ODIN: A NL2SQL Recommender to Handle Schema Ambiguity." pith.science (2026). https://pith.science/paper/GVNGIX42

@misc{pith2026250519302,
  author       = {Pith},
  title        = {Pith review of: ODIN: A NL2SQL Recommender to Handle Schema Ambiguity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GVNGIX42}},
  note         = {Machine review of arXiv:2505.19302}
}
abstract

NL2SQL (natural language to SQL) systems translate natural language into SQL queries, allowing users with no technical background to interact with databases and create tools like reports or visualizations. While recent advancements in large language models (LLMs) have significantly improved NL2SQL accuracy, schema ambiguity remains a major challenge in enterprise environments with complex schemas, where multiple tables and columns with semantically similar names often co-exist. To address schema ambiguity, we introduce ODIN, a NL2SQL recommendation engine. Instead of producing a single SQL query given a natural language question, ODIN generates a set of potential SQL queries by accounting for different interpretations of ambiguous schema components. ODIN dynamically adjusts the number of suggestions based on the level of ambiguity, and ODIN learns from user feedback to personalize future SQL query recommendations. Our evaluation shows that ODIN improves the likelihood of generating the correct SQL query by 1.5-2$\times$ compared to baselines.

Figures

Figures reproduced from arXiv: 2505.19302 by the authors.

Figure 1
Figure 1. Odin Overview schema elements). Through this schema linking, Odin can map nat￾ural language phrases like hometown and roll number from the user’s question to the origin and roll_num columns, respectively. These personalized hints are stored and later used by the Generator and Se￾lector to align future SQL queries with user preferences. In [PITH_FULL_IMAGE:figures/full_fig_p004_1.png] view at source ↗
Figure 2
Figure 2. Prompt template used to score SQL queries using LLMs [PITH_FULL_IMAGE:figures/full_fig_p008_2.png] view at source ↗
Figure 3
Figure 3. Prompt template used to provide hints to LLM [PITH_FULL_IMAGE:figures/full_fig_p009_3.png] view at source ↗
Figures from the paper (4 more)
Figure 4
Figure 4. Figure 4: Different types of ambiguities in the AmbiQT Benchmark. [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 5
Figure 5. Figure 5: Execution Match Accuracy (𝐴𝑣𝑔𝐴𝑐𝑐) versus the average number of results shown to the user (𝐴𝑣𝑔𝑅𝑒𝑠𝑢𝑙𝑡𝑆𝑖𝑧𝑒) across three different ambiguity types for various baselines. Odin can achieve up to twice the accuracy while presenting only half the number of SQL queries to the …
Figure 6
Figure 6. Figure 6: Odin’s Generator results alongside various SQL generation baselines for ambiguity types such as Join, Table Synonyms, and Column Synonyms. Each question has two SQL alternatives, and we assess whether either or both are in the generated result set (EitherInTopK and Bot…
Figure 7
Figure 7. Figure 7: Odin’s Execution Match Accuracy (𝐴𝑣𝑔𝐴𝑐𝑐) versus the average number of results shown to the user (𝐴𝑣𝑔𝑅𝑒𝑠𝑢𝑙𝑡𝑆𝑖𝑧𝑒) on across three ambiguity types, using various Selector scoring functions (LLM-based and SBERT-based). For each baseline, we vary the Selector’s alpha value …

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

38 extracted references · 20 canonical work pages

  1. [1]

    Anastasios N Angelopoulos, Stephen Bates, et al. 2023. Conformal prediction: A gentle introduction.Foundations and Trends®in Machine Learning16, 4 (2023), 494–591

    work page 2023

  2. [2]

    Adithya Bhaskar, Tushar Tomar, Ashutosh Sathe, and Sunita Sarawagi. 2023. Benchmarking and improving text-to-sql generation under ambiguity.arXiv Odin: A NL2SQL Recommender to Handle Schema Ambiguity preprint arXiv:2310.13659(2023)

    arXiv 2023

  3. [3]

    Han Fu, Chang Liu, Bin Wu, Feifei Li, Jian Tan, and Jianling Sun. 2023. Catsql: Towards real world natural language to sql applications.Proceedings of the VLDB Endowment16, 6 (2023), 1534–1547

    2023

  4. [4]

    Matthieu Futeral, Cordelia Schmid, Ivan Laptev, Benoît Sagot, and Rachel Bawden

  5. [5]

    João Gama, Indr˙e Žliobait˙e, Albert Bifet, Mykola Pechenizkiy, and Abdelhamid Bouchachia. 2014. A survey on concept drift adaptation.ACM computing surveys (CSUR)46, 4 (2014), 1–37

    work page 2014

  6. [6]

    Yujian Gan, Xinyun Chen, Jinxia Xie, Matthew Purver, John R Woodward, John Drake, and Qiaofu Zhang. 2021. Natural SQL: Making SQL easier to infer from natural language specifications.arXiv preprint arXiv:2109.05153(2021)

    arXiv 2021

  7. [7]

    Zihui Gu, Ju Fan, Nan Tang, Lei Cao, Bowen Jia, Sam Madden, and Xiaoyong Du

  8. [8]

    Zihui Gu, Ju Fan, Nan Tang, Lei Cao, Bowen Jia, Sam Madden, and Xiaoyong Du. 2023. Few-shot text-to-sql translation using structure and content prompt learning.Proceedings of the ACM on Management of Data1, 2 (2023), 1–28

    work page 2023

Show all 38 references
  1. [9]

    ACM Manag

    Few-shot Text-to-SQL Translation using Structure and Content Prompt Learning.Proc. ACM Manag. Data1, 2, Article 147 (jun 2023)

    2023

  2. [10]

    Bairu Hou, Yujian Liu, Kaizhi Qian, Jacob Andreas, Shiyu Chang, and Yang Zhang. 2024. Decomposing Uncertainty for Large Language Models through Input Clarification Ensembling. InForty-first International Conference on Machine Learning, ICML 2024, Vienna, Austria, July 21-27, 2024

    2024

  3. [11]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2019. The curious case of neural text degeneration.arXiv preprint arXiv:1904.09751(2019)

    2019 arXiv

  4. [12]

    Ralf Klinkenberg. 2004. Learning drifting concepts: Example selection vs. example weighting.Intelligent data analysis8, 3 (2004), 281–300

    2004

  5. [13]

    George Katsogiannis-Meimarakis and Georgia Koutrika. 2023. A survey on deep learning approaches for text-to-SQL.VLDB J.32, 4 (2023), 905–936

    2023

  6. [14]

    Dongryeol Lee, Segwang Kim, Minwoo Lee, Hwanhee Lee, Joonsuk Park, Sang-Woo Lee, and Kyomin Jung. 2023. Asking Clarification Questions to Handle Ambiguity in Open-Domain QA. InFindings of the Association for Computational Linguistics: EMNLP 2023. 11526–11544

    2023

  7. [15]

    Lorenz Kuhn, Yarin Gal, and Sebastian Farquhar. 2022. CLAM: Selective Clarification for Ambiguous Questions with Large Language Models.CoRR abs/2212.07769 (2022)

    2022 arXiv

  8. [16]

    Haoyang Li, Jing Zhang, Cuiping Li, and Hong Chen. 2023. Resdsql: Decoupling schema linking and skeleton parsing for text-to-sql. InProceedings of the AAAI Conference on Artificial Intelligence, Vol. 37. 13067–13075

    2023

  9. [17]

    Boyan Li, Yuyu Luo, Chengliang Chai, Guoliang Li, and Nan Tang. 2024. The Dawn of Natural Language to SQL: Are We Fully Ready?Proc. VLDB Endow.17, 11 (aug 2024), 3318–3331

    2024

  10. [18]

    Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. Codes: Towards building open-source language models for text-to-sql.Proceedings of the ACM on Management of Data2, 3 (2024), 1–28

    2024

  11. [19]

    Haoyang Li, Jing Zhang, Hanbing Liu, Ju Fan, Xiaokang Zhang, Jun Zhu, Renjie Wei, Hongyan Pan, Cuiping Li, and Hong Chen. 2024. CodeS: Towards Building Open-source Language Models for Text-to-SQL.Proc. ACM Manag. Data2, 3, Article 127 (may 2024)

    2024

  12. [20]

    Yihang Li, Shuichiro Shimizu, Weiqi Gu, Chenhui Chu, and Sadao Kurohashi

  13. [21]

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, Nan Huo, Xuanhe Zhou, Chenhao Ma, Guoliang Li, Kevin C. C. Chang, Fei Huang, Reynold Cheng, and Yongbin Li. 2023. Can LLM Already Serve as A Database Interface? A...

    2023 arXiv

  14. [22]

    Nils Reimers and Iryna Gurevych. 2019. Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks. InProceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing (EMNLP-IJCN...

    2019 doi

  15. [23]

    Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. 2021. PICARD: Parsing Incrementally for Constrained Auto-Regressive Decoding from Language Models. arXiv:2109.05093 [cs.CL]

    2021 arXiv

  16. [24]

    Abdul Quamar, Vasilis Efthymiou, Chuan Lei, and Fatma Özcan. 2022. Natural Language Interfaces to Data.Found. Trends Databases11, 4 (2022), 319–414

    2022

  17. [25]

    Noam Shazeer and Mitchell Stern. 2018. Adafactor: Adaptive learning rates with sublinear memory cost. InInternational Conference on Machine Learning. PMLR, 4596–4604

    2018

  18. [27]

    Glenn Shafer and Vladimir Vovk. 2008. A tutorial on conformal prediction. Journal of Machine Learning Research9, 3 (2008)

    2008

  19. [28]

    Janette Vazquez and Julio C Facelli. 2022. Conformal prediction in clinical medical sciences.Journal of Healthcare Informatics Research6, 3 (2022), 241–252

    2022

  20. [29]

    Vladimir Vovk, Alexander Gammerman, and Glenn Shafer. 2005. On-line compression modeling I: Conformal prediction.Algorithmic learning in a random world(2005), 189–221

    2005

  21. [30]

    Shayan Talaei, Mohammadreza Pourreza, Yu-Chen Chang, Azalia Mirhoseini, and Amin Saberi. 2024. CHESS: Contextual Harnessing for Efficient SQL Synthesis. arXiv:2405.16755 [cs.LG] https://arxiv.org/abs/2405.16755

    2024 arXiv

  22. [32]

    Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Qian-Wen Zhang, Zhao Yan, and Zhoujun Li. 2023. Mac-sql: Multi-agent collaboration for text-to-sql. arXiv preprint arXiv:2312.11242(2023)

    2023 arXiv

  23. [33]

    Bing Wang, Yan Gao, Zhoujun Li, and Jian-Guang Lou. 2023. Know What I don’t Know: Handling Ambiguous and Unknown Questions for Text-to-SQL. In Findings of the Association for Computational Linguistics: ACL 2023, Anna Rogers, Jordan Boyd-Graber, and Naoaki Okazaki (Eds.). 5701–5714

    2023

  24. [34]

    Xingjiao Wu, Luwei Xiao, Yixuan Sun, Junhang Zhang, Tianlong Ma, and Liang He. 2022. A survey of human-in-the-loop for machine learning.Future Generation Computer Systems135 (2022), 364–381

    2022

  25. [35]

    Miao Xiong, Zhiyuan Hu, Xinyang Lu, Yifei Li, Jie Fu, Junxian He, and Bryan Hooi

  26. [36]

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. 2021. RAT-SQL: Relation-Aware Schema Encoding and Linking for Text-to-SQL Parsers. arXiv:1911.04942 [cs.CL]

    2021 arXiv

  27. [39]

    Can llms express their uncertainty? an empirical evaluation of confidence elicitation in llms.arXiv preprint arXiv:2306.13063(2023)

    2023 arXiv

  28. [40]

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. 2019. Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task. arXi...

    2019 arXiv

  29. [2022]

    InProceedings of the Thirteenth Language Resources and Evaluation Conference

    VISA: An Ambiguous Subtitles Dataset for Visual Scene-aware Machine Translation. InProceedings of the Thirteenth Language Resources and Evaluation Conference. 6735–6743

  30. [2023]

    InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers)

    Tackling Ambiguity with Images: Improved Multimodal Machine Trans- lation and Contrastive Evaluation. InProceedings of the 61st Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers). 5394–5413

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.