Pith. sign in

REVIEW 4 cited by

A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

Not yet reviewed by Pith; the record is open.

Re-run · record.json Download PDF Read on arXiv ↗

This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.

SPECIMEN: schema-true, not a live event

T0 review · schema-true

One-sentence machine reading of the paper's core claim.

pith:XXXXXXXX · record.json · timestamp

arxiv 2303.13547 v1 pith:67M3X7J7 submitted 2023-03-12 cs.CL cs.AI

A comprehensive evaluation of ChatGPT's zero-shot Text-to-SQL capability

Aiwei Liu , Xuming Hu , Lijie Wen , Philip S. Yu This is my paper
classification cs.CL cs.AI
keywords chatgpttext-to-sqlmodelperformancezero-shotabilitiescomprehensiveconducted
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved
0 comments
read the original abstract

This paper presents the first comprehensive analysis of ChatGPT's Text-to-SQL ability. Given the recent emergence of large-scale conversational language model ChatGPT and its impressive capabilities in both conversational abilities and code generation, we sought to evaluate its Text-to-SQL performance. We conducted experiments on 12 benchmark datasets with different languages, settings, or scenarios, and the results demonstrate that ChatGPT has strong text-to-SQL abilities. Although there is still a gap from the current state-of-the-art (SOTA) model performance, considering that the experiment was conducted in a zero-shot scenario, ChatGPT's performance is still impressive. Notably, in the ADVETA (RPL) scenario, the zero-shot ChatGPT even outperforms the SOTA model that requires fine-tuning on the Spider dataset by 4.1\%, demonstrating its potential for use in practical applications. To support further research in related fields, we have made the data generated by ChatGPT publicly available at https://github.com/THU-BPM/chatgpt-sql.

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 4 Pith papers

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. PExA: Parallel Exploration Agent for Complex Text-to-SQL

    cs.AI 2026-04 unverdicted novelty 6.0

    PExA uses parallel exploration of atomic SQL test cases to ground final generation, achieving 70.2% execution accuracy on Spider 2.0.

  2. AV-SQL: Decomposing Complex Text-to-SQL Queries with Agentic Views

    cs.DB 2026-04 unverdicted novelty 6.0

    AV-SQL uses a pipeline of LLM agents to generate intermediate CTE views that decompose complex Text-to-SQL queries, reaching 70.38% execution accuracy on Spider 2.0.

  3. Knapsack Optimization-based Schema Linking for LLM-based Text-to-SQL Generation

    cs.CL 2025-02 unverdicted novelty 6.0

    KaSLA applies knapsack optimization hierarchically to schema linking for LLM text-to-SQL, claiming better results than large models and improved SQL generation on Spider and BIRD.

  4. Retrieve Only Relevant Tables Whether Few or Many: Adaptive Table Retrieval Method

    cs.IR 2026-04 unverdicted novelty 4.0

    An adaptive thresholding mechanism combined with sliding-window reranking retrieves a query-dependent number of tables from large corpora, improving retrieval and downstream text-to-SQL performance on Spider, BIRD, an...