pith. machine review for the scientific record. sign in

arxiv: 2604.16511 · v1 · submitted 2026-04-15 · 💻 cs.DB · cs.CL

Recognition: unknown

SQL Query Engine: A Self-Healing LLM Pipeline for Natural Language to PostgreSQL Translation

Muhammad Adeel Ijaz
Authors on Pith no claims yet

Pith reviewed 2026-05-10 12:36 UTC · model grok-4.3

classification 💻 cs.DB cs.CL
keywords natural language to SQLLLM pipelineself-healing systemsPostgreSQL querieserror diagnosisquery translation
0
0 comments X

The pith

A self-healing loop lets LLMs fix their own PostgreSQL query errors and raises accuracy by up to 9.3 points.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

This paper describes a service that converts everyday questions into PostgreSQL queries using two LLM stages. The generation stage creates SQL and parses it flexibly from any response format. When the query fails to run, a self-healing stage sends the precise database error back to the LLM so it can try again. Safeguards keep the best result seen so far and accept good queries early to avoid making them worse. Tests on benchmarks show consistent gains from this loop.

Core claim

The central claim is that an iterative self-healing mechanism, in which the language model receives full SQLSTATE error codes and diagnostic details from PostgreSQL to revise failed queries, improves translation accuracy. The approach includes a multi-strategy parser for extracting SQL and mechanisms to prevent performance regressions. On one benchmark it reaches 57.3 percent accuracy, and on another 49.0 percent.

What carries the argument

The self-healing loop that supplies PostgreSQL error messages to the LLM for diagnosis and correction, supported by early acceptance of successful queries and tracking of the best result across attempts.

If this is right

  • Accuracy improves by as much as 9.3 percentage points on the synthetic benchmark with no regressions for the top model.
  • The full system achieves 49.0 percent execution accuracy on the BIRD benchmark using a large model.
  • Schema information stays cached in Redis for session efficiency.
  • The service exposes a standard chat completions endpoint for easy integration.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Similar error-feedback loops could help LLMs handle other structured output tasks like API calls or code generation.
  • The read-only driver constraint ensures the system stays safe for production use even with LLM-generated queries.
  • Streaming progress via Redis Pub/Sub and SSE could support real-time interfaces in applications.

Load-bearing premise

The language model will correctly interpret and fix SQL problems when shown the full error codes and messages from PostgreSQL, while the parser always pulls out a usable query from whatever text the model returns.

What would settle it

A test run on additional natural language questions where the self-healing version shows lower accuracy or more failures than the single-pass version would disprove the benefit of the loop.

Figures

Figures reproduced from arXiv: 2604.16511 by Muhammad Adeel Ijaz.

Figure 1
Figure 1. Figure 1: Synthetic benchmark: execution accuracy (bars, left axis) and average query latency (lines, right axis) before [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Detailed architecture of SQL Query Engine. Stage 1 introspects the schema, generates SQL via the LLM, and [PITH_FULL_IMAGE:figures/full_fig_p004_2.png] view at source ↗
Figure 3
Figure 3. Figure 3: The self-healing evaluation loop. Key design choices are highlighted in [PITH_FULL_IMAGE:figures/full_fig_p006_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Decision flow within the self-healing loop. The two key safety mechanisms are: [PITH_FULL_IMAGE:figures/full_fig_p007_4.png] view at source ↗
Figure 5
Figure 5. Figure 5: Execution accuracy before (Config C) and after (Config A) the self-healing loop on both benchmarks. Scout [PITH_FULL_IMAGE:figures/full_fig_p012_5.png] view at source ↗
read the original abstract

We present SQL Query Engine, an open-source, self-hosted service that translates natural language questions into validated PostgreSQL queries through a two-stage LLM pipeline. The first stage performs automatic schema introspection and SQL generation; a multi-strategy response parser extracts SQL from any LLM output format (JSON, code blocks, or raw text) without requiring structured output APIs. The second stage executes the query against PostgreSQL and, upon failure or empty results, enters an iterative self-healing loop in which the LLM diagnoses the error using full SQLSTATE codes and PostgreSQL diagnostic messages. Two mechanisms prevent regressions: early-accept returns successful queries immediately without LLM re-evaluation, and best-result tracking preserves the best partial result across retries. Schema context is cached per session in Redis, progress events stream via Redis Pub/Sub and SSE, and an OpenAI-compatible /v1/chat/completions endpoint lets existing tools work without modification. All database connections are read-only at the driver level. We evaluate across five LLM backends on a synthetic benchmark (75 questions, three databases) where the self-healing loop yields up to +9.3pp accuracy gains with zero regressions on the best model (Llama 4 Scout 17B, 57.3%), and on BIRD (437 questions, 11 databases migrated from SQLite to PostgreSQL) where the full pipeline reaches 49.0% execution accuracy (GPT-OSS-120B, +4.6pp). Source code: https://github.com/codeadeel/sqlqueryengine.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

4 major / 2 minor

Summary. The manuscript describes SQL Query Engine, an open-source self-hosted service for translating natural language questions to PostgreSQL queries using a two-stage LLM pipeline. The first stage generates SQL with schema introspection and uses a multi-strategy parser to extract SQL from arbitrary LLM outputs. The second stage executes the query and employs an iterative self-healing loop that feeds SQLSTATE codes and diagnostic messages back to the LLM for error correction. The system includes features like Redis caching, streaming, OpenAI-compatible API, and read-only connections. Evaluations on a synthetic benchmark of 75 questions across three databases show accuracy gains of up to 9.3 percentage points with zero regressions for the best model, and on the BIRD dataset (437 questions, 11 databases) achieving 49.0% execution accuracy with a 4.6pp improvement.

Significance. If the reported accuracy improvements from the self-healing loop hold, this offers a practical, deployable system for NL-to-SQL translation that could aid database practitioners through its open-source release, read-only safety features, and compatibility with existing tools. The zero-regression claim on the synthetic benchmark is a notable strength if substantiated. However, the overall significance remains moderate given the empirical focus without detailed breakdowns or ablations.

major comments (4)
  1. Abstract: The claim of up to +9.3pp accuracy gains with zero regressions on the synthetic benchmark (Llama 4 Scout 17B reaching 57.3%) is load-bearing for the central contribution but lacks any per-query breakdown of loop invocations, fix success rates, or cases of empty-result handling, making it impossible to verify that the self-healing mechanism (rather than other pipeline elements) drives the gains.
  2. Abstract: The BIRD evaluation reports 49.0% execution accuracy (+4.6pp for GPT-OSS-120B) after migrating 11 databases from SQLite to PostgreSQL, yet provides no description of the migration process, schema mapping, or verification of semantic fidelity; this directly undermines attribution of the improvement to the self-healing loop.
  3. Abstract: No ablation or baseline comparison is presented that isolates the self-healing loop (e.g., full pipeline vs. single-pass generation), nor is there quantitative evidence on the multi-strategy parser's extraction success rate across LLM output formats; both are central assumptions required for the net-gain claims to hold.
  4. Abstract: The manuscript does not report how 'empty results' are distinguished from genuine zero-row answers or the maximum number of self-healing iterations, leaving open the possibility of non-termination or silent degradation that could contradict the zero-regressions result.
minor comments (2)
  1. The abstract should explicitly state the primary accuracy metric (execution accuracy) and how it is computed to avoid ambiguity with exact-match or other variants.
  2. Clarify whether the five LLM backends were evaluated with identical prompting and temperature settings, as this affects reproducibility of the reported gains.

Simulated Author's Rebuttal

4 responses · 0 unresolved

We thank the referee for the constructive and detailed feedback. We address each major comment below and have revised the manuscript accordingly to strengthen the presentation of our results, including additional breakdowns, clarifications, and analyses.

read point-by-point responses

  1. Referee: Abstract: The claim of up to +9.3pp accuracy gains with zero regressions on the synthetic benchmark (Llama 4 Scout 17B reaching 57.3%) is load-bearing for the central contribution but lacks any per-query breakdown of loop invocations, fix success rates, or cases of empty-result handling, making it impossible to verify that the self-healing mechanism (rather than other pipeline elements) drives the gains.

    Authors: We agree that a per-query breakdown is necessary to substantiate the contribution of the self-healing loop. In the revised manuscript we have added a supplementary table (Table S1) listing, for each of the 75 queries, the number of loop iterations, fix success/failure, empty-result handling, and final accuracy outcome. This table confirms that the observed gains are driven by successful error corrections rather than other pipeline stages, while preserving the zero-regression result. revision: yes

  2. Referee: Abstract: The BIRD evaluation reports 49.0% execution accuracy (+4.6pp for GPT-OSS-120B) after migrating 11 databases from SQLite to PostgreSQL, yet provides no description of the migration process, schema mapping, or verification of semantic fidelity; this directly undermines attribution of the improvement to the self-healing loop.

    Authors: We acknowledge the omission. The revised evaluation section now includes a dedicated subsection describing the migration: automated schema conversion via pgloader followed by manual review of data types, constraints, and indexes. Semantic fidelity was verified by running 50 equivalent queries on both SQLite and PostgreSQL versions and confirming identical result sets. These details allow readers to assess that the reported gains are attributable to the pipeline rather than migration artifacts. revision: yes

  3. Referee: Abstract: No ablation or baseline comparison is presented that isolates the self-healing loop (e.g., full pipeline vs. single-pass generation), nor is there quantitative evidence on the multi-strategy parser's extraction success rate across LLM output formats; both are central assumptions required for the net-gain claims to hold.

    Authors: We agree that explicit isolation of components strengthens the claims. The revised manuscript adds an ablation study (Section 4.3) comparing the full pipeline against a single-pass baseline across all five models, quantifying the incremental accuracy contribution of the self-healing loop. We also report that the multi-strategy parser achieved a 98.4% extraction success rate on 500 sampled LLM outputs spanning JSON, code blocks, and raw text, with fallback mechanisms for the remaining cases. revision: yes

  4. Referee: Abstract: The manuscript does not report how 'empty results' are distinguished from genuine zero-row answers or the maximum number of self-healing iterations, leaving open the possibility of non-termination or silent degradation that could contradict the zero-regressions result.

    Authors: We thank the referee for highlighting this gap. The revised methods section now specifies that empty results are identified by successful execution (no SQLSTATE error) returning zero rows, distinct from error states. The loop is capped at a maximum of three iterations to guarantee termination. Best-result tracking ensures that any iteration cannot degrade below the best prior successful or partial result, directly supporting the zero-regressions observation. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical system description with external benchmarks

full rationale

The paper presents an engineering system for NL-to-PostgreSQL translation using a two-stage LLM pipeline with self-healing error correction and a multi-strategy parser. All reported results (+9.3pp on synthetic, +4.6pp on BIRD) are direct empirical measurements on fixed external benchmarks (75 synthetic questions, 437 BIRD questions after SQLite-to-PostgreSQL migration). No equations, derivations, fitted parameters, or self-citations appear in the provided text. The accuracy claims do not reduce to any internal definition or prior author result by construction; they are produced by running the described pipeline on held-out data. The work is therefore self-contained against external benchmarks with no load-bearing circular steps.

Axiom & Free-Parameter Ledger

0 free parameters · 2 axioms · 0 invented entities

The system depends on standard assumptions about LLM code generation and debugging capabilities rather than new postulates; no free parameters or invented entities are introduced.

axioms (2)
  • domain assumption Large language models can generate valid SQL from natural language and schema context.
    Underpins the first-stage generation.
  • domain assumption LLMs can use database error messages and SQLSTATE codes to diagnose and repair faulty queries.
    Core premise of the self-healing loop.

pith-pipeline@v0.9.0 · 5571 in / 1481 out tokens · 42401 ms · 2026-05-10T12:36:44.801544+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

31 extracted references · 7 canonical work pages · 3 internal anchors

  1. [1]

    Seq2SQL: Generating Structured Queries from Natural Language using Reinforcement Learning

    Victor Zhong, Caiming Xiong, and Richard Socher. Seq2SQL: Generating structured queries from natural language using reinforcement learning. InarXiv preprint arXiv:1709.00103, 2017

    work page internal anchor Pith review arXiv 2017

  2. [2]

    Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task

    Tao Yu, Rui Zhang, Kai Yang, Michihiro Yasunaga, Dongxu Wang, Zifan Li, James Ma, Irene Li, Qingning Yao, Shanelle Roman, Zilin Zhang, and Dragomir Radev. Spider: A large-scale human-labeled dataset for complex and cross-domain semantic parsing and text-to-SQL task. InProceedings of the 2018 Conference on Empirical Methods in Natural Language Processing (...

    2018

  3. [3]

    S pider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to- SQL Task

    doi:10.18653/v1/D18-1425. 14 SQL Query Engine: Self-Healing LLM PipelineTECHNICALREPORT

    work page doi:10.18653/v1/d18-1425

  4. [4]

    GPT-4 Technical Report

    OpenAI. GPT-4 technical report.arXiv preprint arXiv:2303.08774, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  5. [5]

    LLaMA: Open and Efficient Foundation Language Models

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, et al. LLaMA: Open and efficient foundation language models.arXiv preprint arXiv:2302.13971, 2023

    work page internal anchor Pith review Pith/arXiv arXiv 2023

  6. [6]

    DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction

    Mohammadreza Pourreza and Davood Rafiei. DIN-SQL: Decomposed in-context learning of text-to-SQL with self-correction. InAdvances in Neural Information Processing Systems 36 (NeurIPS), 2023

    2023

  7. [7]

    Text-to-SQL empowered by large language models: A benchmark evaluation.Proceedings of the VLDB Endowment, 17(5): 1132–1145, 2024

    Dawei Gao, Haibin Wang, Yaliang Li, Xiuyu Sun, Yichen Qian, Bolin Ding, and Jingren Zhou. Text-to-SQL empowered by large language models: A benchmark evaluation.Proceedings of the VLDB Endowment, 17(5): 1132–1145, 2024

    2024

  8. [8]

    A survey on employing large language models for text-to-SQL tasks.ACM Computing Surveys, 2024

    Liang Shi, Zhengju Tang, Nan Zhang, Xiaotong Zhang, and Zhi Yang. A survey on employing large language models for text-to-SQL tasks.ACM Computing Surveys, 2024. doi:10.1145/3737873

    work page doi:10.1145/3737873 2024

  9. [9]

    Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to-SQL

    Jinyang Li, Binyuan Hui, Ge Qu, Jiaxi Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Ruiying Geng, Nan Huo, et al. Can LLM already serve as a database interface? a big bench for large-scale database grounded text-to-SQL. InAdvances in Neural Information Processing Systems 36 (NeurIPS), 2023

    2023

  10. [10]

    Open WebUI: Self-hosted AI interface

    Open WebUI Community. Open WebUI: Self-hosted AI interface. https://github.com/open-webui/ open-webui, 2024. Accessed: 2026-03-31

    2024

  11. [11]

    RAT-SQL: Relation- aware schema encoding and linking for text-to-SQL parsers

    Bailin Wang, Richard Shin, Xiaodong Liu, Oleksandr Polozov, and Matthew Richardson. RAT-SQL: Relation- aware schema encoding and linking for text-to-SQL parsers. InProceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL), pages 7567–7578, 2020

    2020

  12. [12]

    Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al

    Tom B. Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared Kaplan, Prafulla Dhariwal, Arvind Neelakantan, Pranav Shyam, Girish Sastry, Amanda Askell, et al. Language models are few-shot learners. Advances in Neural Information Processing Systems, 33:1877–1901, 2020

    1901

  13. [13]

    Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models.Advances in Neural Information Processing Systems, 35:24824–24837, 2022

    2022

  14. [14]

    C3: Zero -shot text-to-SQL with ChatGPT

    Xuemei Dong, Chao Zhang, Yuhang Ge, Yuren Mao, Yunjun Gao, Lu Chen, Jinshu Lin, and Dongfang Lou. C3: Zero-shot text-to-SQL with ChatGPT.arXiv preprint arXiv:2307.07306, 2023

    work page arXiv 2023

  15. [15]

    Mohammadreza Pourreza, Hailong Li, Ruoxi Sun, Yeounoh Chung, Shayan Talaei, Gaurav Tarlok Kakkar, Yu Gan, Amin Saberi, Fatma Ozcan, and Sercan O. Arik. CHASE-SQL: Multi-path reasoning and preference optimized candidate selection in text-to-SQL. InProceedings of the International Conference on Learning Representations (ICLR), 2025

    2025

  16. [16]

    Next- generation database interfaces: A survey of LLM-based text-to-SQL.IEEE Transactions on Knowledge and Data Engineering, 2025

    Zijin Hong, Zheng Yuan, Qinggang Zhang, Hao Chen, Junnan Dong, Feiran Huang, and Xiao Huang. Next- generation database interfaces: A survey of LLM-based text-to-SQL.IEEE Transactions on Knowledge and Data Engineering, 2025

    2025

  17. [17]

    MAC-SQL: A multi-agent collaborative framework for text-to-SQL

    Bing Wang, Changyu Ren, Jian Yang, Xinnian Liang, Jiaqi Bai, Linzheng Zhang, Zhao Yan, and Zhoujun Liu. MAC-SQL: A multi-agent collaborative framework for text-to-SQL. InProceedings of the 31st International Conference on Computational Linguistics (COLING), 2025

    2025

  18. [18]

    DTS-SQL: Decomposed text-to-SQL with small large language models

    Mohammadreza Pourreza and Davood Rafiei. DTS-SQL: Decomposed text-to-SQL with small large language models. InFindings of the Association for Computational Linguistics: EMNLP 2024, 2024

    2024

  19. [19]

    CodeS: Towards building open-source language models for text-to-SQL

    Haoyang Li, Binyuan Hui, Jian Qu, Bowen Yang, Binhua Li, Bowen Li, Bailin Wang, Bowen Qin, Rongyu Cao, Ruiying Geng, et al. CodeS: Towards building open-source language models for text-to-SQL. InProceedings of the ACM SIGMOD International Conference on Management of Data, 2024

    2024

  20. [20]

    Teaching large language models to self-debug

    Xinyun Chen, Maxwell Lin, Nathanael Schärli, and Denny Zhou. Teaching large language models to self-debug. InProceedings of the International Conference on Learning Representations (ICLR), 2024

    2024

  21. [21]

    ReFoRCE: A text-to-SQL agent with self-refinement, consensus enforcement, and column exploration.arXiv preprint arXiv:2502.00675, 2025

    Minghang Deng, Ashwin Ramachandran, Canwen Xu, et al. ReFoRCE: A text-to-SQL agent with self-refinement, consensus enforcement, and column exploration.arXiv preprint arXiv:2502.00675, 2025

    work page arXiv 2025

  22. [22]

    MAGIC: Generating self-correction guideline for in-context text-to-SQL.Proceedings of the AAAI Conference on Artificial Intelligence, 2025

    Arian Askari, Christian Poelitz, Xinye Tang, et al. MAGIC: Generating self-correction guideline for in-context text-to-SQL.Proceedings of the AAAI Conference on Artificial Intelligence, 2025

    2025

  23. [23]

    PICARD: Parsing incrementally for constrained auto-regressive decoding from language models

    Torsten Scholak, Nathan Schucher, and Dzmitry Bahdanau. PICARD: Parsing incrementally for constrained auto-regressive decoding from language models. InProceedings of the 2021 Conference on Empirical Methods in Natural Language Processing (EMNLP), pages 9895–9901, 2021

    2021

  24. [24]

    FastAPI: Modern, fast (high-performance) web framework for building APIs with Python

    Sebastián Ramírez. FastAPI: Modern, fast (high-performance) web framework for building APIs with Python. https://fastapi.tiangolo.com/, 2018. Accessed: 2026-03-31. 15 SQL Query Engine: Self-Healing LLM PipelineTECHNICALREPORT

    2018

  25. [25]

    LangChain: Building applications with LLMs through composability

    Harrison Chase. LangChain: Building applications with LLMs through composability. https://github.com/ langchain-ai/langchain, 2022. Accessed: 2026-03-31

    2022

  26. [26]

    Redis: An in-memory data structure store

    Salvatore Sanfilippo. Redis: An in-memory data structure store. https://redis.io/, 2009. Accessed: 2026-03-31

    2009

  27. [27]

    Psycopg 3: PostgreSQL database adapter for Python

    Federico Di Gregorio and Daniele Varrazzo. Psycopg 3: PostgreSQL database adapter for Python. https: //www.psycopg.org/psycopg3/, 2021. Accessed: 2026-03-31

    2021

  28. [28]

    Server-sent events

    Ian Hickson. Server-sent events. W3C Recommendation, https://www.w3.org/TR/eventsource/, 2015. Accessed: 2026-03-31

    2015

  29. [29]

    Faker: A Python package that generates fake data

    Faker Contributors. Faker: A Python package that generates fake data. https://faker.readthedocs.io/,

  30. [30]

    Accessed: 2026-03-31

    2026

  31. [31]

    Docker: Lightweight Linux containers for consistent development and deployment, 2014

    Dirk Merkel. Docker: Lightweight Linux containers for consistent development and deployment, 2014. 16

    2014