{"id":"185aa7fa-26d6-4326-8023-7f60b9cb4642","arxiv_id":"2608.09260","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"SafeQL, an in-DBMS search-based refinement framework, fixes LLM-generated SQL errors by guided AST-level repair, improving execution accuracy by up to 5.8% and cutting token usage up to 15x.","lead":"SafeQL turns the database from a passive error reporter into an active search guide that repairs only the broken parts of a failed SQL query. On the Bird and Spider benchmarks, it beats regeneration-based refineries in accuracy while using far fewer tokens.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Lemma 7 overclaims universal reachability: empty-result errors need not have any admissible refinement edge, so the 'always reaches an executable query' guarantee is not established and is contradicted by the paper's own unresolved-error statistics.","rationale":"The reader identified Lemma 7's empty-result and type-mismatch assumptions as the weakest point; I agree. This is the load-bearing concern because the Abstract and Section 3.2 use Lemma 7 to claim SafeQL 'converges to an executable query' and 'prevents repeated regeneration of errors.' If the lemma is false, the theoretical guarantee is void, and SafeQL's search is an effective heuristic whose success rate is an empirical quantity. The empirical results remain valuable: improved accuracy and token efficiency are measured directly and consistently across Tables 2-4. The paper's own unresolved-error statistics (Figure 9) actually support the weaker, heuristic interpretation, so the fix is to reframe the claim, not to abandon the system. No ad hominem: the issue is an overstrong proof sketch, not dishonesty. The concrete tests are minimal and would definitively settle whether the empty-result case can be repaired by a single admissible refinement; I expect they will fail, confirming the need for a weaker statement. Because the reader already recommended a conditional accept with a request to clarify or weaken the lemma, my read does not change the verdict.","tokens_in":23393,"tokens_out":5469,"duration_ms":52748,"concrete_test":"Build a minimal PostgreSQL instance with an empty table R(a int) and issue SELECT a FROM R WHERE a > 0. Run SafeQL with empty-result refinement enabled. If Lemma 7 holds, the search must find an executable value-refined query returning non-empty rows; since no value substitution can create rows in an empty table, any successful outcome would require a refinement outside R(empty result), demonstrating that the lemma's premise is not what makes the system work. As a second probe, use R(a int) with rows (1),(2) and query SELECT a FROM R WHERE a > 5 AND a < 1; verify whether any single Value Refinement produces a non-empty result. If neither probe yields a refinement edge, Lemma 7 is false as stated and the paper should report the unresolved fraction as expected behavior rather than as a violation of a guarantee.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central theoretical support for SafeQL is Lemma 7, which asserts that every execution error at a node of T_safe,q has an admissible refinement edge that resolves the error. The proof of Case 4 (empty result) does not actually prove this: it states that 'a Value Refinement typically provides a refinement edge' (Section 3.2), which is an empirical tendency, not a guarantee. The claim is false in general. For a database with an empty relation R, any query SELECT ... FROM R WHERE ... returns an empty result, and R(empty result) admits only Value Refinement; no replacement of a constant can produce tuples from an empty relation. Even for non-empty relations, a single Value Refinement cannot repair contradictory predicates: if R(a) contains {1,2}, the query SELECT a FROM R WHERE a > 5 AND a < 1 has no satisfying tuple, and replacing either constant 5 or 1 by any value in {1,2} leaves the predicate unsatisfiable. The paper's own Figure 9 shows 29% (prompt-based) and 16% (agent-based) of empty-result errors are not resolved, and Section 7.3.2 attributes the residue to queries that 'cannot enter the safe query space,' directly contradicting the universal claim of Lemma 7. Case 3.2 similarly assumes the schema has at least one attribute of each required type, an assumption not guaranteed by the database definition and not generally true in real schemas. Thus the reachability lemma is not a theorem; it is a heuristic claim, and any statement that SafeQL 'guarantees' convergence to an executable query must be weakened or removed.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"SafeQL proposes a search-based refinement paradigm for LLM-based Text-to-SQL. Rather than regenerating entire SQL queries after DBMS execution failure, SafeQL localizes the faulty AST component and applies minimal admissible refinement operations (relation, join, attribute, value, function) to build a 'safe refinement tree'; a best-first search over a semantic distance metric (weighted tree-edit and embedding distance) with type-based and top-K pruning selects the executable query closest to the original. The system is implemented as a PostgreSQL extension with an embedding cache, an HNSW vector index, and a hybrid regeneration fallback. Experiments on Bird (full dev and mini dev) and Spider, using DAIL-SQL and OpenSearch-SQL as testbeds and five regeneration baselines, report execution-accuracy gains up to +5.8 percentage points and 1.8–15.1x token reductions, plus ablations of pruning, caching/indexing, and the alpha and K parameters. The theoretical centerpiece is Lemma 7, which asserts that every execution error in the safe refinement tree has an admissible refinement edge that resolves it, guaranteeing convergence to an executable query.","tokens_in":23702,"tokens_out":14732,"duration_ms":127131,"significance":"If the empirical results hold, the paper makes a substantial systems contribution: the DBMS is turned into an active search guide rather than a passive error reporter, and the accuracy gains replicate across two benchmarks, two testbed systems (prompt- and agent-based), and five LLM families (Table 2), while the token-efficiency gains are large. Credit is due for the promised public artifact, the informative ablations (Figures 10-11), and the honest per-error-type analysis (Figure 9). The central accuracy improvement is not an artifact of the semantic distance metric: SafeQL improves over regeneration baselines while using the same LLM and the same initial query. However, the theoretical guarantee is overstated, and the stress-test concern lands: the empty-result case of Lemma 7 is asserted rather than proved, Case 3.2 rests on an unstated schema assumption, and the paper's own error statistics (Section 7.3.2, Figure 9) concede unresolved queries that contradict the 'converging to an executable query' framing in the abstract.","major_comments":[{"comment":"The empty-result case is not proved. The proof states that 'a Value Refinement typically provides a refinement edge,' which is an empirical tendency, not a guarantee; the proof also mentions 'relaxing a comparison,' yet the operation set of Definition 5 contains no operator-relaxation step. For a database containing an empty relation R, the query SELECT a FROM R WHERE a = 1 returns an empty result and every value substitution still returns an empty result, so the node has no admissible edge that resolves the error. Under the tree construction that node is then a leaf, which makes the 'Proof' attached to Definition 8 circular: the claim that every leaf satisfies [q] != epsilon(q) holds only if every error node has at least one child, exactly what Lemma 7 was supposed to establish. The manuscript itself concedes the gap at Section 7.3.2 ('queries that cannot enter the safe query space'), and Figure 9 shows 29% (prompt-based) and 16% (agent-based) of empty-result errors unresolved. I recommend restating Lemma 7 with explicit sufficient conditions or reclassifying it as a heuristic property, and aligning the abstract's convergence claim with the revised statement.","section":"Section 3.2, Lemma 7 (Case 4) and Definition 8"},{"comment":"The type-mismatch repair relies on an assumption that is neither stated in the lemma's hypotheses nor verified for the experimental databases: 'the schema includes at least one attribute of each required type.' Moreover, an Attribute Refinement only resolves the unknown-function error if the replacement attribute belongs to a relation in the current FROM scope (or is made accessible by an admissible step, but Join and Relation refinement are not admissible for unknown-function errors). Under the stated assumption, replacing the invalid argument with a type-compatible attribute outside the current scope would just convert the unknown-function error into an unknown-attribute error, and no admissible edge would restore the violated premise. The assumption must be made explicit and either proven for the benchmarks or the lemma restricted; as written, the universal guarantee for arbitrary D is not established.","section":"Section 3.2, Lemma 7 (Case 3.2)"},{"comment":"All headline numbers come from single runs with no error bars, seeds, or statistical tests, although the pipeline samples from stochastic LLMs. The margins against the strongest regeneration baselines on Bird full dev are small: Table 3 prompt-based shows SafeQL hybrid at 63.3 vs. RED-SQL at 62.9 and CHESS-SQL at 62.6, and search-only at 62.5 vs. OpenSearch-SQL at 62.0; agent-based shows 69.4 vs. 68.5 for CHESS-SQL. The abstract's claim that SafeQL 'significantly improves execution accuracy compared to regeneration-based methods' is therefore not yet supported for those pairwise comparisons. I ask for either an explicit statement of temperature-0, seed-stable decoding, or multiple runs with variance and a paired test (e.g., paired bootstrap over the test queries).","section":"Section 7, Tables 3 and 4"},{"comment":"The five regeneration baselines are re-implemented, but the manuscript does not report their prompts, few-shot counts, regeneration iteration limits, or decoding parameters, and it does not state the alpha, K, and hybrid-fallback settings used for the main results. The alpha/K study in Figure 11 is run on the Bird mini dev split, which is a subset of the same dev set used for the Table 3 headline numbers, so those choices are inherited without disclosure. Because the central efficiency claim (1.8-15.1x fewer tokens, '15x' in the abstract) is measured against these re-implementations, the convergence criteria for the baseline regeneration loops must be specified or the comparison is not reproducible. Please publish the baseline configurations and the exact settings used for Tables 3 and 4.","section":"Section 7.1.3 and Tables 3-4"}],"minor_comments":[{"comment":"The line immediately preceding the proof ('Formally: the error condition indicating a violation of execution premise D|-q=>epsilon(q)') is a dangling fragment, and no actual inference rules for the claimed 'lightweight operational semantics' are given; either provide the rules or label the argument as a proof sketch.","section":"Section 3.2"},{"comment":"Attaching a 'Proof' to a definition is nonstandard, and the proof depends on Lemma 7; I suggest turning the statement into a proposition whose validity is conditional on Lemma 7.","section":"Definition 8"},{"comment":"'the model offers no guaranty' should read 'no guarantee'.","section":"Section 2.2"},{"comment":"'These refinements differs from traditional rule-based generation' has a subject-verb agreement error; it should be 'differ.'","section":"Section 3.2"},{"comment":"The sub-caption '(c) Elapsed time while varying K' appears twice with mismatched panels; please renumber the sub-figures (a)-(d) and ensure each panel has a single caption.","section":"Figure 11"},{"comment":"OpenSearch-SQL serves both as a testbed system (hosting the agent-based no-refinement baseline and SafeQL) and as one of the five comparison methods; this dual role is legitimate but should be stated explicitly to avoid the impression of double counting.","section":"Sections 7.1.2-7.1.3"},{"comment":"DIN-SQL shows DeltaErr = +4.2% while DeltaEX = -0.3pp; since DeltaErr counts resolved execution errors, positive error reduction with negative accuracy change is possible, but the table and Section 7.2.1 should explain this so readers do not read DeltaErr as an accuracy improvement.","section":"Table 3"},{"comment":"The sentence 'the defined safe query space is not only theoretically sound but also practically effective' stands in direct tension with the same section's admission that some queries 'cannot enter the safe query space'; after revising Lemma 7, please align this sentence with the revised claims.","section":"Section 7.3.2"},{"comment":"The regeneration-fallback threshold (default 100 refinement steps) is not ablated; a sentence on its sensitivity, or a small sweep, would strengthen the hybrid-design evaluation.","section":"Section 6.3"}],"recommendation":"major_revision","confidential_remarks":"The empirical contribution is, in my view, publishable in this venue after the theoretical narrative is re-scoped and the experimental configuration details are disclosed; I do not see grounds for rejection. For the editor: the tuning of alpha and K on the Bird mini dev split, which is a subset of the dev set used for the headline Table 3 results, makes the full-dev Bird numbers mildly optimistic relative to a truly held-out test; the Spider results and the model-wise Table 2 provide partial mitigation. Also worth noting: the paper's own admitted unresolved-error statistics (Section 7.3.2) are the strongest internal evidence for the Lemma 7 overclaim, so this criticism does not depend on an external standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"SafeQL is worth reading. It does something the rest of the refinement literature doesn't: instead of throwing away a failing query and asking the LLM to regenerate, it localizes the error to a specific AST node and repairs that node with a guided best-first search over a 'safe query space.' That's a real change of paradigm, and the paper is honest about the debt to automatic program repair. The empirical work is above average for this area. The gains—up to +5.8% execution accuracy on Bird, +4.6% on Spider, with 15x token savings and often zero extra tokens in the search-only variant—hold across prompt-based and agent-based systems and across several model families. The ablations on pruning and system optimizations are well designed and show that the components matter. I'd cite the system design and the token-efficiency numbers. The main soft spot is Lemma 7. The paper claims a guarantee: every error at a node has an admissible refinement edge that resolves it. The proof doesn't establish that. Case 4, the empty-result case, says a Value Refinement 'typically provides' such an edge, which is not a proof. A genuinely empty relation or a contradictory predicate like WHERE a > 5 AND a < 1 can't be fixed by replacing a constant. The paper's own Figure 9 shows that some empty-result errors aren't resolved after refinement, and Section 7.3.2 explains the residue as queries that 'cannot enter the safe query space.' That contradicts the universal lemma. The fix is easy: state Lemma 7 as a heuristic or an empirical property, or restrict it to the cases where the guarantee actually holds. This doesn't sink the contribution—the system works on most queries—but the paper currently overclaims. Other issues are smaller. No error bars or multiple runs, which is common in this literature but still worth flagging. The baselines are re-implemented, and the prompt configurations for them aren't detailed enough to reproduce without help. α and K are tuned on the mini dev set and then evaluated on the full dev set; that's mild selection, but the accuracy gain doesn't depend on those parameters, so it's not load-bearing. The GitHub URL in the paper isn't verifiable from the text; given the claim of artifact availability, an editor should ask for a working link or a hash. Who is this for? Anyone building Text-to-SQL systems, and people interested in constrained generation or grounding LLM output in a DBMS. It's a subfield contribution, not a breakthrough, but it's a useful one. I'd send it to reviewers. With a revised Lemma 7, variance reporting, and a reproducible artifact, I'd be happy to accept.","headline":"A genuinely new refinement idea for Text-to-SQL with real empirical gains; Lemma 7 overclaims a guarantee and needs to be pulled back, but the paper deserves a proper review.","tokens_in":24248,"tokens_out":3613,"would_cite":true,"duration_ms":30274,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SafeQL reframes LLM text-to-SQL repair as a guided search inside the database, fixing only the broken parts instead of regenerating whole queries.","keywords":["text-to-SQL","query refinement","large language models","safe query space","best-first search","semantic distance","database management systems","execution accuracy"],"falsifier":"Run SafeQL in search-only mode on a hand-built database where a query fails with an unknown-function type mismatch but no attribute of the required type exists in the schema, or where an empty-result error stems from a predicate no value substitution can satisfy; if the search terminates without an executable query, the reachability guarantee of Lemma 7 is refuted.","tokens_in":23165,"feed_emoji":"🗄️","tokens_out":6534,"duration_ms":61178,"temperature":0.7,"pith_summary":"SafeQL claims that the standard regeneration-based approach to fixing LLM-generated SQL—executing the query, reading the error message, and asking the LLM to write the whole query again—can be replaced by a guided search over a safe query space. The paper redefines the DBMS as an active participant: its parser, binder, and type analyzer locate the faulty component, and admissible refinement operations repair only that component while preserving the rest of the query. A best-first search, ranked by a semantic distance that blends structural and embedding similarity, selects the executable candidate closest to the original intent. On the Bird and Spider benchmarks, the authors report execution-accuracy gains up to +5.8 percentage points and up to a 15x reduction in LLM token consumption compared with regeneration-based baselines. The broader claim is that localized, execution-guided repair is both more reliable and cheaper than repeated regeneration.","feed_headline":"Search-based SQL repair beats regeneration, 15x fewer tokens","feed_subtitle":"By guiding the DBMS to patch only the broken query parts, SafeQL raises execution accuracy up to 5.8% on Bird and Spider.","key_machinery":"The machinery is the safe query space $\\mathcal{Q}_{\\text{safe}}$ together with its safe refinement tree $\\mathcal{T}_{\\text{safe},q}$. Each node is a candidate SQL query, and each edge is an atomic refinement step permitted for the observed error type: relation, join, attribute, value, or function refinement. Best-first search over this tree, ordered by semantic distance $\\delta$, returns the first executable leaf as the nearest safe query. Type-based pruning removes substitutions that violate the database typing environment $\\Gamma$, while top-$K$ pruning keeps only the most embedding-similar candidates per category, and in-database caching plus a vector index make the similarity computations fast. The paper's guarantee is Lemma 7, which asserts that every error in the tree has an admissible refinement that resolves it.","core_discovery":"The central discovery is that Text-to-SQL refinement can be formulated as a shortest-path-style search into an executable subspace, with each hop being a single structure-preserving edit triggered by the DBMS's own error diagnosis. For a failing query, SafeQL builds a safe refinement tree whose edges are admissible operations selected by error type: relation refinement, join refinement, attribute refinement, value refinement, or function refinement. The paper argues in Lemma 7 that every execution error has at least one such edge that resolves it, so the search eventually reaches a leaf whose execution succeeds, and the set of such leaves is the safe query space $\\mathcal{Q}_{\\text{safe}}$. Because this space can grow exponentially, SafeQL uses type-based pruning and top-$K$ embedding pruning, and it ranks candidates by semantic distance $\\delta = \\alpha d_{\\text{struct}} + (1-\\alpha) d_{\\text{embed}}$ to recover the most faithful executable query. Implemented inside a DBMS, it refines the query AST rather than raw text, with a hybrid fallback that regenerates only when localized search cannot recover.","pith_inferences":["If the reachability result is treated as an empirical heuristic rather than a theorem, the same search structure could be extended to other DBMS-level failures, such as constraint violations, permission errors, or result-shape mismatches, whenever an admissible repair operation can be defined.","The semantic-distance objective is a proxy for user intent; a natural extension would be to re-rank safe candidates by executing them and comparing results against the question or a small validation set, turning the refinement search into a test-guided loop.","The same pattern—use a symbolic checker to localize an error, propose minimal edits, then search over edit sequences—applies beyond SQL to code repair, API misuse, and data-pipeline generation, where regeneration is currently the default repair strategy.","The weakest part of Lemma 7 is the empty-result case, where the paper says a Value Refinement 'typically' supplies a repair; a contradictory predicate that no value substitution can satisfy would be a direct counterexample to the stated guarantee."],"forward_implications":["Refinement cost shifts from expensive LLM regeneration to cheaper in-database search: in the reported Bird and Spider experiments, SafeQL in search-only mode resolves a large share of errors with zero additional LLM tokens.","Weaker and smaller models benefit most: the reported gains of +5 to +12 percentage points for open-source models such as Qwen and Llama suggest that structural, execution-guided repair narrows the gap with larger commercial models.","The DBMS parser, binder, and type analyzer become first-class components of the refinement loop, which implies that future text-to-SQL systems should expose structured error locations rather than plain error strings.","Because only the erroneous AST node changes, valid fragments of the original query are preserved, reducing the chance of reintroducing previously fixed errors compared with full regeneration.","The hybrid fallback bounds worst-case behavior: if search does not converge within a step threshold, regeneration reinitializes the search, keeping the system robust to fundamentally misgenerated queries."],"supporting_citations":[{"why":"Supplies the Bird benchmark, the primary dataset on which SafeQL's accuracy and efficiency are measured.","marker":"[18]"},{"why":"Supplies the Spider benchmark, the second dataset used to test SafeQL's generalizability across databases.","marker":"[57]"},{"why":"DAIL-SQL is the prompt-based text-to-SQL system into which SafeQL is integrated and against which refinement methods are compared.","marker":"[7]"},{"why":"OpenSearch-SQL is the agent-based text-to-SQL system used as an integration target and as the strongest regeneration-based comparison.","marker":"[52]"},{"why":"DIN-SQL represents a regeneration-based baseline that refines without execution feedback, contrasting with SafeQL's execution-guided search.","marker":"[31]"},{"why":"RED-SQL provides a regeneration-based baseline that uses constraint-violation feedback, showing the accuracy and token-cost trade-off SafeQL aims to improve.","marker":"[34]"},{"why":"CHESS-SQL is a regeneration-based baseline using few-shot refinement examples, included in the main accuracy and efficiency comparisons.","marker":"[42]"},{"why":"MAC-SQL is a regeneration-based baseline using raw DBMS error messages, included in the main accuracy and efficiency comparisons.","marker":"[48]"},{"why":"Supplies the neural embedding model used to compute the embedding-based component of the semantic distance.","marker":"[33]"},{"why":"Supplies the normalized tree-edit-distance metric used for the structural component of the semantic distance.","marker":"[60]"}],"fun_headline_variants":["SafeQL: Search-based SQL repair, 15x fewer tokens","DBMS-guided search fixes SQL errors, boosts accuracy","SafeQL: Incremental query repair, not regeneration","Search-based refinement lifts Text-to-SQL reliability","SafeQL: Active DBMS guides SQL fixes efficiently"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is Lemma 7's reachability: every error type must have an admissible refinement that truly resolves it, which presumes the schema contains at least one attribute of the type each failing function argument needs, and presumes every empty-result failure can be fixed by substituting some value in the database.","fun_headline_variants_meta":{"raw":{"variants":["SafeQL: Search-based SQL repair, 15x fewer tokens","DBMS-guided search fixes SQL errors, boosts accuracy","SafeQL: Incremental query repair, not regeneration","Search-based refinement lifts Text-to-SQL reliability","SafeQL: Active DBMS guides SQL fixes efficiently"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000284,"raw_usage":{"total_tokens":1684,"prompt_tokens":964,"completion_tokens":720,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":652}},"tokens_in":580,"tokens_out":720,"duration_ms":6444,"temperature":1.0,"reasoning_tokens":652,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:35:37.440463+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run SafeQL in search-only mode on a hand-built database where a query fails with an unknown-function type mismatch but no attribute of the required type exists in the schema, or where an empty-result error stems from a predicate no value substitution can satisfy; if the search terminates without an executable query, the reachability guarantee of Lemma 7 is refuted.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Bird benchmark, the primary dataset on which SafeQL's accuracy and efficiency are measured."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the Spider benchmark, the second dataset used to test SafeQL's generalizability across databases."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"OpenSearch-SQL is the agent-based text-to-SQL system used as an integration target and as the strongest regeneration-based comparison."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"DIN-SQL represents a regeneration-based baseline that refines without execution feedback, contrasting with SafeQL's execution-guided search."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"MAC-SQL is a regeneration-based baseline using raw DBMS error messages, included in the main accuracy and efficiency comparisons."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the normalized tree-edit-distance metric used for the structural component of the semantic distance."}],"review_version":1}