{"id":"e5cf6bc0-b4e8-476f-b4e5-661b723fd97a","arxiv_id":"2607.22572","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Live schema grounding against Oracle's catalog plus alias/column validation lifts execution-grounded NL2SQL correctness from 2.2% to 62.6% on 500 TPC-H questions.","lead":"SAL is a lightweight middleware that fetches live Oracle schema metadata, injects it into an LLM prompt, and validates every column reference, improving correct SQL answers on 500 TPC-H questions from 2.2% to 62.6%. It matches a hand-written static schema hint without manual curation, but its keyword and join-rule maps were tuned on the same benchmark.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'no manual schema curation' claim is contradicted by the paper's own design: K_t, J, P and thresholds are hand-built with TPC-H knowledge (Secs. 4.2.1–4.2.3, T-I4), making 62.6% EGT a fitted score until shown otherwise.","rationale":"The reader's weakest assumption and my concern converge on the same load-bearing point: the detection components that make SAL work are hand-constructed with knowledge of the single benchmark, and the evaluation questions come from the same team. The paper is transparent about this in T-I4 and the limitations section, but the abstract's 'no manual schema curation' is the headline claim, and it is not supported by the methods as written. This is not an internal inconsistency in the reported measurements; the ablation is plausible and the execution-failure reduction is large. The problem is that the measurements do not establish the transferable, curation-free property that the paper claims. A held-out-schema replication with independently constructed keyword maps and independently authored questions would settle it. Since the reader already reached CONDITIONAL for exactly this reason, my analysis does not change the verdict; it sharpens the specific test needed.","tokens_in":21432,"tokens_out":4447,"duration_ms":51705,"concrete_test":"Run a blinded replication on a held-out Oracle schema (e.g., TPC-DS SF=1, 24 tables) with K_t, J, P, and the three hyperparameters constructed by an independent team using only the live catalog, Oracle constraint metadata, and a written spec that forbids looking at the evaluation questions. Author roughly 500 questions from that schema's business semantics and execute the same C2 vs C3b protocol. If SAL v2 fails to reach statistical parity with the static full-schema hint, or if any manual adjustment of K/J/P/thresholds is needed to do so, the 'no manual schema curation' claim collapses. A cheaper first check on the current benchmark: replace K_t with column-name-derived tokens and J with ALL_CONSTRAINTS-derived foreign-key edges; if EGT drops materially below 62.6%, the manual maps are already load-bearing in the reported result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"For the central claim to hold, SAL's table selection must be both automatic and transferable. The paper's own description shows it is neither. Section 4.2.1 defines per-table keyword vocabularies K_t ('lineitem', 'revenue', 'shipdate', ...) that are curated domain terms, not derivable from USER_TAB_COLUMNS. Section 4.2.2 builds the JOIN adjacency J 'constructed once from the known TPC-H foreign-key graph'. Section 4.2.3's complexity phrase set P and the thresholds (|T_direct| < 3 and cplx(q) >= 2) were 'chosen to trade off prompt length against missed JOIN context on our 500-question benchmark.' Section 6.2 says the 500 questions were authored by the same team that designed these maps. Section 10.1 T-I4 labels the resulting dataset leakage as the 'primary internal validity threat.' Thus the 62.6% EGT is a fitted score for TPC-H, not evidence for the abstract's 'no manual schema curation' claim. The claim would be true only if K_t, J, and P could be generated automatically or transfer unchanged; nothing in the paper shows either. The large C1→C3b gain and Hidx recovery are real but do not isolate the transferable parts of the pipeline.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents Schema-Aware Localisation (SAL), a zero-retraining middleware layer for Oracle NL2SQL. SAL loads live schema metadata from USER_TAB_COLUMNS, selects a question-relevant subset of tables via scored keyword matching, JOIN-chain expansion, and a complexity gate, injects that schema context into the LLM prompt, and then validates the generated SQL with a Hallucination Index (Hidx) that checks alias.column references against the live schema, applies deterministic prefix corrections, and triggers structured LLM retries. Evaluation on 500 TPC-H questions executed against a live Oracle Autonomous Database 23c instance with GPT-4o-mini reports 2.2% EGT with no hint, 62.0% with a hand-written static hint, 58.4% for SAL v1, and 62.6% for SAL v2, with execution failures reduced from 97.6% to 2.6%. The paper claims that SAL achieves this 'with no manual schema curation.'","tokens_in":21860,"tokens_out":5292,"duration_ms":53645,"significance":"If the transferability concern is resolved, the result is significant for enterprise NL2SQL: it demonstrates that live catalog grounding plus deterministic identifier validation can recover most of the benefit of a hand-written static schema hint, at lower prompt cost, without retraining, and with a large reduction in Oracle execution errors. The live-database evaluation, the four-condition ablation, the explicit threat-to-validity analysis, and the plan to release implementation and harness as open source are genuine strengths. However, the central 'no manual schema curation' claim is not established by the evidence in this version. The paper's own design shows that the keyword vocabularies K_t, the JOIN-chain adjacency map J, the complexity phrase set P, and the gate thresholds were hand-built with TPC-H knowledge and tuned on the same 500-question benchmark used for evaluation. The 62.6% EGT figure is therefore best read as a fitted benchmark score until replication on a held-out schema with independently constructed detection components is provided.","major_comments":[{"comment":"The abstract's claim that SAL works 'with no manual schema curation' is contradicted by the paper's own system description. Section 4.2.1 defines per-table keyword vocabularies K_t that are 'curated' domain terms; Section 4.2.2 builds the JOIN adjacency map J 'from the known TPC-H foreign-key graph'; Section 4.2.3 says the complexity phrase set P and thresholds (|T_direct|<3, cplx(q)>=2) were 'chosen to trade off prompt length against missed JOIN context on our 500-question benchmark.' Section 6.2 states the 500 questions were authored by the same team that designed these maps. Section 10.1 (T-I4) labels this dataset leakage the 'primary internal validity threat,' and Section 12(1) concedes the maps 'were designed with knowledge of TPC-H table names.' Consequently, the 62.6% EGT is a fitted score for TPC-H, not evidence for the headline general claim. The revision should either remove th","section":"Abstract; Sections 4.2.1–4.2.3, 6.2; T-I4; Section 12(1)"},{"comment":"The abstract and Definition 5 state that Hidx 'validates every alias.column reference,' but the implementation does not. The alias-map parser is the regex (?:FROM|JOIN)\\s+([A-Z_]+)\\s+([A-Z_]+), and the limitation paragraph in Section 4.3.1 says subqueries, CTEs, quoted identifiers, and aliases introduced by subqueries/CTEs are 'out of scope and are skipped in subsequent validation.' Thus the numerator and denominator in Eq. (2) cover only references bound by simple FROM/JOIN clauses. Given that the complex tier includes correlated subqueries and CTEs, Hidx can silently skip exactly the references where hallucinated columns often occur. The claim 'validates every alias.column reference' should be scoped, or the parser/validator should be extended to handle nested query aliases.","section":"Section 4.3.1; Definition 5"},{"comment":"The paper presents Online Execution-Grounded Verification (OEGV) as a general post-execution verification component, but the deployment-mode description is explicitly TPC-H-specific. Section 5.4 says expected row-count ranges are 'computed from lightweight schema statistics ... for the specific benchmark schema (TPC-H SF=1),' and the sample feedback in Section 5.5 hard-codes the dataset window 1992-01-01 to 1998-12-31 and the anonymised naming convention ('Region#2', not 'ASIA'). No deployment-mode precision or recall is measured. In the evaluation, OEGV relies on cached reference results and Section 5.7 reports that its semantic retry recovered zero additional queries. The manuscript should not imply that OEGV is a validated deployment guard; it is currently either a benchmark-only reference-comparison mechanism or an unvalidated heuristic.","section":"Sections 5.4–5.5"}],"minor_comments":[{"comment":"The missing C2 tier-level logs are acknowledged, but the table's 'C2: Static hint' row could be marked 'not retained' more prominently to avoid readers inferring tier-level parity. The text already handles this correctly.","section":"Section 7.4, Table 9"},{"comment":"The 98.7% schema-hallucination suppression rate is extrapolated from a 50-query manual sample of the 488 C1 failures to an assumed 469 schema-class failures. This should be labeled an estimate with a confidence interval or a conservative range, not a precise rate.","section":"Section 8.5"},{"comment":"The mitigation for the harness-as-confounder threat says 'running a baseline condition whose EGT is consistent with prior work.' Which prior work? The no-hint baseline of 2.2% is hard to compare because there is no prior Oracle ADB benchmark. Please specify the comparison or remove the phrase.","section":"Section 10.1, T-I2"},{"comment":"The JOIN-chain map J is said to be 'equivalently derived from Oracle constraint metadata,' but the implementation uses the hand-built TPC-H adjacency graph. If constraint metadata derivation is feasible, the authors should implement it or at least describe how it would eliminate the manual step; as written, this sentence undercuts the 'no manual curation' claim.","section":"Section 4.2.2"},{"comment":"The sentence 'academic systems address (iv) but not (i)–(iii)' is ambiguous because the related-work systems in Table 1 are academic and do address the 'no retrain' dimension. Rephrase to clarify that SAL is the first to combine all four dimensions.","section":"Section 2.6 / Table 1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: This is an honest, well-structured engineering paper with a real live-Oracle evaluation and a valuable ablation. The core 2.2%→62.6% effect is large and internally coherent. The problem is not the ablation but the framing: the 'no manual schema curation' claim is explicitly contradicted by the authors' own T-I4 and Section 12(1). This is fixable within the paper's scope by reframing the claim and/or adding a held-out schema experiment, so I do not recommend rejection. The revision should also tighten the Hidx coverage claim and clarify that OEGV's deployment-mode heuristics are not validated. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, read this one for the 2.2-to-62.6 effect, not for the 'no manual schema curation' claim. The core result is real: on 500 TPC-H questions against a live Oracle ADB, GPT-4o-mini with no schema hint gets 2.2% execution-grounded truth; with SAL's live catalog grounding plus Hidx validation it gets 62.6%, and execution failures drop from 97.6% to 2.6%. That's a large and internally coherent effect, and the Oracle-specific ablation is genuinely new — I know of no other published evaluation that measures live-catalog grounding on Oracle ADB.\n\nThe paper is also admirably transparent. It names T-I4 as the primary internal validity threat: the keyword maps K_t, the JOIN adjacency graph J, the complexity phrase set P, and the gate thresholds were all built by the same team with knowledge of TPC-H, and the 500 questions were authored by that same team. It also tells you C2's tier-level logs were lost, OEGV recovered 0 queries in this evaluation, and each condition ran once. That honesty deserves credit.\n\nBut the abstract says 'SAL, with no manual schema curation, achieves 62.6% EGT.' That sentence is not supported by the paper's own methods. K_t is a curated domain vocabulary. J is constructed from the known TPC-H foreign-key graph. The complexity gate thresholds were 'chosen to trade off prompt length against missed JOIN context on our 500-question benchmark.' The 62.6% is thus a fitted score for one schema, not evidence that the detection components transfer. If a new enterprise schema needs fresh manual curation of these maps, the headline collapses. To support it, the authors would need either fully automatic derivation of K_t and J from catalog metadata, or replication on a held-out schema with independently constructed maps. Neither is here.\n\nAlso missing: code and data. The paper says the repository link is omitted for double-blind review, which is normal for a preprint, but the result cannot be checked. With a single run, no significance testing on the key comparison, and closed artifacts, 62.6% should be read as a well-documented benchmark score, not a deployed-system guarantee.\n\nWho should read this: anyone working on NL2SQL for enterprise databases, especially Oracle. The architecture is sensible and the measurement approach is sound even if the generalization claim overreaches. It deserves a serious referee; a good review would ask for artifact release and a held-out schema replication, and would ask them to rewrite the abstract to match the body. As it stands, I'd cite it only as 'reported result awaiting independent validation.'","headline":"The paper's real finding — schema grounding moves Oracle NL2SQL from 2.2% to 62.6% EGT — is plausible and well-measured, but the headline 'no manual schema curation' is contradicted by the paper's own methods.","tokens_in":22303,"tokens_out":2813,"would_cite":false,"duration_ms":29242,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"LLM-generated SQL fails on real Oracle databases mostly because the model never sees the real column names; a live-catalog grounding layer recovers most of that gap without retraining.","keywords":["NL2SQL","schema grounding","hallucination validation","execution-grounded truth","Oracle SQL","schema injection","TPC-H","retrieval-augmented generation"],"falsifier":"Run the same pipeline on a fresh Oracle schema outside the benchmark domain, building keyword maps and the JOIN adjacency without ever reading the evaluation questions; if execution-grounded truth falls well below the hand-written static hint, the generalisability claim fails. A cheaper internal check is to rerun SAL v2 without the Hallucination Index—the paper's own decomposition predicts EGT drops to 55.2%.","tokens_in":21336,"feed_emoji":"🗄️","tokens_out":4823,"duration_ms":46957,"temperature":0.7,"pith_summary":"This paper argues that most failures of LLM-generated SQL on real Oracle databases are schema hallucinations—references to tables and columns that do not exist—and that a lightweight runtime layer can fix most of them without retraining. The system reads the live column catalog, picks a question-specific subset of tables, injects the exact column names into the prompt, and then validates every alias.column reference in the generated SQL, auto-correcting prefix omissions and retrying with itemised corrections. On 500 questions over a standard supply-chain schema on a live Oracle database, this lifts execution-grounded truth (executes and matches the reference result) from 2.2% with no schema context to 62.6%, equal to a hand-written static schema hint, while cutting execution failures from 97.6% to 2.6%. The transferable value is the claim that grounding plus validation, not model retraining, is the main unlock for enterprise text-to-SQL.","feed_headline":"Live schema lookup lifts NL2SQL truth from 2.2% to 62.6%","feed_subtitle":"Injecting real Oracle column names plus a validator eliminates 98.7% of schema hallucinations.","key_machinery":"Three components carry the argument: (1) a live schema cache populated from Oracle's USER_TAB_COLUMNS catalog, so the prompt contains exact, current column names; (2) complexity-gated table selection that scores tables with keyword vocabularies, expands matches along a JOIN-chain adjacency map, and falls back to the full schema when multi-table complexity signals fire; and (3) the Hallucination Index validator, which statically checks alias.column references against the cache, auto-corrects prefix omissions, and drives at most two feedback retries. The combination converts most of the schema-hallucination failure class into a deterministic validation problem.","core_discovery":"The central claim is that schema grounding can be automated to match a hand-curated hint. SAL's table detector scores each table by keyword hits, expands along join edges, and falls back to the full schema for multi-table questions, so the model always sees real column names. The Hallucination Index then parses the generated SQL, maps aliases to tables, and checks each alias.column against the live catalog; predictable prefix errors like O.ORDERDATE for O.O_ORDERDATE are rewritten without an LLM call, and anything else gets a structured retry with itemised corrections. The reported result is 62.6% execution-grounded truth (96% simple, 95% medium, 40.7% complex), with 37 of the 313 correct an","pith_inferences":["The honest generalisation test is on a schema the designers never saw: if the keyword vocabularies, JOIN adjacency map, and complexity phrase set need re-authoring for each enterprise schema, the 'no manual schema curation' claim would not transfer, even if the validator itself does.","A natural extension is to generate candidate alias-prefix corrections algorithmically from the catalog itself—e.g., detect common table-initial column prefixes—instead of relying on a hand-tuned prefix pattern.","Swapping in a stronger base model should shift the complex-tier 40.7% ceiling upward; comparing that shift would separate grounding effects from reasoning effects.","Using Hidx corrections as few-shot demonstrations or as a training signal for a smaller local model could reduce the remaining LLM retries and make the pipeline cheaper."],"forward_implications":["Schema grounding, not model reasoning, is the binding constraint in this Oracle setting: removing it drops execution-grounded truth from 62.6% to 2.2%.","A dynamic, question-adaptive hint can match a full static schema hint at lower mean token cost (roughly 208 fewer tokens per query) while staying current with schema changes.","Schema hallucinations are not random: the paper's sampled no-hint failures are mostly phantom identifiers, and a token-precise validator can catch most of them before execution.","The residual ceiling is a reasoning boundary: 174 of 187 failures execute but return semantically wrong results, so further gains need better query reasoning, not more schema context."],"fun_headline_variants":["Automated schema grounding lifts NL2SQL truth from 2.2% to 62.6%","No hand-written hints: schema grounding hits 62.6% execution truth","Live Oracle catalog stops NL2SQL hallucinations, lifting truth to 62.6%","Schema grounding automates what hand-tuned hints do: 62.6% execution truth"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that the keyword vocabularies, JOIN-chain adjacency map, complexity phrase set, and gate thresholds—all constructed with knowledge of the benchmark schema—transfer to new Oracle schemas without manual re-curation; if they do not, the 'no manual schema curation' claim collapses and the 62.6% figure is a fitted benchmark score.","fun_headline_variants_meta":{"raw":{"variants":["Automated schema grounding lifts NL2SQL truth from 2.2% to 62.6%","No hand-written hints: schema grounding hits 62.6% execution truth","Live Oracle catalog stops NL2SQL hallucinations, lifting truth to 62.6%","Schema grounding automates what hand-tuned hints do: 62.6% execution truth"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000584,"raw_usage":{"total_tokens":2641,"prompt_tokens":859,"completion_tokens":1782,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":603,"completion_tokens_details":{"reasoning_tokens":1697}},"tokens_in":603,"tokens_out":1782,"duration_ms":13946,"temperature":1.0,"reasoning_tokens":1697,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-02T12:22:56.531771+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same pipeline on a fresh Oracle schema outside the benchmark domain, building keyword maps and the JOIN adjacency without ever reading the evaluation questions; if execution-grounded truth falls well below the hand-written static hint, the generalisability claim fails. A cheaper internal check is to rerun SAL v2 without the Hallucination Index—the paper's own decomposition predicts EGT drops to 55.2%.","supporting_citations":[],"review_version":1}