{"id":"ca179448-b30a-42fd-b36c-1ea6028bc95e","arxiv_id":"2607.06799","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":5,"one_line_summary":"On hard multi-table text-to-SQL, verification-based LLM judges beat self-consistency and log-probability for predicting execution correctness, and fine-tuned verifiers fail to transfer across schemas.","lead":"Black-box uncertainty signals for text-to-SQL top out near 0.68 AUROC, while multi-provider LLM judges reach about 0.82 and enable useful abstention. Anyone shipping NL-to-SQL needs this when deciding whether to return a query or refuse.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.5","headline":"No significant objection identified beyond the reader's already-flagged transfer-scope caveat; the comparative ceiling claim is well-supported within the tested regime.","rationale":"The strongest claim is an empirical ranking comparison under a fixed execution-correctness label, not a universal claim about all possible verifiers. Tables 1–2, the Spider replication, the second-generator check, and the multi-provider ensemble all point the same direction with paired bootstrap intervals that exclude zero for the stronger judges. The modal-query / K=8 / fixed-slice design is a secondary design choice already noted by the reader; it does not invert the relative ordering of signals. The transfer experiments (Table 4, Appendix A) are the softest part of the argument, but the paper itself reports the failure of the levers it tried and does not over-claim a proof that no fine-tuned model can ever transfer. Because that limitation is already reflected in the CONDITIONAL verdict and high-confidence-within-regime assessment, no further downward adjustment is warranted. The concrete test above is the natural next measurement that would either harden or revise the open-schema recommendation without threatening the black-box-ceiling result.","tokens_in":11987,"tokens_out":566,"duration_ms":6617,"concrete_test":"Re-run the LODO transfer protocol of Table 4 / Appendix A after adding at least one substantially larger open-weight generative verifier (e.g., 32B–70B class) fine-tuned on the same 6 400-pair regime plus the 28-schema diversity set; if its macro LODO AUROC remains ≤0.70 while the frozen GPT-4o judge stays ~0.71–0.77, the transfer conclusion is reinforced; if it reaches ≥0.75 the open-schema recommendation weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central comparative claim—that black-box statistical signals (string/structural/execution self-consistency, schema relevance, executability) and white-box log-probability plateau ~0.61–0.68 AUROC while verification-based judges clear that ceiling (Tables 1–3, Spider replication, two generators, two providers)—is internally consistent and multi-condition supported. The reader's weakest assumption (LODO transfer on ≤28 schemas / ≤7B models underwriting the open-vs-in-domain deployment split) is real but already correctly scoped as a CONDITIONAL caveat rather than a soundness failure of the ranking results. No deeper load-bearing flaw (e.g., circular labeling, metric misuse, or unacknowledged confound that would reverse the ceiling-vs-verifier ordering) is present in the reported experiments.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.5","summary":"The paper studies which signals predict execution correctness of generated SQL on hard multi-table text-to-SQL (BIRD and Spider), framing the problem as selective prediction. Black-box statistical signals (string, structural, and execution self-consistency, schema relevance, executability) and white-box log-probability occupy a narrow band of roughly 0.61–0.68 AUROC, with string self-consistency strongest at 0.675. LLM-as-judge verifiers clear that band (0.72–0.78 alone; 0.82 for a cross-fit two-provider ensemble with ECE 0.03) and yield usable risk–coverage frontiers that self-consistency cannot match. Fine-tuned encoder and generative verifiers reach ~0.77–0.79 AUROC in-distribution but fall to ~0.66 under leave-one-database-out transfer; scale to 7B, schema diversity, rationale distillation, and cross-benchmark training do not close the gap. The authors conclude that correctness uncertainty lives in reasoning-based signals, with fine-tuned verifiers useful in-domain and large frozen judges preferred for open/cross-schema use.","tokens_in":12265,"tokens_out":1405,"duration_ms":24691,"significance":"If the comparative ranking holds, the paper gives practitioners a clear, execution-grounded map of which UQ signals are worth paying for in text-to-SQL: sampling agreement and log-probability are weak correctness predictors on multi-table data, while logic-aware verification (especially multi-provider ensembles) enables calibrated abstention. The separation of selective prediction from self-correction, the paired-bootstrap AUROC comparisons, ECE, held-out risk–coverage, and LODO transfer ablations are concrete strengths. The work is timely for high-stakes database interfaces and complements agentic self-correction literature by asking when to trust a final query rather than how to improve it. The in-domain vs open-schema deployment split is a useful practical message even if the transfer evidence remains limited in scale.","major_comments":[{"comment":"Section 4.4, Table 4, Appendix A, and Discussion: the claim that “cross-schema transfer appears to track model scale and reasoning rather than fine-tuning” is only weakly supported by the reported experiments. LODO is run on 8–28 schemas with generative judges up to 7B; the frozen GPT-4o/Claude judges that transfer better are far larger and differently pretrained, so scale, reasoning ability, and model family are confounded. The negative ablations (diversity, distillation, cross-benchmark) are informative but do not isolate the causal claim. Soften the wording to what the data show—current fine-tunes up to 7B fail to transfer on these schema sets—and treat the open-vs-in-domain split as a practical recommendation under present evidence rather than a settled mechanism.","section":"Section 4.4, Table 4, Appendix A, Discussion"},{"comment":"Section 3 and Tables 1–3: all primary BIRD numbers are on a fixed 800-question slice over 8 databases, scoring the modal query among K=8 samples. The Spider replication and second-generator check reduce the risk that the black-box ceiling is slice-specific, but the absolute AUROC band (0.61–0.68) and the selective-prediction coverages in Table 6 still depend on this design. Please report sensitivity of the ceiling and ensemble AUROC to (i) non-modal selection (e.g., highest-logprob or random sample) and (ii) a second independent BIRD slice or full available multi-table set, so readers can judge how tightly the “ceiling” is tied to the modal-query / fixed-slice protocol.","section":"Section 3, Tables 1–3, Table 6"}],"minor_comments":[{"comment":"Table 7: the GROUP BY row is the only feature subset where the verifier underperforms string self-consistency (0.609 vs 0.651). A short discussion of why (e.g., grouping errors that are hard for the judge prompt, or small n) would help readers interpret the exception.","section":"Table 7, Section 4.6"},{"comment":"Figure 1 is referenced for per-database transfer but the manuscript text does not state how many questions per held-out database or whether error bars are shown; add n per bar and, if possible, bootstrap intervals so the “leads on every schema” claim is visually checkable.","section":"Figure 1, Section 4.4"},{"comment":"Section 4.3 self-correction baseline: one reflection round with GPT-4o-mini is a useful negative control, but the prompt and revision protocol are not in Appendix B. Include them for reproducibility.","section":"Section 4.3, Appendix B"},{"comment":"Abstract and Introduction use “auroc” in lowercase inconsistently with standard AUROC; pick one capitalization and stick to it. Also fix the spacing glitches such as “0 .68”, “0 .78”, “0 .66” that appear throughout.","section":"Abstract, Introduction, Tables"},{"comment":"Related work cites conformal abstention and schema-linking graph models but does not position the present selective-prediction results against DIN-SQL / MAC-SQL style agentic pipelines beyond the one-round reflection check. A short paragraph clarifying complementarity would help.","section":"Section 2"},{"comment":"Limitations correctly note the lack of an open-weight generator; if space allows, even a small open-weight generator check (or an explicit “left to future work” with expected failure modes) would strengthen the multi-family claim.","section":"Section 6"}],"recommendation":"minor_revision","confidential_remarks":"Solid empirical systems paper; the central black-box-ceiling vs verification ranking is multi-condition supported and suitable for a methods/applications venue. The transfer mechanism claim is the main overreach and is fixable with wording plus optional sensitivity checks. I would not require new large-scale fine-tunes for acceptance. Fit is good for a selective-prediction / UQ-for-LLMs track; less so if the venue expects theoretical conformal guarantees as the primary contribution."},"author_rebuttal":null,"desk_editor":{"model":"grok-4.5","letter":"The useful takeaway is clean: on hard multi-table text-to-SQL, sampling agreement, structural/execution consistency, schema relevance, executability, and even sequence log-prob all sit in a narrow ~0.61–0.68 AUROC band, while LLM judges clear that ceiling (up to ~0.78 alone, 0.82 for a GPT-4o + Claude ensemble with ECE 0.03) and produce selective-prediction frontiers that self-consistency cannot. Fine-tuned verifiers work in-domain (~0.77–0.79) but drop to ~0.66 LODO; the paper’s ablations (scale to 7B, schema diversity, rationale distillation, cross-benchmark) do not close it.\n\nWhat is new is not the ingredients—self-consistency, LLM-as-judge, selective prediction—but the execution-grounded head-to-head on BIRD/Spider with paired bootstrap CIs, two generators, two judge providers, calibration, and held-out risk–coverage. The multi-provider ensemble result is the strongest practical finding: different errors (r=0.43) yield both better ranking and usable calibration. The self-correction baseline is a fair negative control; one reflection round barely moves accuracy and produces badly miscalibrated confidence. Error analysis (arithmetic, nesting, CASE, etc.) explains why schema relevance is weak and why verification helps where the failures actually are.\n\nSoft spots are real but proportionate. The open-vs-in-domain deployment split rests on LODO over at most ~28 schemas and generative models ≤7B; that is enough to show current fine-tunes do not transfer, not enough to prove larger scale or broader schema collections never will. The main BIRD numbers use a fixed 800-question slice and modal queries from K=8; the pattern replicates on Spider and a second generator, so the ceiling claim is not slice-fragile, but code/data release would make it more durable. Distribution-free certificates are weak, as the paper itself notes, given ~45% base accuracy. Citation pattern is appropriate; no circularity in the labels or metrics.\n\nThis is for people building or evaluating reliable text-to-SQL systems who need a concrete ranking of uncertainty signals rather than another agent loop. It deserves a serious referee. I would engage with it and cite the ceiling and ensemble numbers when discussing UQ for structured generation.","headline":"Solid empirical ranking study: black-box UQ plateaus ~0.61–0.68 AUROC on hard multi-table SQL while multi-provider verification clears it and enables real abstention; transfer claim is scoped but still the softest part.","tokens_in":12852,"tokens_out":614,"would_cite":true,"duration_ms":6175,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.5","headline":"Correctness uncertainty for text-to-SQL lives in reasoning-based verification, not sampling agreement or log-probability.","keywords":["text-to-SQL","selective prediction","uncertainty quantification","self-consistency","LLM-as-judge","verification","calibration","cross-schema transfer"],"falsifier":"Train a larger generative verifier (or one trained on far more diverse schemas) and re-run leave-one-database-out transfer: if transfer AUROC rises to match a strong frozen judge, the open-versus-in-domain deployment split collapses.","tokens_in":12860,"feed_emoji":"🧮","tokens_out":680,"duration_ms":6138,"temperature":0.7,"pith_summary":"When a system turns a natural-language question into SQL, it also needs a reliable way to know whether that SQL is correct so it can answer or abstain. On hard multi-table benchmarks, cheap signals that look only at how much a model agrees with itself—string, structural, and execution self-consistency, schema relevance, executability, and even the model’s own log-probability—all plateau in roughly the same band (about 0.61–0.68 AUROC). Verification changes the picture: an independent model that reads the question, schema, and candidate SQL and judges whether the computation matches the request moves past that ceiling, and judges from different providers make different errors. Combining two such judges yields the strongest and best-calibrated score, supporting useful abstention frontiers that self-consistency cannot form. Fine-tuned verifiers work well when the schemas are already known, but they do not transfer to unseen databases; generalizing across schemas currently requires a large frozen reasoning model rather than more fine-tuning.","feed_headline":"SQL correctness needs a reasoning judge, not self-consistency","feed_subtitle":"Sampling agreement plateaus; a calibrated two-provider verifier enables abstention that self-consistency cannot","key_machinery":"AUROC ranking of candidate SQL correctness, grounded in execution match to a human reference, used to compare black-box self-consistency signals against logic-aware LLM judges and to measure selective-prediction risk–coverage frontiers.","core_discovery":"On hard multi-table text-to-SQL, black-box statistical signals and white-box log-probability form a correctness ceiling of roughly 0.61–0.68 AUROC, while LLM-as-judge verification exceeds it (up to about 0.78 alone and 0.82 for a calibrated two-provider ensemble). The pattern holds across two benchmarks, two generators, and two judge providers. Fine-tuned verifiers reach high in-domain AUROC but collapse on leave-one-database-out transfer, so cross-schema correctness prediction currently tracks model scale and reasoning rather than fine-tuning.","pith_inferences":[],"forward_implications":[],"fun_headline_variants":["LLM judges clear self-consistency ceiling on text-to-SQL correctness","Reasoning verifiers beat 0.68 AUROC plateau for SQL accuracy","Two-provider ensemble ranks correct SQL at 0.82 AUROC","Fine-tuned SQL verifiers collapse on unseen schemas","Cross-schema SQL correctness tracks reasoning not fine-tuning"],"cache_read_input_tokens":128,"weakest_assumption_plain":"The claim that cross-schema transfer tracks reasoning scale rather than fine-tuning rests on leave-one-database-out tests over a modest number of schemas and generative verifiers no larger than 7B.","fun_headline_variants_meta":{"raw":{"variants":["LLM judges clear self-consistency ceiling on text-to-SQL correctness","Reasoning verifiers beat 0.68 AUROC plateau for SQL accuracy","Two-provider ensemble ranks correct SQL at 0.82 AUROC","Fine-tuned SQL verifiers collapse on unseen schemas","Cross-schema SQL correctness tracks reasoning not fine-tuning"]},"model":"grok-4.5","effort":"low","cost_usd":0.004768,"raw_usage":{"total_tokens":1514,"prompt_tokens":983,"num_sources_used":0,"completion_tokens":73,"cost_in_usd_ticks":47680000,"prompt_tokens_details":{"text_tokens":983,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":458,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":983,"tokens_out":73,"duration_ms":4891,"temperature":1.0,"reasoning_tokens":458,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-07-10T21:04:54.919271+00:00","model_set":{"reader":"grok-4.5"},"falsifier":"Train a larger generative verifier (or one trained on far more diverse schemas) and re-run leave-one-database-out transfer: if transfer AUROC rises to match a strong frozen judge, the open-versus-in-domain deployment split collapses.","supporting_citations":[],"review_version":1}