{"id":"606246f1-c56a-4816-aa20-708222d206e3","arxiv_id":"2606.12387","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":4.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"TAHOE builds a Hint Bank from error traces to raise Text-to-SQL pass rates on Spider 2.0-Snow from 61.95% to 79.42% for GPT-5.5 without parameter updates.","lead":"TAHOE learns reusable hints from past debugging errors to guide LLMs in generating SQL without retraining the model. This could make natural-language database access more reliable and cheaper to deploy in production settings with strict dialects and large schemas.","discovery_kind":"new_method","skeptic_critique":{"model":"grok-4.3","headline":"No significant objection identified","rationale":"Reader's weakest assumption matches the generalization step, but the supplied full text contains no internal contradiction or missing control that would invalidate the development-phase results on the reported set. The limitation (deployment-time updates left for future work) is already stated by the authors.","tokens_in":1841,"tokens_out":264,"duration_ms":12769,"concrete_test":"Re-run the 113-example evaluation after removing any hint whose supporting trace shares a query or schema element with the test item; if pass rate drops below 70% the generalization claim weakens.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim rests on the Hint Bank (Syntax Hints from compiler feedback, Semantic Hints from execution/user feedback) plus Strategy Layer being retrieved and applied effectively at inference. The paper explicitly evaluates only the development-phase workflow on the 113 Spider 2.0-Snow-0212 examples and reports the stated gains plus 100% syntax pass rate and reduced critic rounds. Because the full manuscript supplies the experimental setup, retrieval mechanism, and attribution statistics, the reported numbers are internally consistent with the described pipeline; no hidden assumption about leakage or non-conflict is required for the numbers themselves to hold on the evaluated set.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper presents Tahoe, a system that frames Text-to-SQL prompt optimization as a data management problem. It builds a Hint Bank by distilling compiler feedback into Syntax Hints and execution/user feedback into Semantic Hints during a development phase, introduces a Strategy Layer to handle conflicting intents with attribution statistics, and at inference retrieves hints to guide Logic Planning then SQL Synthesis. The development-phase workflow is evaluated on 113 supervised Spider 2.0-Snow-0212 examples with GPT-5.5, reporting pass-rate gains from 61.95% to 79.42%, pass-at-4 from 72.57% to 87.61%, 100% Snowflake syntax pass rate, reduced critic rounds from 2.79 to 0.12, and transfer gains on Doubao-2.0-lite; deployment-time updates are left for future work.","tokens_in":1950,"tokens_out":577,"duration_ms":15451,"significance":"If the reported gains hold under proper generalization testing, the approach of consolidating debugging traces into a reusable, attributed Hint Bank offers a practical, parameter-free method to adapt LLMs to dialect-specific and schema-specific Text-to-SQL requirements. The explicit transfer results to a weaker backbone and the reduction in critic rounds are concrete strengths that could reduce reliance on expensive test-time scaling or fine-tuning in production settings.","major_comments":[{"comment":"Evaluation section: the reported performance gains (pass rate 61.95% → 79.42%, etc.) are obtained on the same 113 supervised examples used to generate the Hint Bank via development-phase debugging traces. This setup does not test whether the distilled hints remain effective and non-conflicting on truly unseen queries, which is the central assumption required for the deployment claim; the manuscript explicitly defers deployment-time evaluation to future work.","section":"Evaluation"},{"comment":"The manuscript supplies no information on statistical significance, variance across multiple runs, or confidence intervals for the reported percentage-point gains, nor does it detail the exact baseline prompting strategy that produced the 61.95% pass rate; without these, the magnitude of improvement cannot be assessed as robust.","section":"Evaluation"}],"minor_comments":[{"comment":"The abstract and introduction use “GPT-5.5” and “Doubao-2.0-lite” without citing the precise model versions or API endpoints used; add these for reproducibility.","section":"Abstract"},{"comment":"Figure captions and table headers should explicitly state whether the 113 examples are the full development set or a subset, and whether any train/test split was applied within them.","section":"Evaluation"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive comments and for recognizing the potential of the Hint Bank approach. We address each major comment below.","responses":[{"response":"We agree that evaluation on unseen queries would be required to fully support deployment claims. The current results are explicitly scoped to the development-phase workflow, in which the Hint Bank is constructed from error traces on the 113 supervised examples; the manuscript already states that deployment-time human-feedback updates are left for future work. The reported transfer gains on Doubao-2.0-lite provide limited cross-model evidence. We will revise the manuscript to more explicitly delimit the development-phase scope and restate the limitation regarding unseen queries.","revision_made":"partial","referee_comment":"[Evaluation] Evaluation section: the reported performance gains (pass rate 61.95% → 79.42%, etc.) are obtained on the same 113 supervised examples used to generate the Hint Bank via development-phase debugging traces. This setup does not test whether the distilled hints remain effective and non-conflicting on truly unseen queries, which is the central assumption required for the deployment claim; the manuscript explicitly defers deployment-time evaluation to future work."},{"response":"We agree that these details would strengthen the evaluation. The 61.95% baseline reflects standard prompting (zero-shot with the same GPT-5.5 model and no Hint Bank). Experiments were performed in a single run owing to compute limits, so variance, confidence intervals, and significance tests are unavailable. We will revise the manuscript to describe the baseline prompting strategy in detail and to note the lack of multi-run statistics as an acknowledged limitation.","revision_made":"partial","referee_comment":"[Evaluation] The manuscript supplies no information on statistical significance, variance across multiple runs, or confidence intervals for the reported percentage-point gains, nor does it detail the exact baseline prompting strategy that produced the 61.95% pass rate; without these, the magnitude of improvement cannot be assessed as robust."}],"tokens_in":1578,"tokens_out":434,"duration_ms":21998,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"Tahoe gets real gains on Text-to-SQL by distilling debugging traces into a Hint Bank of syntax and semantic hints, plus a strategy layer for conflicts. On the 113 Spider 2.0-Snow examples it boosts pass rate from 62% to 79% with GPT-5.5 and drops compiler rounds sharply.\n\nThe new part is treating the hints as a managed data store with attribution stats on success and harm, and separating compiler-driven syntax rules from execution-driven semantics. That lets them keep 100% syntax compliance and transfer the bank to a weaker model for a 20-point lift.\n\nIt does the empirical side cleanly, with clear before-and-after numbers and no parameter updates needed.\n\nThe main limitation is that they only ran the development-phase workflow. Deployment with live human feedback is future work, so we don't know yet how well the retrieved hints hold up on truly unseen queries without causing new conflicts. The set is also small and the examples are supervised.\n\nThis paper is for engineers and researchers working on making Text-to-SQL reliable in specific database environments like Snowflake. It has enough concrete results and a coherent system to merit a serious referee review, even if more deployment experiments would strengthen it.\n\nI'd recommend sending it out for peer review.","headline":"Tahoe shows clear accuracy lifts on Text-to-SQL by turning error traces into a managed hint bank, but the gains are measured only on the development workflow.","tokens_in":2426,"tokens_out":337,"would_cite":false,"duration_ms":19530,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"Tahoe improves Text-to-SQL by distilling debugging traces into a reusable Hint Bank that guides LLMs at inference without model updates.","keywords":["Text-to-SQL","Hint optimization","LLM prompting","Database query generation","Error-driven learning","Spider benchmark","Prompt engineering","SQL synthesis"],"falsifier":"Running the same 113 Spider 2.0-Snow-0212 examples with the Hint Bank disabled versus enabled and observing no gain or a loss in pass rate would falsify the central claim.","tokens_in":2750,"feed_emoji":"🗃️","tokens_out":743,"duration_ms":17591,"temperature":0.7,"pith_summary":"Tahoe frames prompt optimization for Text-to-SQL as a data management task that builds a Hint Bank from error traces across development and deployment phases. Compiler feedback becomes Syntax Hints for dialect rules while execution and user feedback become Semantic Hints for schema logic, and a Strategy Layer tracks competing intents with success statistics. At inference the system retrieves hints to steer Logic Planning then SQL Synthesis. On 113 supervised Spider 2.0-Snow examples with GPT-5.5 this raises pass rate from 61.95 percent to 79.42 percent and pass-at-4 from 72.57 percent to 87.61 percent while cutting compiler feedback rounds from 2.79 to 0.12. The same bank also lifts performance on a weaker backbone by 19.7 points.","feed_headline":"Hint bank raises Text-to-SQL pass rate from 62 to 79 percent","feed_subtitle":"Tahoe turns past debugging traces into syntax and semantic hints that guide LLMs on new queries without fine-tuning or extra scaling.","key_machinery":"The Hint Bank, a structured store of distilled Syntax Hints, Semantic Hints, and strategy attributions drawn from compiler, execution, and user feedback traces.","core_discovery":"Tahoe consolidates debugging traces into a structured Hint Bank of Syntax Hints for dialect-specific rules and Semantic Hints for schema- and user-specific logic, together with a Strategy Layer that models conflicting intents under shared triggers and records empirical success, harm, inertness, and support; at inference the bank supplies hints that improve an LLM's Logic Planning and SQL Synthesis on unseen queries without any parameter updates.","pith_inferences":["A similar error-driven hint pipeline could replace some supervised fine-tuning in other LLM code-generation settings.","Adding live user-feedback updates to the Strategy Layer would let the bank adapt to shifting preferences over time.","The separation of syntax and semantic hints suggests the method could generalize to other structured output tasks that must respect both rules and domain logic."],"forward_implications":["Tahoe raises pass rate from 61.95 percent to 79.42 percent and pass-at-4 from 72.57 percent to 87.61 percent on the evaluated examples.","It achieves 100 percent Snowflake syntax pass rate while cutting average compiler-feedback critic rounds from 2.79 to 0.12 per candidate.","The Hint Bank transfers to weaker backbones, delivering a 19.7 percentage-point pass-rate gain on Doubao-2.0-lite.","The system handles strict SQL dialects and massive schemas through reusable hints instead of fine-tuning or repeated agentic scaling."],"fun_headline_variants":["Hint Bank improves Text-to-SQL pass rate from 62 to 79 percent","Tahoe forms Hint Bank from debugging traces for LLM SQL synthesis","Syntax and Semantic Hints boost Text-to-SQL on Snowflake without tuning","Strategy Layer resolves conflicting intents in Text-to-SQL queries","Hint Bank guides Logic Planning and SQL Synthesis on unseen queries"],"cache_read_input_tokens":64,"weakest_assumption_plain":"Hints distilled from development-phase debugging traces remain effective and non-conflicting when retrieved and applied at inference time on unseen queries.","fun_headline_variants_meta":{"raw":{"variants":["Hint Bank improves Text-to-SQL pass rate from 62 to 79 percent","Tahoe forms Hint Bank from debugging traces for LLM SQL synthesis","Syntax and Semantic Hints boost Text-to-SQL on Snowflake without tuning","Strategy Layer resolves conflicting intents in Text-to-SQL queries","Hint Bank guides Logic Planning and SQL Synthesis on unseen queries"]},"model":"grok-4.3","cost_usd":0.004264,"raw_usage":{"total_tokens":2203,"prompt_tokens":780,"num_sources_used":0,"completion_tokens":84,"cost_in_usd_ticks":42637000,"prompt_tokens_details":{"text_tokens":780,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":1339,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":780,"tokens_out":84,"duration_ms":11131,"temperature":1.0,"reasoning_tokens":1339,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-27T07:26:48.325351+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Running the same 113 Spider 2.0-Snow-0212 examples with the Hint Bank disabled versus enabled and observing no gain or a loss in pass rate would falsify the central claim.","supporting_citations":[],"review_version":1}