{"id":"fa526241-bb6f-4286-a6b5-ff439005827f","arxiv_id":"2608.07213","paper_version":1,"verdict":"ACCEPT","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"Storing verified corrected text-to-SQL queries in a per-database bank improves held-out first-attempt accuracy by 4.34 percentage points, capturing 44.4% of on-demand repair headroom.","lead":"Keeping repaired text-to-SQL answers in a per-database memory bank can help future questions, not just the question that paid for the repair. This paper measures that effect and finds a 4.34 percentage point accuracy gain on held-out questions, about 44% of the gain from repairing the same questions on demand.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified.","rationale":"The reader's weakest-assumption (reliable verification) is the most natural boundary of the central claim, and I agree it is the main condition for deployment transfer. However, the paper explicitly scopes the headline to a verified bank, demonstrates that ungated storage is harmful, and tests robustness to verification noise. This is a limitation, not a defect in the measured benchmark claim. I also considered whether the local-vs-foreign contrast could confound database-specific content with question similarity, but the lesson-only control (full local card vs. natural-language lesson) and the mispairing control isolate the contribution of SQL/database details while holding locality roughly fixed. No internal inconsistency or unsupported leap was found. The paper's confirmatory-analysis discipline, two-stage bootstrap, leakage checks, and honest reporting of null results (richer formats, retriever methods, repair-specific advantage) further support the ACCEPT verdict. Since the code and artifacts were not executed during this review, the concrete test above is a reproducibility verification rather than a response to a specific identified flaw.","tokens_in":28787,"tokens_out":13616,"duration_ms":148580,"concrete_test":"Run the open-source artifact's main confirmatory contrast (W0 verbatim memory vs. no memory, three paired seeds, BIRD dev) on a fresh serving host and verify that the held-out transfer gain reproduces within the stated rerun-drift band (median 0.65pp, maximum 1.74pp, Appendix D). As a secondary check, confirm that the ungated self-vote bank remains below the no-memory floor in all seeds, since this is the paper's own necessary-condition result.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The closest candidate for a load-bearing concern is the dependence on a reliable verification gate: the headline +4.34pp gain is measured with an oracle that compares execution results against gold, and the paper's own ungated self-vote ablation is 2.03pp below the no-memory floor (Section 5.3). This, however, is an explicitly stated and tested boundary rather than an internal flaw. The central claim is conditional on a verified bank; the paper does not claim the benefit survives without a trustworthy gate, and it reports synthetic noise experiments showing the gain persists under 5-20% random verification flips. The mechanism claim (database-specific content, not matched examples) is supported by local-foreign, grounding-stripping, mispairing, and similarity-targeting controls that are internally consistent. The statistical protocol, leakage checks, and confirmatory-versus-exploratory split are carefully executed. Within its stated scope, the argument holds; no unresolved internal inconsistency undermines the central claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper formalizes the 'crystallization problem' for text-to-SQL memory: how to measure the future value of verified test-time repair episodes rather than collapsing everything into one end-to-end score. With a fixed single-shot solver, the authors build per-database banks from repair episodes on a 70% collection split, then evaluate exact-query replay, cross-question retention, and held-out same-database transfer on BIRD (with Spider as a secondary probe). The headline result is that a verbatim bank of oracle-verified corrected queries raises held-out first-attempt accuracy from 62.04% to 66.38% (+4.34pp, two-stage bootstrap CI [+1.50, +7.49], p=0.0034), corresponding to 44.4% (CI [24,65]%) of the on-demand repair headroom measured on the same questions. Controlled interventions show that database-specific content, not matched question-SQL examples, is the main carrier; reliable verification and wider retrieval coverage help, while richer card formats and more elaborate retrievers do not show statistically supported gains. Cost, robustness, leakage, and statistical-discipline analyses are reported in depth.","tokens_in":28866,"tokens_out":18699,"duration_ms":167662,"significance":"The contribution is primarily methodological but consequential: it decomposes the single end-to-end memory score into replay, retention, and transfer, and anchors transfer in a descriptive crystallization ratio against a measured repair reference. The experimental protocol is unusually careful: pre-specified confirmatory comparisons, three paired seeds, two-stage hierarchical bootstrap over databases and questions, sentinel-based leakage checks, rerun-drift quantification, and explicit exploratory-versus-confirmatory labeling. The verification-gate dependence is the clearest boundary of the central claim; the paper explicitly identifies it and shows with the ungated ablation that the bank is harmful without a trustworthy gate. Because the claim is conditional on a verified bank, this is a scoped limitation rather than an internal inconsistency. If the results hold, they give practitioners concrete guidance: verified database-scoped banks can substitute for a substantial share of on-demand repair on recurring or same-database workloads.","major_comments":[],"minor_comments":[{"comment":"The retrieval operator rho_k is used in Equations (2) and (4) before being formally defined; please add an explicit definition of rho_k(q; M_d) near Eq. (2) so the notation is self-contained.","section":"Section 3.3 / Eq. (2)"},{"comment":"The 'remaining headroom +5.42pp' label and the focused scale in panel (a) could be misread; consider adding explicit numeric labels on the bars for P0, PM, and the repair reference, or annotating the scale break directly.","section":"Section 5.1 / Figure 3"},{"comment":"The sentence 'one repair round captures 80% of what three memoryless rounds achieve at 59% of the calls' is terse; please specify exactly which configurations enter this comparison so the reader can verify the 80% and 59% figures.","section":"Section 5.4 / Table 2"},{"comment":"The worked-example text says capsule exact-query replay 'stays at 100%,' while Table 5 reports a three-seed mean of 99.7%; please align the wording to avoid an apparent inconsistency.","section":"Appendix E / Table 5"},{"comment":"The release does not redistribute the quality-controlled split or raw transcripts; consider releasing question IDs or hashes so the disjointness and leakage checks can be independently verified by readers.","section":"Section 7 / Appendix E"},{"comment":"Because the verification gate is central to practical applicability, please consider reporting the key synthetic verification-noise numbers in the main text rather than only referring to Appendix D.","section":"Section 5.3 / Appendix D"}],"recommendation":"accept","confidential_remarks":"I see no concerns about citation practices or novelty; the related-work mapping is useful and the paper is well within scope for cs.CL. The main practical caveat is the dependence on a trustworthy verification signal, but the paper is transparent about this boundary and its own ungated ablation makes the condition explicit. The manuscript is unusually rigorous in its statistical protocol and leakage checks; I am comfortable accepting it."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Punchline: this is the evaluation paper that the self-evolving text-to-SQL line has been missing. It fixes the solver and varies one memory choice at a time, separately measuring replay, cross-question retention, and held-out same-database transfer. That decomposition, plus the CR normalization against repair headroom, is genuinely new relative to the bundled end-to-end scores that dominate the cited literature. The headline result — +4.34pp transfer from verbatim corrected queries, about 44% of the on-demand repair headroom — is backed by a paired three-seed design, a two-stage bootstrap CI of [+1.50, +7.49], p=0.0034, positive in 10 of 11 databases. I read the appendices: the leakage sentinel, the confirmatory-versus-exploratory split, and the per-question logs all look like the real thing, not window dressing.\n\nThe mechanism arms are the interesting part. Local cards beat foreign cards by a wide margin; mispairing all the local SQL preserves 73% of the aligned lift; grounding stripping hurts. That paints a coherent picture: the memory works mainly by exposing database-specific schema/value knowledge, not by giving the model matched examples to copy. And the null results on richer formats and fancier retrievers are a useful antidote to the field's reflex to make memory more elaborate.\n\nSoft spots, in proportion. The biggest is the dependence on a reliable verification gate, and the paper does not dodge it: the ungated self-vote bank is 2pp below the no-memory floor. In benchmark mode that gate is an execution oracle against gold. The paper argues user confirmation or a business check can play that role, but that is a deployment assumption, not something they tested. It is an explicit boundary rather than an internal contradiction, but it means the headline gain will not transfer to settings without a trustworthy admission signal. Second, the multi-seed evidence is two Qwen scales on BIRD dev; the other model families are single-seed probes, and Spider's transfer deltas are small because its floor is high. They label these as preliminary, so it is a scope limit, not a statistical overreach. Third, the CR ratio is descriptive and has a wide CI, which they also say.\n\nOverall: the central claim holds within its stated scope. It deserves a serious referee and would be a strong addition to the agent-memory and text-to-SQL evaluation literature. I would bring it to reading group and I would cite it in my own work, especially the mispairing result and the replay/retention/transfer split.","headline":"A clean, carefully scoped measurement of when and why text-to-SQL memory helps; the verification dependence is real but explicitly tested.","tokens_in":29476,"tokens_out":2251,"would_cite":true,"duration_ms":22234,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Stored verified repair episodes for text-to-SQL transfer to unseen questions, capturing 44.4% of the accuracy headroom that on-demand repair would have spent on the same questions.","keywords":["text-to-SQL","test-time scaling","agent memory","experience reuse","execution feedback","evaluation methodology","crystallization ratio","memory verification"],"falsifier":"A held-out transfer test on a benchmark pack where no database appears in the collection split should show no positive memory lift, because the paper's foreign-card control predicts a negative trend below no memory; a positive gain in that setting would contradict the database-specific-content account.","tokens_in":28560,"feed_emoji":"🗄️","tokens_out":6926,"duration_ms":60299,"temperature":0.7,"pith_summary":"The paper tries to establish that the computation spent repairing failed text-to-SQL queries can be turned into reusable memory instead of being discarded: a verified, database-scoped bank of corrected episodes should help future queries over the same database. On the BIRD benchmark, with a fixed single-shot solver, storing corrected queries verbatim raises held-out first-attempt accuracy from 62.04% to 66.38%, a gain of 4.34 percentage points that equals 44.4% of the headroom provided by on-demand repair on those same questions. The paper further claims that the transfer is carried mainly by database-specific content, such as identifiers, values, joins, and query structures, rather than by matched question-to-SQL examples to copy, and that an independently reliable verification signal is necessary for the benefit. A sympathetic reader should care because end-to-end memory scores conflate replay of recurring questions with help on unseen ones, and this study separates those settings and attributes the gain to a specific, testable mechanism.","feed_headline":"Stored repair memory captures 44% of text-to-SQL repair headroom","feed_subtitle":"Verified corrected queries lift held-out accuracy by 4.34 points; a verification gate is essential.","key_machinery":"The central object is the verified, database-scoped memory bank $M_d = \\{ W(e) : e = (q, \\hat{y}, y^+, \\tau) \\in \\mathcal{E}_d,\\ v(y^+) = 1 \\}$, where each episode consists of the question, first attempt, corrected query, and interaction trace, $W$ is a card writer, and $v$ is a verification signal that decides admission. The identity that carries the measurement is the crystallization ratio $\\mathrm{CR} = (P_M - P_0)/(P_K - P_0)$: the held-out transfer lift divided by the on-demand repair headroom on the same questions. It is a descriptive ratio, not a probability, and it is only meaningful when the solver, split, repair budget, and collection procedure are held fixed. The paper's controlled-comparison design, varying acquisition, admission, writing, and consumption one at a time while keeping the same greedy solver, is what lets it attribute the gain to database-specific payload rather than to any single design choice.","core_discovery":"Under controlled evaluation, the paper's central discovery is that a per-database bank of verified, verbatim corrected queries produces value in three distinct future-use settings: exact-query replay (96.1% accuracy when the question's own card is eligible), cross-question retention (56.0% when that card is removed), and held-out same-database transfer (+4.34 pp, CR = 44.4%). The crystallization ratio CR normalizes the memory lift against the on-demand repair headroom measured on the same held-out questions, making the memory gain commensurable with the compute it would otherwise cost to repair. Interventions that permute question–SQL pairings inside the bank preserve 73% of the aligned lift, while equally valid cards from other databases do not transfer, leading the paper to conclude that the cards mainly expose database-specific content rather than supply copyable worked examples. The same experiments show that without a reliable verification gate, storing self-voted corrections is 2.03 pp below no memory and 4.85 pp below the verified bank, so verification is described as necessary, not optional.","pith_inferences":["The 44.4% CR is likely a function of the workload mix rather than a universal constant: workloads dominated by novel questions should be bounded by the transfer tier, while recurring questions push effective value toward the near-100% replay tier.","Because mispairing preserves most of the lift, a testable extension is to strip question text from cards entirely and retrieve on SQL structure alone; if the gain survives, the useful signal is payload rather than natural-language similarity.","The paper's fix/break symmetry suggests a retrieval-side guard: suppressing cards whose foreign SQL elements, such as LIMIT, aggregation, or literal formats, would alter answer cardinality could reduce the 67 broken cases without losing the 127 fixes.","The measured amortization horizon of roughly 7.5–9.9K future queries invites a decision rule for when to build memory versus repair on demand, but combining the conditional reuse values into one expected-utility number also requires the workload's recurrence distribution."],"forward_implications":["If the central claim holds, storing verified corrected queries in a per-database bank substitutes for roughly 44% of per-question repair on seen databases, and the remaining repair headroom is still available on top.","Recurring questions are nearly free under memory: exact-query replay reaches 96.1% when the originating card is retrieved, so workloads with repeated questions will amortize construction cost faster.","Verification is the dominant admission decision: gated storage beats ungated by 4.85 pp, and the ungated bank is below no memory, so deployment must supply an independent correctness channel.","Widening retrieval from $k=1$ to $k=10$ adds 3.18 pp, which means cumulative coverage of database-specific content, not a smarter matching algorithm, is what increases transfer.","Richer card formats and more elaborate retrievers have no statistically supported advantage, so the simple operating point, verbatim corrected SQL, broad same-database retrieval, and a reliable gate, is the supported default."],"supporting_citations":[{"why":"Supplies the execution-guided self-debugging repair loop that generates the episodes stored in the bank.","marker":"[8]"},{"why":"Provides the BIRD benchmark, the dataset whose development set carries the main held-out transfer and repair-headroom measurements.","marker":"[25]"},{"why":"Supplies the mechanistic demonstration methodology, random labels preserving in-context learning, that the controlled attribution design adapts.","marker":"[31]"},{"why":"Supplies the rationale-irrelevance control that frames the paper's mispaired-card correspondence intervention.","marker":"[50]"},{"why":"A prior corrected-query memory system whose end-to-end evaluation motivates the need for separated crystallization measurement.","marker":"[11]"},{"why":"A prior experience-driven self-correction text-to-SQL memory line whose mechanism is re-instantiated as the verbatim-card cell.","marker":"[56]"},{"why":"Supplies the distilled-guidelines memory mechanism that the capsule-card format is compared against.","marker":"[2]"}],"fun_headline_variants":["Stored repair memory captures 44% of text-to-SQL headroom","Verified query memories lift text-to-SQL accuracy by 4.34 points","Memory bank for text-to-SQL: 44% of repair benefit from verified corrections","Crystallized memories: reusable text-to-SQL repairs yield 44% gain"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benefit requires an independently reliable verification signal that decides which repaired episodes are stored; without such a gate, the paper's own ablation shows the bank hurts accuracy.","fun_headline_variants_meta":{"raw":{"variants":["Stored repair memory captures 44% of text-to-SQL headroom","Verified query memories lift text-to-SQL accuracy by 4.34 points","Memory bank for text-to-SQL: 44% of repair benefit from verified corrections","Crystallized memories: reusable text-to-SQL repairs yield 44% gain"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000544,"raw_usage":{"total_tokens":2611,"prompt_tokens":960,"completion_tokens":1651,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":576,"completion_tokens_details":{"reasoning_tokens":1565}},"tokens_in":576,"tokens_out":1651,"duration_ms":11862,"temperature":1.0,"reasoning_tokens":1565,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T12:38:08.526860+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A held-out transfer test on a benchmark pack where no database appears in the collection split should show no positive memory lift, because the paper's foreign-card control predicts a negative trend below no memory; a positive gain in that setting would contradict the database-specific-content account.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the execution-guided self-debugging repair loop that generates the episodes stored in the bank."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the BIRD benchmark, the dataset whose development set carries the main held-out transfer and repair-headroom measurements."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the mechanistic demonstration methodology, random labels preserving in-context learning, that the controlled attribution design adapts."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the rationale-irrelevance control that frames the paper's mispaired-card correspondence intervention."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the distilled-guidelines memory mechanism that the capsule-card format is compared against."}],"review_version":1}