{"id":"a90d0931-056b-4136-93c3-cdca3c46c6dd","arxiv_id":"2506.00391","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A three-stage small-model assistant corrects text-to-SQL output by converting SQL into pandas-like action trajectories and refining schema and logic before the LLM rewrites the query.","lead":"SHARE is a pipeline of three small language models that turn a faulty SQL query into a step-by-step action plan, fix schema and logic errors in that plan, then let a large language model rewrite the SQL. On BIRD and Spider, it lifts GPT-4o's execution accuracy by about 15% and 11% relative in one correction round, at roughly a tenth of the token cost.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"No significant objection identified; the main open risk is the unmeasured fidelity of the BAM action-trajectory intermediate, which the Reader already flags and which the paper's own round-trip filtering partly mitigates.","rationale":"The paper deliberately constructs BAM training data using a round-trip filter (Section 3.2): only (SQL, trajectory) pairs that can be reverted to the original SQL are kept. This is a real, partially machine-checked guarantee of trajectory fidelity on the training distribution, and it directly addresses the Reader's weakest assumption about trajectory faithfulness. The concern is therefore not that the assumption is demonstrably false; it is that the paper never measures how well this fidelity transfers to unseen dev-set SQL, and never measures the reverse conversion from refined trajectory to final SQL. The Reader identified exactly this gap. I agree with the Reader's weakest_assumption, and I did not find a stronger load-bearing concern. The absence of error bars is a reproducibility weakness, not a threat to the central argument; SHARE's single-interaction improvement is large (14.80% BIRD, 11.41% SPIDER relative) and consistent across four benchmarks and seven generator models, which makes it unlikely that the result is purely a variance artifact. The in-house baseline reimplementation and promise-only code release are standard for this literature and are adequately handled by the Reader's CONDITIONAL verdict. Honest non-finding: my proposed check is a verification step, but if it fails it would move the verdict toward REJECT or CONDITIONAL-with-major-revision, so the appropriate current verdict remains the Reader's CONDITIONAL, i.e., UNCHANGED.","tokens_in":27153,"tokens_out":1809,"duration_ms":15897,"concrete_test":"On the BIRD dev set, run the full SHARE inference pipeline and record, for each instance, (1) whether the BAM trajectory, when converted back to SQL via the same SQLGeneration prompt (Figure 13) without any SAM/LOM refinement, reproduces the initial SQL exactly; and (2) whether the final refined SQL, when converted back to a trajectory via BAM, matches the refined trajectory exactly. If round-trip fidelity is high (e.g., >90%) on both checks, the trajectory intermediate is faithful and the central mechanism holds; if fidelity is low in a substantial fraction of the instances where SHARE improves EX, the gains may come from the LLM's regeneration rather than from trajectory-level correction and the central claim would need to be weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that an 8B-parameter assistant composed of BAM, SAM, and LOM materially improves LLM self-correction for text-to-SQL, with relative EX gains of 14.80% on BIRD and 11.41% on SPIDER (Table 2). For that claim to hold, the action-trajectory representation must faithfully encode the SQL reasoning and the refined trajectory must be reliably convertible back to correct SQL. The paper provides strong indirect evidence for this: BAM training data is filtered to include only (SQL, trajectory) pairs that round-trip back to the original SQL (Section 3.2), which directly tests representational sufficiency on the training distribution; the ablation removing SAM or LOM degrades performance (Table 5); and the case study in Appendix C.6 shows a trajectory refinement that fixes both schema and logic errors. What is missing is a direct measurement of BAM's trajectory fidelity on the dev set (e.g., what fraction of BAM trajectories round-trip to the initial SQL) and of the final LLM conversion accuracy (how often the refined trajectory converts to the refined SQL). This matters because if BAM introduces spurious or lossy steps on unseen queries, the downstream SAM/LOM corrections could be applied to a corrupted intermediate, and the observed gains could partly come from the LLM's own SQL regeneration rather than from SHARE's trajectory-level corrections. This is a correctness-risk concern about an unmoved intermediate link, not an internal inconsistency; the paper's own Section 3.2 filtering and the ablations are genuine partial support, so the concern is a call for a missing measurement rather than a demonstrated flaw.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes SHARE, a pipeline of three small language models (BAM, SAM, and LOM) that assists an LLM generator in self-correcting text-to-SQL outputs. BAM converts an initial SQL query into a pandas-like action trajectory intended to expose the query's reasoning; SAM corrects schema-linking errors in the trajectory; LOM corrects logic errors; and the generator LLM then converts the refined trajectory back into SQL. The models are trained with a GPT-4o teacher for BAM, while BAM itself synthesizes training data for SAM and LOM through masking and error perturbation. Experiments on BIRD, SPIDER, DK, and REALISTIC report execution accuracy (EX) gains over GPT-4o and several other generator models, together with ablations, a cost analysis, a low-resource data analysis, dialect-generalization experiments, and an open-source-teacher variant.","tokens_in":27584,"tokens_out":8541,"duration_ms":80645,"significance":"If the reported results hold, SHARE is a practical contribution: it shows that an 8B-parameter assistant can improve LLM self-correction in text-to-SQL while reducing token cost relative to calling a large proprietary model repeatedly. The paper is empirically broad, covering four benchmarks, multiple generator models, ablations that support the role of each module, a standalone-inference experiment (Appendix C.3), an overcorrection analysis (Appendix C.4), and a comparison of teacher models (Table 7). The work also ships a reproducibility commitment (code and models). However, the central mechanism depends on the faithfulness of an action-trajectory intermediate that is never directly measured on the evaluation sets, and several reporting gaps make some comparisons hard to assess.","major_comments":[{"comment":"The paper never directly measures the fidelity of the action-trajectory intermediate on the evaluation sets. The entire mechanism assumes that BAM's trajectory faithfully encodes the SQL semantics and that the generator can reliably convert a refined trajectory back into correct SQL. The round-trip filter used during BAM data construction tests reversibility only on the training distribution, and the case study in Appendix C.6 gives one positive example. The paper should report, on the BIRD and SPIDER dev sets, (i) what fraction of BAM-generated trajectories round-trip to the initial SQL, and (ii) how often the generator's conversion of the refined trajectory produces executable SQL that matches the intended correction. Without these measurements, part of the observed gains could come from the generator's own regeneration rather than from the trajectory-level corrections made by SAM and LOM.","section":"Section 3.2 and Figure 13"},{"comment":"The key comparison table is incomplete and lacks variance information. The MAGIC row reports SPIDER only as a total (85.66) with all difficulty-level sub-scores missing, while every other method in the table has a full breakdown; no explanation is provided. In addition, all EX numbers in Tables 2–5 are point estimates, with Appendix B.2 only stating that results are 'the average of five repeated trials.' Several differences are small in absolute terms (for example, +0.24 over MAGIC on the SPIDER total and many per-difficulty gaps under 3 points), so without standard deviations, confidence intervals, or significance tests, the reader cannot tell whether those differences are meaningful. The authors should report variance or CIs and either complete or explicitly justify the MAGIC row.","section":"Section 4.2, Table 2"},{"comment":"The dialect-generalization experiment is missing its experimental protocol. The text claims that SHARE performs well on MySQL and PostgreSQL 'even without additional training,' but it does not describe how the SQLite-based BIRD dev set was converted to these dialects, how the GPT-4o baseline was prompted in the new dialects, or whether the same action-trajectory inference and evaluation pipeline was used. Without this protocol, the comparison in Figure 4 is not interpretable. A concise description of the SQL translation, generation, and execution steps is needed.","section":"Section 4.5, Figure 4"},{"comment":"The composition of LOM training data is not reported, which matters for the generalizability claim. The paper states that erroneous trajectories come from two sources: erroneous initial SQLs (generated by GPT-4o using the same baseline prompt used at evaluation) and synthetic perturbations of verified trajectories. Since the main evaluation uses GPT-4o as the generator, a large share of real GPT-4o errors in LOM training could align the assistant with GPT-4o's specific error distribution. The cross-generator results in Table 4 mitigate this concern, but the paper should state the fraction of the 15k LOM examples from each source and ideally show a version trained only on synthetic perturbations.","section":"Section 3.1 and Appendix A.3"}],"minor_comments":[{"comment":"The text says 'Table 5 presents the correction performance across fine-grained error categories,' but Table 5 is the ablation study and the fine-grained error analysis appears in Figure 5. Please correct this cross-reference.","section":"Section 4.7"},{"comment":"The number of repeated trials is inconsistent: Appendix B.2 says results are the average of five repeated trials, while the low-resource paragraph says each experiment was repeated three times. Please state the number of repetitions used for each reported table.","section":"Appendix B.2 and Section 4.2"},{"comment":"The caption contains a typo: 'Logic Optimation Model' should be 'Logic Optimization Model.'","section":"Table 8 caption"},{"comment":"The value of K, the number of error perturbations per verified trajectory, is never specified. Please report the value used to construct the 15k LOM training set.","section":"Section 3.4 and Algorithm 1"},{"comment":"The x-axis labels in the rendered figure are illegible, and the text refers to 0%, 10%, 20%, 50%, and presumably 100% of training data, but these points are not clearly marked. Please redraw the figure with explicit axis labels.","section":"Figure 3"},{"comment":"The symbol s_i is used for both the initial SQL and the refined SQL in the problem definition; using a prime or a different subscript for the refined query would avoid ambiguity.","section":"Section 2.1, Eq. (2)"}],"recommendation":"major_revision","confidential_remarks":"The paper is a solid empirical systems contribution, and I do not see a fundamental flaw in the central idea. The most important risk is the unmeasured fidelity of the action-trajectory intermediate; if dev-set round-trip rates turn out to be low, the mechanism story would need substantial revision. I would prioritize that analysis over additional benchmark results. Also, please double-check the MAGIC SPIDER total of 85.66 in Table 2: it is much higher than the GPT-4o baseline of 77.10 and has no breakdown, so it is worth verifying before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"I read SHARE. It's a legitimate contribution to the text-to-SQL self-correction literature, and the empirical case is stronger than I expected. The core idea is that instead of having an LLM repeatedly self-correct, you train three small models (BAM, SAM, LOM) to convert a SQL query into a pandas-like action trajectory, fix schema and logic errors in that trajectory, and then feed the refined trajectory back to the LLM to regenerate the SQL. That's a genuinely useful division of labor. The paper shows relative gains of 14.8% on BIRD and 11.4% on Spider over a GPT-4o baseline, with ablations confirming that each module contributes, and it generalizes across several generator models.\n\nWhat impressed me: the training data construction includes a round-trip filter on the (SQL, trajectory) pairs, which is real evidence that the action-trajectory representation is reversible on the training distribution. The cost analysis is also transparent and shows a large reduction in API tokens. The cross-generator results are more than cherry-picked.\n\nNow the soft spots. The biggest one is exactly what the reader flagged: the trajectory is the load-bearing intermediate, but the paper never directly measures how often BAM produces a faithful trajectory on the dev set, nor how often the LLM converts the refined trajectory back to correct SQL. The round-trip filter and the ablations are partial support, but they leave open the possibility that some of the gain comes from the LLM simply regenerating SQL after seeing a differently structured prompt. That's a missing measurement, not a demonstrated flaw, but it should be addressed before I'd call the result fully explained.\n\nThe other issues are more standard. The comparison numbers come from in-house reimplementations of the baselines, and there are no error bars or significance tests—the paper just says \"average of five repeated trials\" in the appendix. For a paper that leans heavily on single-percentage-point differences, that's a real weakness. The code and checkpoints are promised but not released, which makes independent verification harder.\n\nOverall, this is a paper a serious referee should take. The architecture is plausible, the empirical work is broad, and the central claim holds up under scrutiny. I'd accept it with requests for variance statistics, a fidelity measurement, and artifact release. If you work in text-to-SQL or on LLM self-correction pipelines, this is worth reading.\n\nRecommendation: send it to peer review.","headline":"A solid empirical paper on SLM-assisted self-correction for text-to-SQL; the central claim holds up, but the unmeasured trajectory fidelity and missing error bars keep it from being definitive.","tokens_in":28057,"tokens_out":1818,"would_cite":true,"duration_ms":15877,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"SHARE claims that a pipeline of three small language models, each under 8B parameters, can localize and repair SQL errors more precisely than the large generator model can on its own, improving GPT-4o's execution accuracy by 14.80% on…","keywords":["text-to-SQL","self-correction","small language models","action trajectory","schema linking","logic optimization","hierarchical self-evolution","execution accuracy"],"falsifier":"Take the set of BIRD and SPIDER instances where SHARE still fails after correction, translate the original SQL and the final SQL into action trajectories, and measure round-trip fidelity: whether converting each trajectory back to SQL reproduces the same execution results over the database. If a substantial fraction of failures coincides with trajectories that lose or distort SQL semantics (for example, dropping join conditions or set operations), the central assumption fails; if failures occur despite faithful trajectories, the bottleneck lies elsewhere, such as in the generator's ability to read the repaired trajectory.","tokens_in":26928,"feed_emoji":"🗄️","tokens_out":7051,"duration_ms":60094,"temperature":0.7,"pith_summary":"SHARE is an attempt to make LLM self-correction for text-to-SQL both more accurate and far cheaper by replacing the recursive self-calls of a large model with a one-pass pipeline of three small language models, each under 8 billion parameters. The paper's central claim is that converting a declarative SQL query into a stepwise \"action trajectory\" of pandas-like operations exposes the query's reasoning path, so that errors can be localized and repaired more precisely than the LLM can do on its own. In experiments with GPT-4o as the generator, SHARE-8B raises execution accuracy by a relative 14.80% on BIRD and 11.41% on SPIDER in a single correction round, at roughly one-tenth the inference cost of the cheapest compared self-correction method. If this holds, it means small, privacy-friendly models can serve as general correction assistants without repeated expensive LLM calls.","feed_headline":"Three small AI models fix SQL errors better and cheaper than GPT-4o","feed_subtitle":"Converts SQL into stepwise actions, repairs schema and logic, lifts GPT-4o's accuracy 11-15% at one-tenth the cost.","key_machinery":"The central object is the action trajectory: a declarative SQL query decomposed into a sequence of pandas-like function calls (where, select, groupby, orderby, limit, distinct, union, plus aggregation and operator actions) that expose the query's reasoning steps. It carries the argument because it turns \"correct the SQL\" into \"correct the trajectory\", a task where schema errors and logic errors can be isolated in separate stages. In the inference loop, the Base Action Model (BAM) performs the SQL-to-trajectory conversion, the Schema Augmentation Model (SAM) masks and refills schema tokens to repair schema linking, and the Logic Optimization Model (LOM) edits the trajectory to repair logic, after which the generator LLM translates the final trajectory back to SQL. Training relies on three scripted perturbation types (ADD, DELETE, SUBSTITUTE) applied to error-free trajectories to synthesize erroneous ones, plus a hierarchical self-evolution loop where BAM, after teacher distillation, generates the augmentation data for the other two models.","core_discovery":"The paper claims that the reason LLMs fail at self-correcting SQL is that they cannot see the reasoning behind a declarative query, and that this failure can be circumvented by an intermediate representation. SHARE converts the initial SQL into an action trajectory, a sequence of pandas-like functions such as where, select, groupby, and orderby, that mirrors the query's stepwise logic. A Schema Augmentation Model then masks and re-fills schema elements to fix schema-linking mistakes, and a Logic Optimization Model revises the trajectory to fix logical errors, with both models trained under a hierarchical self-evolution strategy in which the Base Action Model generates augmented data after initial distillation from GPT-4o. The refined trajectory is returned to the LLM, which regenerates the SQL. The paper reports that this single-round zero-shot feedback loop improves execution accuracy for several generator models, holds up with as little as 50% of the training data, and generalizes to MySQL and PostgreSQL without dialect-specific supervision.","pith_inferences":["The paper never directly measures how faithfully an action trajectory preserves SQL semantics; a trajectory can be internally consistent yet drop a join condition or set-operation nuance. A direct round-trip fidelity test (trajectory to SQL back to trajectory on a held-out error set) would show whether correction failures cluster where the trajectory distorts the original query.","Because the assistant is demonstrated only for a single correction round, a natural next step is to test whether repeated rounds compound gains or accumulate drift; the current evidence does not rule out either outcome.","The trajectory abstraction is a general idea for code correction beyond SQL, a direction the paper leaves open; the mechanism only requires a set of elementary operations into which a declarative program can be decomposed.","The error-category analysis shows weak gains on \"mathematical delusion\" errors (only a 1.63% reduction), which hints that trajectory repair fixes structure and schema but does not add numerical reasoning power the generator itself lacks."],"forward_implications":["Single-round correction with SHARE-8B outperforms the strongest compared baseline (MAGIC) by 4.61 execution-accuracy points on BIRD while costing about a tenth as much per thousand instances.","The same trained assistant improves accuracy for other generators, including Claude-3.5-Sonnet (up 28.64% relatively), GPT-4o-mini, Llama-3.1-70B, and a 6.7B open-source model, not just the GPT-4o teacher it was distilled from.","Training with only 50% of the data still beats the strongest baseline, suggesting the method is data-efficient; the paper also reports a threshold effect where gains on hard instances stay erratic until training data exceeds about 20% of the full set.","Without additional training, SHARE transfers to MySQL and PostgreSQL, supporting the claim that the models learn low-level reasoning-path corrections rather than dialect-specific patterns.","Replacing the GPT-4o teacher with Llama-3.1-70B keeps most of the gain, indicating the pipeline does not depend on the teacher's particular error patterns."],"supporting_citations":[{"why":"Supplies the BIRD benchmark, the main evaluation set and source of the baseline SQL-generation prompt.","marker":"(Li et al., 2024a)"},{"why":"Supplies the SPIDER benchmark, the second main evaluation set used for the core accuracy results.","marker":"(Yu et al., 2018)"},{"why":"Provides the action-trajectory design for SQL reasoning and the error taxonomy used in the quantitative error-correction analysis.","marker":"(Qu et al., 2024)"},{"why":"Provides MAGIC, the strongest self-correction baseline that SHARE is compared against in both accuracy and cost.","marker":"(Askari et al., 2024)"},{"why":"Provides LoRA, the parameter-efficient fine-tuning method used to train all three SLMs in the pipeline.","marker":"(Hu et al., 2022)"},{"why":"Provides the evidence that LLMs cannot reliably self-correct their own reasoning, which motivates building an external assistant.","marker":"(Huang et al., 2024)"},{"why":"Provides the Self-Consistency baseline that SHARE is compared against in the main results.","marker":"(Wang et al., 2023)"},{"why":"Supplies the general notion of action models that the Base Action Model instantiates.","marker":"(Zhang et al., 2024)"}],"fun_headline_variants":["SLM pipeline slashes SQL self-correction cost while boosting accuracy","Action trajectories reveal SQL reasoning, letting small models fix errors","Hierarchical SLMs teach LLMs to fix SQL in one round, not many","SHARE's small models lift GPT-4o SQL accuracy 11-15% at 1/10th cost","SHARE trains SQL fixers with half the data, keeping accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The pipeline assumes that the pandas-like action trajectory is a faithful, verifiable intermediate that preserves all correction-relevant meaning of the SQL query, and that the LLM can reliably convert a repaired trajectory back into correct SQL, even though neither of those steps is measured in the paper.","fun_headline_variants_meta":{"raw":{"variants":["SLM pipeline slashes SQL self-correction cost while boosting accuracy","Action trajectories reveal SQL reasoning, letting small models fix errors","Hierarchical SLMs teach LLMs to fix SQL in one round, not many","SHARE's small models lift GPT-4o SQL accuracy 11-15% at 1/10th cost","SHARE trains SQL fixers with half the data, keeping accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001202,"raw_usage":{"total_tokens":4955,"prompt_tokens":951,"completion_tokens":4004,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":3901}},"tokens_in":567,"tokens_out":4004,"duration_ms":30148,"temperature":1.0,"reasoning_tokens":3901,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T12:06:11.287911+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the set of BIRD and SPIDER instances where SHARE still fails after correction, translate the original SQL and the final SQL into action trajectories, and measure round-trip fidelity: whether converting each trajectory back to SQL reproduces the same execution results over the database. If a substantial fraction of failures coincides with trajectories that lose or distort SQL semantics (for example, dropping join conditions or set operations), the central assumption fails; if failures occur despite faithful trajectories, the bottleneck lies elsewhere, such as in the generator's ability to read the repaired trajectory.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the evidence that LLMs cannot reliably self-correct their own reasoning, which motivates building an external assistant."}],"review_version":1}