{"id":"30037db1-9a48-4a0d-bb7f-00f1c7e9fcf8","arxiv_id":"2412.05850","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"Two LLM agents that each hold only part of a database schema can cooperate to write SQL almost as accurately as a single agent with the full schema.","lead":"This paper proposes CSMA, a system where several AI agents that each know only part of a database schema cooperate to write SQL queries from natural-language questions. The main result is that two partial-schema agents nearly match one full-schema agent, but the evaluation uses the answer key to choose its prompting strategy.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Sec 4.3 selects the SQL-generation prompt using the complexity of the golden SQL, so the headline TwoPart-versus-OneAll comparison in Table 1 is contaminated by label information and cannot support the cooperative-schema claim on its own.","rationale":"I agree with the reader's rejection but not with the emphasis in the weakest_assumption field. The table-disjoint partition assumption is a limitation of scope, but the decisive problem is internal: Sec 4.3's golden-SQL difficulty signal makes the evaluation circular on test sets. This is a stronger objection than the schema-partition assumption because it challenges the internal validity of the central comparison, not just transfer to real segmented databases. The reader's rationale does mention the same leakage, so my concern is not new, but I would make it the single load-bearing point. A question-only re-run would settle whether the tables change. If no such re-run is available, the paper should be revised to remove golden SQL from inference or explicitly report both selectors. The absence of error bars reinforces the need; the reported gaps are under one point. There are also unexplained discrepancies across tables: Table 1 gives TwoPart BIRD dev EX 49.28, while Table 2 gives Agent-2 53.89 and Table 3 gives CSMA 54.05, suggesting the settings differ in ways not described. The leakage alone is sufficient to keep the reader's rejection, so the verdict should remain unchanged.","tokens_in":10579,"tokens_out":4680,"duration_ms":47401,"concrete_test":"Recompute Table 1 on Spider dev and test and BIRD dev with the Sec 4.3 strategy selector replaced by a question-only selector: for example, always use the decomposed prompt for gpt-3.5-turbo-0125 and the direct prompt for gpt-4, or estimate difficulty from the question text alone. Repeat with at least 5 independent schema partitions and report means and 95% confidence intervals. If TwoPart still matches OneAll within noise under no-golden-SQL selection, the leakage concern is refuted; if the gap flips or shrinks, the cooperative claim is not supported by the paper's evidence.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that TwoPart reaches almost the same level as OneAll and TwoAll. The measurement behind that claim is not clean. In Sec 4.3, the paper says: 'When evaluating the difficulty of question, we treat the structure complexity of its golden SQL as criteria' and then decides whether to use the direct-reasoning or decomposed prompting strategy on that basis. For a test set, this requires access to the ground-truth SQL before prediction. The accuracy values in Table 1 are therefore not an independent estimate of the method's performance on unseen questions; the answer key is used inside the pipeline to choose the prompt. Since both OneAll and TwoPart use the same contaminated selector, the absolute numbers may be inflated, and the differential claim (TwoPart roughly matches OneAll) may be either enhanced or masked by the leakage. No error bars, seeds, or code are given, so the 0.77-point Spider-dev gap and 0.39-point BIRD-dev gap are not shown to be outside sampling noise. The underlying idea is plausible and the ablation of retention, exchange, and checking is a useful first step, but the headline comparison as run does not establish the claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CSMA, a multi-agent LLM framework for text-to-SQL over segmented databases, in which each agent holds only a partial schema and agents cooperate through a shared global schema. The framework has three stages: question-related schema collection, iterative SQL generation, and SQL correctness checking. Experiments on Spider and BIRD compare four internal configurations (OnePart, OneAll, TwoPart, TwoAll) and report that TwoPart reaches nearly the same execution accuracy as OneAll and TwoAll, supporting the paper's central claim that partial-schema agents can compensate for missing schema through cooperation. Ablation studies and a few-shot in-context-learning experiment are also reported.","tokens_in":10801,"tokens_out":5040,"duration_ms":52425,"significance":"If the central claim were supported by a clean evaluation, the idea that partial-schema agents can match full-schema performance would be practically relevant for segmented or privacy-sensitive databases. The three-stage framework is clearly described, and the ablation of retention, exchange, and checking components is a useful first step. However, the main empirical claim is undermined by a test-time leakage in the prompt-selection procedure, and the evaluation lacks external baselines, variance estimates, and reproducibility details. The paper does not currently provide a valid measurement of the claimed cooperative-schema benefit.","major_comments":[{"comment":"The prompt-selection procedure uses the golden SQL: the paper states, 'When evaluating the difficulty of question, we treat the structure complexity of its golden SQL as criteria, including the count of tables involved in the JOIN clause, queries nested in the SQL and so on.' This means the ground-truth answer is an input to the prediction pipeline at test time. The accuracies in Table 1 are therefore not independent estimates of performance on unseen questions, and the differential claim that TwoPart (75.92 EX on Spider dev, 49.28 on BIRD dev) nearly matches OneAll (75.15, 48.89) cannot be interpreted. The authors need to either select the prompting strategy using question-only or predicted-complexity features, or report results without the adaptive selection, before the headline comparison can be evaluated.","section":"Sec. 4.3"},{"comment":"The abstract claims CSMA achieves performance 'comparable to the state-of-the-arts,' but no external state-of-the-art or baseline number is reported. The paper names MAC-SQL as the baseline but never gives its accuracy on the same settings. The main table only compares four internal configurations. A fair evaluation requires the accuracy of at least one existing text-to-SQL method under the same evaluation protocol, or the abstract's comparative claim should be removed.","section":"Sec. 5.1.3 and Table 1"},{"comment":"The evaluation uses a single unseeded random split of the schema into partial-schema pieces, with no error bars or multiple seeds. The observed gaps between TwoPart and OneAll are 0.77 points on Spider dev and 0.39 points on BIRD dev, which are well within the range of sampling noise for these benchmarks. Additionally, the BIRD 'All' accuracy for Agent-2 in Table 2 (53.89) differs from the CSMA row in Table 3 (54.05) even though these appear to describe the same configuration; the discrepancy needs an explanation. The authors should report means and standard deviations over several schema splits and random seeds.","section":"Sec. 5.2 and Tables 1-3"},{"comment":"The schema-partition assumption is restrictive: the experiments split the database into disjoint sets of whole tables, and the global merging rule (keep a table if it appears in one schema, union columns if it appears in both) is lossless only under this table-disjoint ownership. Real segmented databases can involve column-level ownership, overlapping or conflicting schemas, or row-level restrictions, and the paper does not test any of these. The authors should state this limitation explicitly and, ideally, add experiments with column-level splits or overlapping/conflicting schema ownership to delimit the scope of the claim.","section":"Sec. 4.2 and experiments"},{"comment":"The privacy claim is not substantiated. The framework exchanges schema descriptions (table names, column names, and keys) between agents, not database rows, but the abstract says the method 'holding the private data in these individual agents.' No privacy metric, threat model, or leakage analysis is provided. If privacy is a contribution, the paper needs to define what information is protected and empirically measure what an agent can infer about other agents' private schemas or data; otherwise the claim should be weakened to 'data are not directly shared.'","section":"Abstract and Sec. 5"}],"minor_comments":[{"comment":"The setup section says the paper evaluates the test set of BIRD, but Table 1 reports only the BIRD dev set; the inconsistency should be resolved.","section":"Sec. 5.1.1 and Table 1"},{"comment":"There are several typos, including 'benckmark' in the abstract, 'Cooerative' in Section 1, and 'prolems' in Section 2.2; the manuscript needs a careful proofread.","section":"Abstract and Introduction"},{"comment":"The DEC-POMDP formalization in Equations (1)-(5) is not used in the rest of the paper, and some notation is unclear (e.g., the superscript i on s_g^i and the definition of J_s). The authors should either connect this formalism to the actual algorithm or remove it as an unused abstraction.","section":"Sec. 3"},{"comment":"The few-shot experiment does not state which configuration (e.g., TwoPart vs. CSMA) is evaluated, nor does it report variance across examples or seeds; these details should be provided for reproducibility.","section":"Sec. 5.4 and Table 4"},{"comment":"The paper does not report several key hyperparameters and implementation details, including the retention threshold delta, the number of rounds, the exact LLM versions, and the prompts used. Releasing code and prompts would substantially strengthen reproducibility.","section":"Sec. 5.1.3 and Appendix"}],"recommendation":"reject","confidential_remarks":"The test-time use of the golden SQL in Sec. 4.3 is not a minor artifact; it invalidates the primary measurement on which the abstract's claim rests. Even setting that aside, the absence of external baselines, variance estimates, and reproducibility details would require substantial additional experiments. The idea is interesting, so a future version with a decontaminated evaluation and proper baselines might be worth considering, but the current manuscript does not support its headline conclusion."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The distributed-schema setting is the real news here. Prior MAC-SQL agents all share the full schema; DBCopilot selects from a large schema but doesn't model ownership. CSMA's idea - agents each holding a slice, exchanging question-relevant schema through a shared global schema, then generating and checking SQL iteratively - is a legitimate new problem setup, and the Table 1 relative result (TwoPart 75.92 EX vs OneAll 75.15 on Spider dev) is suggestive that cooperation can compensate for missing schema. The ablation in Table 3 gives some evidence that retention, exchange, and checking each contribute. I'd credit that. The soft spot is load-bearing, and it's in the evaluation, not the idea. Section 4.3 says the choice between direct-reasoning and decomposed prompting is made using 'the structure complexity of its golden SQL.' That means the gold answer is an input to the prediction procedure. The numbers in Table 1 are therefore not an independent estimate of performance on unseen questions. Both OneAll and TwoPart use the same contaminated selector, so the relative gap might survive a clean re-run, but as written we can't read either the absolute accuracies or the 'almost the same' claim as established. There are no error bars, no seeds, no code, and the 0.77-point Spider gap / 0.39-point BIRD gap could easily be sampling noise. Other issues are proportionate. The abstract says 'comparable to state-of-the-arts' but there is no SOTA comparison table; the privacy claim ('holding the private data') is not tested - schema partitioning is not the same as privacy, and no privacy experiment exists. The merging rule is lossless only when every table needed by a question is wholly owned by one agent, which is a real assumption for enterprise segmented databases; column-level splits and overlapping ownership are not tested. These are fixable. The DEC-POMDP formalism in Section 3 is decorative and never used in the algorithm, but that's a minor cost. The citation pattern is honest: MAC-SQL and DBCopilot are the right anchors. Who is this for? Someone building multi-agent text-to-SQL over schemas with ownership boundaries would want to know the setting exists, but they shouldn't take the numbers at face value. I'd send it to review rather than desk reject - the problem setup is worth refereeing, and the leakage is fixable by re-running with a complexity oracle or a fixed strategy. But the revision needs a clean evaluation, error bars, and either a SOTA table or a dropped claim.","headline":"The distributed-schema setting is genuinely new, but the headline numbers are contaminated because Section 4.3 uses the golden SQL's complexity to choose the prompting strategy, so the evaluation needs a clean re-run before the claims can be trusted.","tokens_in":11335,"tokens_out":2525,"would_cite":false,"duration_ms":24935,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Schema-partitioned agents match full-schema SQL accuracy","keywords":["text-to-SQL","large language model agents","segmented databases","schema exchange","global schema","multi-agent collaboration","Spider benchmark","BIRD benchmark"],"falsifier":"Split a benchmark schema at the column level rather than the table level, so that no single agent holds all columns of a table a question needs, and run the TwoPart condition: if execution accuracy falls to the OnePart level, the table-disjoint partition is essential to the result.","tokens_in":1314,"feed_emoji":"🤖","tokens_out":1895,"duration_ms":54901,"temperature":0.7,"pith_summary":"The paper proposes CSMA, a framework in which several large-language-model agents, each owning only part of a database schema, cooperate to answer natural-language questions with SQL. The central claim is that this cooperation lets agents with partial schemas reach almost the same execution accuracy as a single agent that sees the whole schema, and the reported numbers on Spider and BIRD support that claim. The point of the setting is that real enterprise databases are often split across departments or permission boundaries, so a method that works without any agent seeing all tables has a privacy and data-governance advantage. The authors argue that their three-stage interaction, schema collection, SQL generation, and correctness checking, is what makes the compensation possible.","feed_headline":"Half-schema agents match full-schema SQL accuracy","feed_subtitle":"Two LLM agents, each holding part of the schema, generate SQL as accurately as one agent with the whole schema.","key_machinery":"The load-bearing mechanism is a shared 'global schema' that agents update in rounds by merging their private schema with the current global schema using a fixed rule: a table present in only one schema is kept as is, and a table present in both keeps the union of its columns. A retention threshold $\\delta$ ensures each extracted table keeps at least $\\delta$ columns, guarding against over-aggressive pruning. The updated global schema becomes the basis for the next agent's SQL generation, and the SQL produced in one round is checked by the next agent, whose positive verdict terminates the loop. This exchange-and-check cycle is what lets schema fragments from different agents combine into a complete picture for a given question.","core_discovery":"The paper's central discovery, stated in Section 5.2, is that the 'TwoPart' condition, two agents each holding half of the tables, performs at the same level as 'OneAll', a single agent holding the complete schema, and even slightly above it on the measured dev sets (75.92 vs 75.15 execution accuracy on Spider dev, 49.28 vs 48.89 on BIRD dev). This is presented as evidence that the agents' partial schemas complement each other through the interaction mechanism, so that no single agent needs access to the whole database to produce accurate SQL. The paper frames this as achieving its intended objective: high performance on text-to-SQL while keeping each agent's private schema local.","pith_inferences":["The table-disjoint evaluation is the paper's core convenience; if the schema were split at the column level or with overlapping ownership, the union-of-columns merge rule could produce incomplete or contradictory schemas, so the method's privacy claim is strongest exactly in the setting it tests.","The same global-schema interaction could be applied to other structured-data tasks, such as cross-silo question answering over distributed knowledge graphs, where the merging rule would need to handle entity alignment rather than table unions.","A direct testable extension would be to log which tables each agent contributes to the global schema and compare the final assembled schema against the question's gold schema; if irrelevant tables enter, the retention mechanism may be under-tuned."],"forward_implications":["If the result holds, text-to-SQL can be run on databases whose schemas are intentionally partitioned across departments, without consolidating all data in one place.","The framework suggests that schema-relevant information exchange, not full schema visibility, is what drives accuracy in LLM-based text-to-SQL.","The retention threshold and union-based merge rule give a concrete recipe for lossless schema assembly from disjoint table sets, a recipe that could transfer to other schema-based reasoning tasks.","The authors' scaling experiment implies a sweet spot: two agents beat one, but adding more agents degrades accuracy, especially on harder benchmarks like BIRD."],"supporting_citations":[{"why":"Supplies the Spider dataset used for the main evaluation.","marker":"Yu et al. (2018)"},{"why":"Supplies the BIRD benchmark, used to test robustness on larger, more complex databases.","marker":"Li et al. (2023)"},{"why":"Provides the MAC-SQL multi-agent baseline and the decomposition-based prompting strategy that CSMA adapts for easy questions.","marker":"Wang et al. (2024)"},{"why":"Supplies the DEC-POMDP formalism that the paper uses to define the multi-agent text-to-SQL setting.","marker":"Spaan et al. (2006)"},{"why":"Provides the execution accuracy (EX) and valid efficiency score (VES) metrics used for evaluation.","marker":"Zhong et al. (2020)"}],"fun_headline_variants":["Split-schema LLM agents match full-schema SQL accuracy","Cooperative agents with partial schemas match full-schema SQL","Two half-schema agents equal one full-schema agent on SQL","Split-schema cooperation yields full-schema SQL performance","Partial-schema agents cooperate to match whole-schema SQL"],"cache_read_input_tokens":13440,"weakest_assumption_plain":"The evaluation assumes that the database schema splits cleanly into table-disjoint pieces, so that every table and all the columns needed to answer a question are wholly owned by a single agent, making the merge rule lossless.","fun_headline_variants_meta":{"raw":{"variants":["Split-schema LLM agents match full-schema SQL accuracy","Cooperative agents with partial schemas match full-schema SQL","Two half-schema agents equal one full-schema agent on SQL","Split-schema cooperation yields full-schema SQL performance","Partial-schema agents cooperate to match whole-schema SQL"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000592,"raw_usage":{"total_tokens":2752,"prompt_tokens":896,"completion_tokens":1856,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":512,"completion_tokens_details":{"reasoning_tokens":1772}},"tokens_in":512,"tokens_out":1856,"duration_ms":13610,"temperature":1.0,"reasoning_tokens":1772,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:17:33.667940+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Split a benchmark schema at the column level rather than the table level, so that no single agent holds all columns of a table a question needs, and run the TwoPart condition: if execution accuracy falls to the OnePart level, the table-disjoint partition is essential to the result.","supporting_citations":[{"cited_title":"In: Adaptive Agents and Multi-Agent Systems (2006)","cited_arxiv_id":null,"evidence_quote":"Supplies the DEC-POMDP formalism that the paper uses to define the multi-agent text-to-SQL setting."}],"review_version":1}