REVIEW 5 major objections 5 minor 1 cited by
Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents
T0 review · 5 major / 5 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read Schema-partitioned agents match full-schema SQL accuracy
desk verdict The distributed-schema setting is genuinely new, but the headline numbers are contaminated because Section 4.3 uses the golden SQL's complexity to choose the prompting strategy, so the evaluation needs a clean re-run before the claims can be trusted. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing mechanism is a shared 'global schema' that agents update in rounds by merging their private schema with the current global schema using a fixed rule: a table present in only one schema is kept as is, and a table present in both keeps the union of its columns. A retention threshold $\delta$ ensures each extracted table keeps at least $\delta$ columns, guarding against over-aggressive pruning. The updated global schema becomes the basis for the next agent's SQL generation, and the SQL produced in one round is checked by the next agent, whose positive verdict terminates the loop. This exchange-and-check cycle is what lets schema fragments from different agents combine into a complete picture for a given question.
What would settle it
Split a benchmark schema at the column level rather than the table level, so that no single agent holds all columns of a table a question needs, and run the TwoPart condition: if execution accuracy falls to the OnePart level, the table-disjoint partition is essential to the result.
Extended reading notes
Core claim
The paper's central discovery, stated in Section 5.2, is that the 'TwoPart' condition, two agents each holding half of the tables, performs at the same level as 'OneAll', a single agent holding the complete schema, and even slightly above it on the measured dev sets (75.92 vs 75.15 execution accuracy on Spider dev, 49.28 vs 48.89 on BIRD dev). This is presented as evidence that the agents' partial schemas complement each other through the interaction mechanism, so that no single agent needs access to the whole database to produce accurate SQL. The paper frames this as achieving its intended objective: high performance on text-to-SQL while keeping each agent's private schema local.
Load-bearing premise
The evaluation assumes that the database schema splits cleanly into table-disjoint pieces, so that every table and all the columns needed to answer a question are wholly owned by a single agent, making the merge rule lossless.
Editorial extensions
If this is right
- If the result holds, text-to-SQL can be run on databases whose schemas are intentionally partitioned across departments, without consolidating all data in one place.
- The framework suggests that schema-relevant information exchange, not full schema visibility, is what drives accuracy in LLM-based text-to-SQL.
- The retention threshold and union-based merge rule give a concrete recipe for lossless schema assembly from disjoint table sets, a recipe that could transfer to other schema-based reasoning tasks.
- The authors' scaling experiment implies a sweet spot: two agents beat one, but adding more agents degrades accuracy, especially on harder benchmarks like BIRD.
Reading between the lines
- The table-disjoint evaluation is the paper's core convenience; if the schema were split at the column level or with overlapping ownership, the union-of-columns merge rule could produce incomplete or contradictory schemas, so the method's privacy claim is strongest exactly in the setting it tests.
- The same global-schema interaction could be applied to other structured-data tasks, such as cross-silo question answering over distributed knowledge graphs, where the merging rule would need to handle entity alignment rather than table unions.
- A direct testable extension would be to log which tables each agent contributes to the global schema and compare the final assembled schema against the question's gold schema; if irrelevant tables enter, the retention mechanism may be under-tuned.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes CSMA, a multi-agent LLM framework for text-to-SQL over segmented databases, in which each agent holds only a partial schema and agents cooperate through a shared global schema. The framework has three stages: question-related schema collection, iterative SQL generation, and SQL correctness checking. Experiments on Spider and BIRD compare four internal configurations (OnePart, OneAll, TwoPart, TwoAll) and report that TwoPart reaches nearly the same execution accuracy as OneAll and TwoAll, supporting the paper's central claim that partial-schema agents can compensate for missing schema through cooperation. Ablation studies and a few-shot in-context-learning experiment are also reported.
Significance. If the central claim were supported by a clean evaluation, the idea that partial-schema agents can match full-schema performance would be practically relevant for segmented or privacy-sensitive databases. The three-stage framework is clearly described, and the ablation of retention, exchange, and checking components is a useful first step. However, the main empirical claim is undermined by a test-time leakage in the prompt-selection procedure, and the evaluation lacks external baselines, variance estimates, and reproducibility details. The paper does not currently provide a valid measurement of the claimed cooperative-schema benefit.
major comments (5)
- [Sec. 4.3] The prompt-selection procedure uses the golden SQL: the paper states, 'When evaluating the difficulty of question, we treat the structure complexity of its golden SQL as criteria, including the count of tables involved in the JOIN clause, queries nested in the SQL and so on.' This means the ground-truth answer is an input to the prediction pipeline at test time. The accuracies in Table 1 are therefore not independent estimates of performance on unseen questions, and the differential claim that TwoPart (75.92 EX on Spider dev, 49.28 on BIRD dev) nearly matches OneAll (75.15, 48.89) cannot be interpreted. The authors need to either select the prompting strategy using question-only or predicted-complexity features, or report results without the adaptive selection, before the headline comparison can be evaluated.
- [Sec. 5.1.3 and Table 1] The abstract claims CSMA achieves performance 'comparable to the state-of-the-arts,' but no external state-of-the-art or baseline number is reported. The paper names MAC-SQL as the baseline but never gives its accuracy on the same settings. The main table only compares four internal configurations. A fair evaluation requires the accuracy of at least one existing text-to-SQL method under the same evaluation protocol, or the abstract's comparative claim should be removed.
- [Sec. 5.2 and Tables 1-3] The evaluation uses a single unseeded random split of the schema into partial-schema pieces, with no error bars or multiple seeds. The observed gaps between TwoPart and OneAll are 0.77 points on Spider dev and 0.39 points on BIRD dev, which are well within the range of sampling noise for these benchmarks. Additionally, the BIRD 'All' accuracy for Agent-2 in Table 2 (53.89) differs from the CSMA row in Table 3 (54.05) even though these appear to describe the same configuration; the discrepancy needs an explanation. The authors should report means and standard deviations over several schema splits and random seeds.
- [Sec. 4.2 and experiments] The schema-partition assumption is restrictive: the experiments split the database into disjoint sets of whole tables, and the global merging rule (keep a table if it appears in one schema, union columns if it appears in both) is lossless only under this table-disjoint ownership. Real segmented databases can involve column-level ownership, overlapping or conflicting schemas, or row-level restrictions, and the paper does not test any of these. The authors should state this limitation explicitly and, ideally, add experiments with column-level splits or overlapping/conflicting schema ownership to delimit the scope of the claim.
- [Abstract and Sec. 5] The privacy claim is not substantiated. The framework exchanges schema descriptions (table names, column names, and keys) between agents, not database rows, but the abstract says the method 'holding the private data in these individual agents.' No privacy metric, threat model, or leakage analysis is provided. If privacy is a contribution, the paper needs to define what information is protected and empirically measure what an agent can infer about other agents' private schemas or data; otherwise the claim should be weakened to 'data are not directly shared.'
minor comments (5)
- [Sec. 5.1.1 and Table 1] The setup section says the paper evaluates the test set of BIRD, but Table 1 reports only the BIRD dev set; the inconsistency should be resolved.
- [Abstract and Introduction] There are several typos, including 'benckmark' in the abstract, 'Cooerative' in Section 1, and 'prolems' in Section 2.2; the manuscript needs a careful proofread.
- [Sec. 3] The DEC-POMDP formalization in Equations (1)-(5) is not used in the rest of the paper, and some notation is unclear (e.g., the superscript i on s_g^i and the definition of J_s). The authors should either connect this formalism to the actual algorithm or remove it as an unused abstraction.
- [Sec. 5.4 and Table 4] The few-shot experiment does not state which configuration (e.g., TwoPart vs. CSMA) is evaluated, nor does it report variance across examples or seeds; these details should be provided for reproducibility.
- [Sec. 5.1.3 and Appendix] The paper does not report several key hyperparameters and implementation details, including the retention threshold delta, the number of rounds, the exact LLM versions, and the prompts used. Releasing code and prompts would substantially strengthen reproducibility.
Circularity Check
Sec. 4.3 selects the SQL-generation prompt using the golden SQL's structural complexity, so the headline TwoPart-versus-OneAll accuracy comparison in Table 1 is contaminated by label information and does not independently support the cooperative-schema claim.
-
self definitional
[Sec. 4.3 (Question-corresponding SQL Generation); affects Tables 1-3]
"Choosing between them mainly depends on the difficulty of question and the capability of LLM. When evaluating the difficulty of question, we treat the structure complexity of its golden SQL as criteria, including the count of tables involved in the ”JOIN” clause, queries nested in the SQL and so on."
The method's choice between direct-reasoning and decomposed prompting strategies is a function of the ground-truth SQL Y*. The predicted SQL is therefore generated by a pipeline whose inputs include structural features of the target answer. On dev/test examples, the answer key participates in constructing the prediction, so the accuracies in Table 1 and the central claim that TwoPart reaches almost the same level as OneAll/TwoAll are not an independent measurement. The leak is shared across settings, but it can interact differently with each agent configuration, and no error bars or seeds are given to bound the effect.
full rationale
The paper's central empirical claim is that agents with partial schemas can cooperate to match a full-schema agent. The evidence for this claim is Table 1, which reports accuracies obtained under a selection rule that uses the golden SQL's structural complexity to choose the prompting strategy. This is a direct leakage of the target into the prediction procedure: the answer key is an input to the method, so the reported absolute accuracies are not an independent estimate of performance on unseen questions. The relative comparison between TwoPart and OneAll may be less directly forced, but it is still measured inside the same contaminated pipeline, and no significance testing or code is provided to separate the cooperative effect from the leaked selection signal. No other circularity patterns are present: the self-citations in the paper are not load-bearing, there are no imported uniqueness theorems, and the framework's components are not renamed versions of a prior result. The core idea is plausible and the ablation is useful as a first step, but the headline validation reduces to using the golden answer to guide the generation, warranting a high circularity score rather than a clean bill of health.
Assumptions & free parameters
free parameters (3)
- retention threshold delta =
not reported
- number of agents n =
2
- prompt strategy selection =
per-question choice from golden SQL structure
assumptions (5)
- domain assumption An LLM agent can reliably extract question-relevant schema, generate SQL, and judge SQL correctness from its current schema.
- domain assumption Schema merging by keeping tables present in either schema and taking the union of columns for shared tables is lossless for the target question.
- ad hoc to paper The structure complexity of the golden SQL is an available and legitimate criterion for choosing the prompting strategy.
- domain assumption Agents' LLM-based checking verdicts are a reliable stopping criterion.
- domain assumption Sharing the global schema across agents does not violate the privacy of the individual agents' data.
Cite this review
Pith. "Pith review of Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents." pith.science (2026). https://pith.science/paper/XFNLCOPA
@misc{pith2026241205850,
author = {Pith},
title = {Pith review of: Cooperative SQL Generation for Segmented Databases By Using Multi-functional LLM Agents},
year = {2026},
howpublished = {\url{https://pith.science/paper/XFNLCOPA}},
note = {Machine review of arXiv:2412.05850}
}
read the original abstract
Text-to-SQL task aims to automatically yield SQL queries according to user text questions. To address this problem, we propose a Cooperative SQL Generation framework based on Multi-functional Agents (CSMA) through information interaction among large language model (LLM) based agents who own part of the database schema seperately. Inspired by the collaboration in human teamwork, CSMA consists of three stages: 1) Question-related schema collection, 2) Question-corresponding SQL query generation, and 3) SQL query correctness check. In the first stage, agents analyze their respective schema and communicate with each other to collect the schema information relevant to the question. In the second stage, agents try to generate the corresponding SQL query for the question using the collected information. In the third stage, agents check if the SQL query is created correctly according to their known information. This interaction-based method makes the question-relevant part of database schema from each agent to be used for SQL generation and check. Experiments on the Spider and Bird benckmark demonstrate that CSMA achieves a high performance level comparable to the state-of-the-arts, meanwhile holding the private data in these individual agents.
Forward citations
Cited by 1 Pith paper
-
Agentic LLMs for Question Answering over Tabular Data
A five-stage NL-to-SQL pipeline with GPT-4o achieves 70.5% on DataBench QA and 71.6% on DataBench Lite QA, beating baselines of 26% and 27%.
Reference graph
Works this paper leans on
-
[1]
https://arxiv.org/abs/1905.06241
Bogin, B., Gardner, M., Berant, J.: Representing Schema Structure with Graph Neural Networks for Text-to-SQL Parsing (2019). https://arxiv.org/abs/1905.06241
arXiv 2019
-
[2]
https:// arxiv.org/abs/2106.01093
Cao, R., Chen, L., Chen, Z., Zhao, Y., Zhu, S., Yu, K.: LGESQL: Line Graph Enhanced Text-to-SQL Model with Mixed Local and Non-Local Relations (2021). https:// arxiv.org/abs/2106.01093
arXiv 2021
-
[3]
ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser
Chen, Z., Chen, L., Zhao, Y., Cao, R., Xu, Z., Zhu, S., Yu, K.: ShadowGNN: Graph Projection Neural Network for Text-to-SQL Parser (2021). https://arxiv.org/abs/ 2104.04689
work page Pith review arXiv 2021
-
[4]
https://arxiv.org/abs/2004.03125
Choi, D., Shin, M.C., Kim, E., Shin, D.R.: RYANSQL: Recursively Applying Sketch- based Slot Fillings for Complex Text-to-SQL in Cross-Domain Databases (2020). https://arxiv.org/abs/2004.03125
arXiv 2020
-
[5]
https://arxiv.org/abs/ 1810.04805
Devlin, J., Chang, M.-W., Lee, K., Toutanova, K.: BERT: Pre-training of Deep Bidi- rectional Transformers for Language Understanding (2019). https://arxiv.org/abs/ 1810.04805
arXiv 2019
-
[6]
https://arxiv.org/abs/1905.08205
Guo, J., Zhan, Z., Gao, Y., Xiao, Y., Lou, J.-G., Liu, T., Zhang, D.: Towards Complex Text-to-SQL in Cross-Domain Database with Intermediate Representation (2019). https://arxiv.org/abs/1905.08205
arXiv 2019
-
[7]
https: //arxiv.org/abs/2201.07207
Huang, W., Abbeel, P., Pathak, D., Mordatch, I.: Language Models as Zero-Shot Planners: Extracting Actionable Knowledge for Embodied Agents (2022). https: //arxiv.org/abs/2201.07207
arXiv 2022
-
[8]
https://arxiv.org/abs/2203.06958
Hui, B., Geng, R., Wang, L., Qin, B., Li, B., Sun, J., Li, Y.: S 2SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder for Text-to-SQL Parsers (2022). https://arxiv.org/abs/2203.06958
arXiv 2022
Show all 32 references
-
[9]
https://arxiv.org/ abs/2002.00557
Kelkar, A., Relan, R., Bhardwaj, V., Vaichal, S., Khatri, C., Relan, P.: Bertrand-DR: Improving Text-to-SQL using a Discriminative Re-ranker (2020). https://arxiv.org/ abs/2002.00557
2020 arXiv
-
[10]
https://arxiv.org/abs/2305.03111
Li, J., Hui, B., Qu, G., Yang, J., Li, B., Li, B., Wang, B., Qin, B., Cao, R., Geng, R., Huo, N., Zhou, X., Ma, C., Li, G., Chang, K.C.C., Huang, F., Cheng, R., Li, Y.: Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs (2...
2023 arXiv
-
[11]
https://arxiv.org/abs/2303.13547
Liu, A., Hu, X., Wen, L., Yu, P.S.: A comprehensive evaluation of ChatGPT’s zero-shot Text-to-SQL capability (2023). https://arxiv.org/abs/2303.13547
2023 arXiv
-
[12]
https: //arxiv.org/abs/1910.13461
Lewis, M., Liu, Y., Goyal, N., Ghazvininejad, M., Mohamed, A., Levy, O., Stoy- anov, V., Zettlemoyer, L.: BART: Denoising Sequence-to-Sequence Pre-training 14 for Natural Language Generation, Translation, and Comprehension (2019). https: //arxiv.org/abs/1910.13461
2019 arXiv
-
[13]
https://arxiv.org/abs/2304.11015
Pourreza, M., Rafiei, D.: DIN-SQL: Decomposed In-Context Learning of Text-to-SQL with Self-Correction (2023). https://arxiv.org/abs/2304.11015
2023 arXiv
-
[14]
https://arxiv.org/abs/2208.13629
Qin, B., Hui, B., Wang, L., Yang, M., Li, J., Li, B., Geng, R., Cao, R., Sun, J., Si, L., Huang, F., Li, Y.: A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions (2022). https://arxiv.org/abs/2208.13629
2022 arXiv
-
[15]
https://arxiv.org/abs/1910.10683
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., Liu, P.J.: Exploring the Limits of Transfer Learning with a Unified Text-to-Text Transformer (2023). https://arxiv.org/abs/1910.10683
2023 arXiv
-
[16]
In: Adaptive Agents and Multi-Agent Systems (2006)
Spaan, M.T.J., Gordon, G.J., Vlassis, N.A.: Decentralized planning under uncertainty for teams of communicating agents. In: Adaptive Agents and Multi-Agent Systems (2006). https://api.semanticscholar.org/CorpusID:1751957
2006
-
[17]
https://arxiv.org/abs/2212.04088
Song, C.H., Wu, J., Washington, C., Sadler, B.M., Chao, W.-L., Su, Y.: LLM-Planner: Few-Shot Grounded Planning for Embodied Agents with Large Language Models (2023). https://arxiv.org/abs/2212.04088
2023 arXiv
-
[18]
https://arxiv.org/abs/2404.10179
Wong, N., York, S., Young, N.: Scaling Instructable Agents Across Many Simulated Worlds (2024). https://arxiv.org/abs/2404.10179
2024 arXiv
-
[19]
https://arxiv.org/abs/ 2312.03463
Wang, T., Lin, H., Han, X., Sun, L., Chen, X., Wang, H., Zeng, Z.: DBCopilot: Scaling Natural Language Querying to Massive Databases (2024). https://arxiv.org/abs/ 2312.03463
2024 arXiv
-
[20]
https://arxiv.org/abs/2312.11242 15
Yin, D., Sun, X., Li, Z.: MAC-SQL: A Multi-Agent Collaborative Framework for Text-to-SQL (2024). https://arxiv.org/abs/2312.11242 15
2024 arXiv
-
[21]
https://arxiv.org/abs/1804.09769
Yu, T., Li, Z., Zhang, Z., Zhang, R., Radev, D.: TypeSQL: Knowledge-based Type- Aware Neural Text-to-SQL Generation (2018). https://arxiv.org/abs/1804.09769
2018 arXiv
-
[22]
https://arxiv.org/abs/2005.08314
Yin, P., Neubig, G., Yih, W.-t., Riedel, S.: TaBERT: Pretraining for Joint Under- standing of Textual and Tabular Data (2020). https://arxiv.org/abs/2005.08314
2020 arXiv
-
[23]
https://arxiv.org/abs/2009.13845
Xiong, C.: GraPPa: Grammar-Augmented Pre-Training for Table Semantic Parsing (2021). https://arxiv.org/abs/2009.13845
2021 arXiv
-
[24]
https://arxiv.org/abs/1810.05237
Yu, T., Yasunaga, M., Yang, K., Zhang, R., Wang, D., Li, Z., Radev, D.: Syn- taxSQLNet: Syntax Tree Networks for Complex and Cross-DomainText-to-SQL Task (2018). https://arxiv.org/abs/1810.05237
2018 arXiv
-
[25]
(eds.) Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp
Chiang, D., Hockenmaier, J., Tsujii, J. (eds.) Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, pp. 3911–3921. Association for Computational Linguistics, Brussels, Belgium (2018). https://doi.org/10.18653/ v1/D18-1425 . https://aclantholo...
2018
-
[26]
IEEE Transactions on Big Data 9(01), 118–132 (2023)
Zhang, Y., An, R., Liu, S., Cui, J., Shang, X.: Predicting and understanding stu- dent learning performance using multi-source sparse attention convolutional neural networks. IEEE Transactions on Big Data 9(01), 118–132 (2023)
2023
-
[27]
https://arxiv.org/abs/2307.02485
Zhang, H., Du, W., Shan, J., Zhou, Q., Du, Y., Tenenbaum, J.B., Shu, T., Gan, C.: Building Cooperative Embodied Agents Modularly with Large Language Models (2024). https://arxiv.org/abs/2307.02485
2024 arXiv
-
[28]
Knowledge-Based Systems 205, 106290 (2020)
Zhang, Y., Dai, H., Yun, Y., Liu, S., Lan, A., Shang, X.: Meta-knowledge dictionary learning on 1-bit response data for student knowledge diagnosis. Knowledge-Based Systems 205, 106290 (2020)
2020
-
[29]
Neural Computing and Applications, 1–14 (2022)
Zhang, Y., Liu, S., Qu, X., Shang, X.: Multi-instance discriminative contrastive learning for brain image representation. Neural Computing and Applications, 1–14 (2022)
2022
-
[30]
IEEE Transactions on Neural Networks and Learning Systems (2023)
Zhang, Y., Wang, Y., Li, Y., Xu, Y., Wei, S., Liu, S., Shang, X.: Federated discrimina- tive representation learning for image classification. IEEE Transactions on Neural Networks and Learning Systems (2023)
2023
-
[31]
https://arxiv.org/abs/1909.00786
Zhang, R., Yu, T., Er, H.Y., Shim, S., Xue, E., Lin, X.V., Shi, T., Xiong, C., Socher, R., Radev, D.: Editing-Based SQL Query Generation for Cross-Domain Context- Dependent Questions (2019). https://arxiv.org/abs/1909.00786
2019 arXiv
-
[32]
https://arxiv.org/abs/2010.02840 17
Zhong, R., Yu, T., Klein, D.: Semantic Evaluation for Text-to-SQL with Distilled Test 16 Suites (2020). https://arxiv.org/abs/2010.02840 17
2020 arXiv
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.