REVIEW 4 major objections 5 minor 37 references
Towards Scalable Schema Mapping using Large Language Models
T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read With careful sampling and bidirectional merging, an open-source 70B LLM rivals a GPT-4-based system at schema alignment.
desk verdict Nice prompt-augmentation idea for LLM schema matching, but the headline GPT-4 parity claim rests on an uncontrolled comparison. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central mechanism is a bidirectional stable matching over LLM outputs. Starting from an N-1 prompt that serializes N source attributes against one target attribute in JSON, the pipeline permutes column order, resamples data values, and swaps the source and target schemas to obtain multiple samples; it then asks the model to rank candidate matches, converts output logits into confidence scores, and applies the stable matching algorithm to the ranked lists from both directions. Majority-vote or multiplicative merging of the two directions produces the final candidate set, with data-type prefiltering shrinking the comparison pool before any LLM call.
What would settle it
Run the exact bidirectional prompt-sampling pipeline on a benchmark with gold GLaV/FRD mappings, such as the Amalgam-derived seven-rule setup used in the paper, and measure table and join overlap between predicted and gold target instances; if swapping the source and target roles changes the generated rule set materially or the pipeline's F1 is no better than a single well-chosen prompt, the scalability claim for full mappings fails.
Extended reading notes
Core claim
On its own terms, the paper's discovery is that careful sampling and aggregation around a well-structured N-1 prompt can make a smaller open-source model highly competitive with a larger proprietary one for schema alignment. The key comparison is bidirectional stable matching: match attributes in both directions by swapping which schema plays source and target, rank candidates with logit-derived confidence scores, and merge the two ranked lists with a stable matching algorithm. With this design, the open-source model beats the reported accuracy of a GPT-4 pipeline on the MIMIC dataset and matches it on Synthea. The paper explicitly limits the claim: the evidence is for atomic attribute-pair alignments, and extending the sampling techniques to full GLaV/FRD rules is not validated, since logical equivalence of such rules is undecidable in general.
Load-bearing premise
The load-bearing premise is that gains measured on pairwise attribute alignment transfer to full multi-relation mapping rules, but the paper validates only the pairwise case and itself notes that rule equivalence is undecidable, so the full-mapping claim rests on an extrapolation.
Editorial extensions
If this is right
- If the alignment results hold, organizations can run schema matching on an open-source model they host themselves, avoiding sending schemas and sample data to a proprietary API.
- The N-1 prompt with attribute descriptions provides the largest performance boost, so enriching schemas with natural-language metadata is likely to matter more than scaling model size for matching tasks.
- Data-type prefiltering and chunking high-overlap rules together cut token use by more than half, which lowers the cost of LLM-based integration at scale.
- Because candidate sets are intended for human validation, the practical contribution is a better starting point for the human-in-the-loop mapping process, not a fully automated mapping system.
Reading between the lines
- A testable extension is to apply the same symmetric-transformation sampling to other structured prediction problems whose outputs are sets of atomic, order-insensitive pairs, such as entity resolution or column-level data discovery.
- The paper's alignment-only evaluation suggests the title's 'schema mapping' claim is really established for schema alignment; a reader should expect the full GLaV/FRD case to need new output representations before scaling conclusions transfer.
- The confidence from logits plus stable matching could possibly be replaced by a cheaper heuristic such as co-occurrence frequency across samples and still capture much of the benefit, which would further reduce cost.
- If the trend holds, the bottleneck in LLM-based data integration shifts from model quality to prompt and aggregation engineering, making the field more about pipeline design than about chasing the newest model.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper identifies three challenges for LLM-based schema mapping—output inconsistency, limited context for expressive (GLaV/FRD) mappings, and high computational cost—and proposes solutions: sampling with symmetric transformations and aggregation, bidirectional schema matching with confidence scoring and stable matching, and rule chunking. The evaluation on the MIMIC-OMOP and Synthea-OMOP datasets compares a Llama-3.1-70B model against COMA, Parciak et al., and MatchMaker, reporting precision/recall/F1 and accuracy@1. The authors claim that with careful prompt design, open-source LLMs can be competitive with GPT-4-based MatchMaker. A preliminary Amalgam experiment examines FRD mapping under different chunking settings. The paper explicitly limits the main experimental results to schema alignment and acknowledges in Section 3.3.2 that extending the sampling techniques to full FRD mappings is future work and is theoretically problematic for non-invertible rules.
Significance. If the comparison were properly controlled, the paper would provide a valuable data point that a 70B open-source model with prompt variation, structured output, and aggregation can reach alignment quality comparable to a proprietary model on two medical benchmarks. The formalization of the schema mapping problem and the honest discussion of why sampling is harder for mappings are useful contributions. However, the central claim currently rests on an uncontrolled comparison (Section 3.3.1/Table 3) and is supported only for attribute-pair alignment, not for schema mapping as defined in Section 2. The paper is commendably explicit about these limitations, but the title and abstract overstate the achieved scope.
major comments (4)
- [§3.3/Table 3] The headline claim that open-source Llama-3.1-70B is competitive with GPT-4 rests on an uncontrolled comparison. Section 3.3 (Baselines) states that the authors' MatchMaker re-implementation excludes the pre-filtering step 'for a fair comparison,' yet Section 3.3.1 and Table 3 compare the proposed method against the original MatchMaker's reported accuracy@1, i.e., the full pipeline with pre-filtering. These are different system configurations. In addition, the datasets are not shown to be identical: the paper reports populating OMOP with sample data from MIMIC-IV, whereas the original MatchMaker's MIMIC-OMOP numbers may be based on MIMIC-III and different schema pairs or ground truth. Unless the authors verify identical splits, data values, and pipeline configuration, the claimed parity/superiority over GPT-4 is not established.
- [Tables 1–3] The metric used for the central comparison is not consistent with the rest of the evaluation. Table 3 reports 'Accuracy@1' values of 0.78 (MIMIC) and 0.69 (Synthea) for Bidirectional (Stable Matching), while Tables 1 and 2 report F1@1 values of 0.64 and 0.60 for the same method. The Accuracy@1 metric is never defined in the Metrics paragraph of Section 3.3, and the relationship between Accuracy@1 and F1@1 is not explained. Moreover, Table 3 lists MatchMaker's values as 62.20±2.40 and 70.20±1.70 (percentages) alongside the proposed method's values as 0.78 and 0.69 (fractions), making the units incomparable as printed. This directly affects the validity of the headline comparison and must be corrected.
- [§4 and §3.3.2] The paper's title and abstract promise progress on schema mapping, but the main evaluation is exclusively on schema alignment (attribute-pair matching). Section 3.3.2 explicitly concedes that the sampling and bidirectional techniques are 'incorrect in a theoretical sense' for non-invertible FRD rules and that extending them to full mappings is future work. The Amalgam experiments in Section 4 do not apply the proposed sampling/aggregation methods: the text states that outputs from the 20 seeds are not combined, and that combining is 'saved for future works.' Therefore the evidence does not support a claim about scalability of schema mapping; it supports a claim about alignment. The paper should either reframe its scope as alignment with mapping as a vision, or add experiments that evaluate the proposed pipeline on FRD mappings.
- [§3.3 and §4.3] The paper defers essential technical details to an external technical report, including the exact prompt templates, the confidence-score computation, the stable matching parameterization, the evaluation seeds, and the aggregation results. For example, Section 3.2 says 'for more details on our prompts, how we compute the confidence score, and the stable matching algorithm, please refer to the technical report,' and Section 4.3 says 'More details on the evaluation setup can be found in our technical report.' Without these details, the reported results are not independently reproducible from the manuscript alone. Please include the core details (or an appendix) so that the method can be understood and reimplemented from the paper.
minor comments (5)
- [Figure 3] Figure 3 is not readable in the manuscript: the axis labels and legend contain literal '/uni00000014' escape sequences, so the claimed trend of decreasing recall with MRPP cannot be verified from the figure.
- [Tables 1–2] Tables 1 and 2 report averages over three seeds without any variance or significance test; given that several F1@1 values differ by 0.01–0.02, please add standard deviations or confidence intervals to support the qualitative comparisons.
- [§3.3.1] Section 3.3 says 'The results in this paper use the majority vote aggregation method,' but Tables 1 and 2 label the method simply as 'Aggregation'; specify the aggregation function in the table or its caption.
- [Tables 4–5] Tables 4 and 5 are not directly comparable as printed: Table 4 reports per-prompt token counts while Table 5 appears to report totals over the number of prompts needed to cover all seven rules. Clarify the units and how the totals were computed.
- [Throughout] There are several typographical errors, e.g., 'do to' in Section 3.3.2, 'on the other head' in Section 4.2, 'It would tedious' in Section 5.1, and 'imprecisely speaking' in Section 6; these should be corrected.
Circularity Check
No significant circularity: the paper's central empirical comparison is against external benchmarks and its self-citations are not load-bearing.
full rationale
The manuscript does not present a derivation chain that reduces to its own inputs. The three proposed techniques (sampling and aggregation, bidirectional matching, and token-efficiency chunking) are evaluated on MIMIC-OMOP, Synthea-OMOP, and Amalgam using external ground truth, so the reported accuracies are not fitted parameters renamed as predictions. The only self-references are to the authors' technical report for prompt and stable-matching details and to their prior NIH project; neither defines the target quantities nor justifies the headline parity claim, so these are normal citations rather than load-bearing circularity. Table 3 compares the Llama-3.1-70B pipeline against MatchMaker's reported accuracy@1 from the original paper, which is an external comparison rather than a self-derived result. A remaining correctness risk is that Table 3 appears to mix the authors' own MatchMaker reimplementation (which excludes pre-filtering) with the original full-pipeline reported numbers, and it mixes percentage and fractional units; however, this is an experimental-control concern, not a circularity of the kind defined by the reviewer instructions.
Assumptions & free parameters
free parameters (3)
- Max. Rules per Prompt (MRPP) =
1 to 7
- Number of prompt samples =
3
- Top-K in stable matching
assumptions (4)
- domain assumption LLM outputs sampled via different prompt permutations are statistically informative about the true set of alignments
- domain assumption Schema alignment is symmetric: (A,B) is identical to (B,A), so swapping source and target is a valid transformation
- domain assumption Execution accuracy overlap on a generated 100-row target instance is a valid proxy for mapping correctness
- standard math Logical equivalence of queries is undecidable
Cite this review
Pith. "Pith review of Towards Scalable Schema Mapping using Large Language Models." pith.science (2026). https://pith.science/paper/HISRMVWD
@misc{pith2026250524716,
author = {Pith},
title = {Pith review of: Towards Scalable Schema Mapping using Large Language Models},
year = {2026},
howpublished = {\url{https://pith.science/paper/HISRMVWD}},
note = {Machine review of arXiv:2505.24716}
}
read the original abstract
The growing need to integrate information from a large number of diverse sources poses significant scalability challenges for data integration systems. These systems often rely on manually written schema mappings, which are complex, source-specific, and costly to maintain as sources evolve. While recent advances suggest that large language models (LLMs) can assist in automating schema matching by leveraging both structural and natural language cues, key challenges remain. In this paper, we identify three core issues with using LLMs for schema mapping: (1) inconsistent outputs due to sensitivity to input phrasing and structure, which we propose methods to address through sampling and aggregation techniques; (2) the need for more expressive mappings (e.g., GLaV), which strain the limited context windows of LLMs; and (3) the computational cost of repeated LLM calls, which we propose to mitigate through strategies like data type prefiltering.
Figures
Reference graph
Works this paper leans on
-
[1]
Serge Abiteboul, Richard Hull, and Victor Vianu. 1995. Foundations of databases. Vol. 8. Addison-Wesley Reading
work page 1995
-
[2]
Menatalla Abououf, Shakti Singh, Hadi Otrok, Rabeb Mizouni, and Anis Ouali
-
[3]
Ted T. Ashburn and Karl B. Thor. 2004. Drug repositioning: identifying and developing new uses for existing drugs. Nature Reviews Drug Discovery 3, 8 (2004), 673–683
work page 2004
-
[4]
David Aumueller, Hong-Hai Do, Sabine Massmann, and Erhard Rahm. 2005. Schema and ontology matching with COMA++. In Proceedings of the 2005 ACM SIGMOD International Conference on Management of Data (Baltimore, Mary- land) (SIGMOD ’05). Association for Computing Machinery, New York, NY, USA, 906–908. doi:10.1145/1066157.1066283
arXiv 2005
-
[5]
Angela Bonifati, Giansalvatore Mecca, Alessandro Pappalardo, Salvatore Raunich, and Gianvito Summa. 2008. Schema mapping verification: the spicy way. In Proceedings of the 11th international conference on Extending database technology: Advances in database technology . 85–96
work page 2008
-
[6]
Xinyun Chen, Renat Aksitov, Uri Alon, Jie Ren, Kefan Xiao, Pengcheng Yin, Sushant Prakash, Charles Sutton, Xuezhi Wang, and Denny Zhou. 2023. Universal Self-Consistency for Large Language Model Generation. arXiv:2311.17311 [cs.CL] https://arxiv.org/abs/2311.17311
arXiv 2023
-
[7]
Hong-Hai Do and Erhard Rahm. 2002. Chapter 53 - COMA — A system for flexible combination of schema matching approaches. In VLDB ’02: Proceedings of the 28th International Conference on Very Large Databases , Philip A. Bernstein, Yannis E. Ioannidis, Raghu Ramakrishnan, and Dimitris Papadias (Eds.). Morgan Kaufmann, San Francisco, 610–621. doi:10.1016/B978...
-
[8]
AnHai Doan, Alon Halevy, and Zachary Ives. 2012. Principles of Data Integration (1st ed.). Morgan Kaufmann Publishers Inc., San Francisco, CA, USA
work page 2012
Show all 37 references
-
[9]
Ronald Fagin, Phokion G Kolaitis, Renée J Miller, and Lucian Popa. 2005. Data exchange: semantics and query answering. Theoretical Computer Science 336, 1 (2005), 89–124
2005
-
[10]
National Science Foundation and National Institutes of Health. 2021. Smart Health and Biomedical Research in the Era of Artificial Intelligence and Advanced Data Science (SCH). https://www.nsf.gov/pubs/2021/nsf21530/nsf21530.htm
2021
-
[11]
Halevy, George A
Behzad Golshan, Alon Y. Halevy, George A. Mihaila, and Wang-Chiew Tan. 2017. Data Integration: After the Teenage Years. In PODS
2017
-
[12]
Zezhou Huang, Jia Guo, and Eugene Wu. 2024. Transform Table to Database Using Large Language Models. Proceedings of the VLDB Endowment. ISSN 2150 (2024), 8097
2024
-
[13]
Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, and Yusuke Iwasawa. 2023. Large Language Models are Zero-Shot Reasoners. arXiv:2205.11916 [cs.CL] https://arxiv.org/abs/2205.11916
2023 arXiv
-
[14]
Christos Koutras, George Siachamis, Andra Ionescu, Kyriakos Psarakis, Jerry Brons, Marios Fragkoulis, Christoph Lofi, Angela Bonifati, and Asterios Katsi- fodimos. 2021. Valentine: Evaluating Matching Techniques for Dataset Discovery. In 2021 IEEE 37th International Conference...
2021
-
[15]
Sebastian Kruse, Paolo Papotti, and Felix Naumann. 2015. Estimating Data Integration and Cleaning Effort.. In EDBT. 61–72
2015
-
[16]
Maurizio Lenzerini. 2002. Data integration: A theoretical perspective. In Proceed- ings of the twenty-first ACM SIGMOD-SIGACT-SIGART symposium on Principles of database systems. 233–246
2002
-
[17]
Xinyu Liu, Shuyu Shen, Boyan Li, Peixian Ma, Runzhi Jiang, Yuxin Zhang, Ju Fan, Guoliang Li, Nan Tang, and Yuyu Luo. 2024. A Survey of NL2SQL with Large Language Models: Where are we, and where are we going? arXiv preprint arXiv:2408.05109 (2024)
2024
-
[18]
Xuanqing Liu, Runhui Wang, Yang Song, and Luyang Kong. 2024. GRAM: Gen- erative Retrieval Augmented Matching of Data Schemas in the Context of Data Security. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Dis- covery and Data Mining (Barcelona, Spain) (KDD ’24)....
2024
-
[19]
Yurong Liu, Aecio Santos, Eduardo HM Pena, Roque Lopez, Eden Wu, and Juliana Freire. [n. d.]. Enhancing Biomedical Schema Matching with LLM-based Training Data Generation. In NeurIPS 2024 Third Table Representation Learning Workshop
2024
-
[20]
Sabine Massmann, Salvatore Raunich, David Aumüller, Patrick Arnold, Erhard Rahm, et al. 2011. Evolution of the COMA match system. Ontology Matching 49 (2011), 49–60
2011
-
[21]
Miller, Daniel Fisla, Mary Huang, David Kymlicka, Fei Ku, and Vi- vian Lee
Renée J. Miller, Daniel Fisla, Mary Huang, David Kymlicka, Fei Ku, and Vi- vian Lee. 2001. The Amalgam Schema and Data Integration Test Suite. (2001). www.cs.toronto.edu/ miller/amalgam
2001
-
[22]
Peeters, and Stijn Vansummeren
Marcel Parciak, Brecht Vandevoort, Frank Neven, Liesbet M. Peeters, and Stijn Vansummeren. 2024. Schema Matching with Large Language Models: an Experi- mental Study. arXiv:2407.11852 [cs.DB] https://arxiv.org/abs/2407.11852
2024 arXiv
-
[23]
Lucian Popa, Yannis Velegrakis, Renee J Miller, Mauricio A Hernandez, and Ronald Fagin. 2002. Translating web data. In VLDB’02: Proceedings of the 28th International Conference on Very Large Databases . Elsevier, 598–609
2002
-
[24]
Nabeel Seedat and Mihaela van der Schaar. 2024. Matchmaker: Self-Improving Compositional LLM Programs for Table Schema Matching. InNeurIPS 2024 Third Table Representation Learning Workshop . https://openreview.net/forum?id= KCklcYUlLb
2024
-
[25]
Eitam Sheetrit, Menachem Brief, Moshik Mishaeli, and Oren Elisha
-
[26]
Ananya Singha, José Cambronero, Sumit Gulwani, Vu Le, and Chris Parnin
-
[27]
Yuan Sui, Mengyu Zhou, Mingjie Zhou, Shi Han, and Dongmei Zhang. 2024. Table Meets LLM: Can Large Language Models Understand Structured Table Data? A Benchmark and Empirical Study. InProceedings of the 17th ACM Interna- tional Conference on Web Search and Data Mining (Merida, ...
2024
-
[28]
Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, and Denny Zhou. 2023. Self-Consistency Improves Chain of Thought Reasoning in Language Models. arXiv:2203.11171 [cs.CL] https://arxiv.org/abs/2203.11171
2023 arXiv
-
[29]
Jules White, Quchen Fu, Sam Hays, Michael Sandborn, Carlos Olea, Henry Gilbert, Ashraf Elnashar, Jesse Spencer-Smith, and Douglas C. Schmidt. 2023. A Prompt Pattern Catalog to Enhance Prompt Engineering with ChatGPT. arXiv:2302.11382 [cs.SE] https://arxiv.org/abs/2302.11382
2023 arXiv
-
[30]
E. C. Wood, Amy K. Glen, Lindsey G. Kvarfordt, Finn Womack, Liliana Acevedo, Timothy S. Yoon, Chunyu Ma, Veronica Flores, Meghamala Sinha, Yodsawalai Chodpathumwan, Arash Termehchy, Jared C. Roach, Luis Mendoza, Andrew S. Hoffman, Eric W. Deutsch, David Koslicki, and Stephen A...
2021
-
[31]
Yongqin Xu, Huan Li, Ke Chen, and Lidan Shou. 2024. KcMF: A Knowledge- compliant Framework for Schema and Entity Matching with Fine-tuning-free LLMs. arXiv:2410.12480 [cs.CL] https://arxiv.org/abs/2410.12480
2024 arXiv
-
[32]
Yifan Zeng, Ojas Tendolkar, Raymond Baartmans, Qingyun Wu, Lizhong Chen, and Huazheng Wang. 2024. LLM-RankFusion: Mitigating Intrinsic Inconsistency in LLM-based Ranking. arXiv:2406.00231 [cs.IR] https://arxiv.org/abs/2406.00231
2024 arXiv
-
[33]
Haochen Zhang, Yuyang Dong, Chuan Xiao, and Masafumi Oyamada. 2024. Large Language Models as Data Preprocessors. arXiv:2308.16361 [cs.AI] https: //arxiv.org/abs/2308.16361
2024 arXiv
-
[34]
Jing Zhang, Bonggun Shin, Jinho D Choi, and Joyce C Ho. 2021. SMAT: An attention-based deep learning solution to the automation of schema matching. In Advances in Databases and Information Systems: 25th European Conference, ADBIS 2021, Tartu, Estonia, August 24–26, 2021, Proce...
2021
-
[2019]
IEEE Access 7 (2019), 3694–3703
Gale-Shapley Matching Game Selection—A Framework for User Satisfac- tion. IEEE Access 7 (2019), 3694–3703. doi:10.1109/ACCESS.2018.2888696
2019
-
[2023]
arXiv:2310.10358 [cs.CL] https://arxiv.org/abs/ 2310.10358
Tabular Representation, Noisy Operators, and Impacts on Table Structure Understanding Tasks in LLMs. arXiv:2310.10358 [cs.CL] https://arxiv.org/abs/ 2310.10358
-
[2024]
arXiv:2403.01567 [cs.DB] https://arxiv.org/abs/2403.01567
ReMatch: Retrieval Enhanced Schema Matching with LLMs. arXiv:2403.01567 [cs.DB] https://arxiv.org/abs/2403.01567
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.