{"id":"a9f729c8-7d03-4af9-aee5-97309e795d53","arxiv_id":"2501.08686","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"KG-RAG4SM retrieves relevant Wikidata subgraphs and feeds them to an LLM to decide whether two schema attributes match.","lead":"A team combined large language models with knowledge graph retrieval to decide when two database columns mean the same thing, and reports accuracy gains on healthcare benchmarks. The evaluation has gaps, including no error bars, test-set tuning, and missing comparisons with the closest retrieval-based baselines.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The SOTA comparison rests on degenerate baseline operating points: SMAT and Jellyfish achieve near-100% recall with very low precision, so the reported relative gains are not evidence of SOTA-level schema matching.","rationale":"The reader's weakest assumption (few-shot prompt leakage) is a real problem, but it primarily affects the CMS benchmark, where the prompt examples in Figures 4 and 8 are literally CMS attribute pairs; it does not directly impeach the MIMIC/Synthea headline numbers. The more load-bearing concern is that the comparison baselines are not credible SOTA systems: SMAT-trained and Jellyfish-8B are evaluated at operating points with near-100% recall and very low precision, which is consistent with near-trivial high-recall prediction rather than a properly tuned schema matcher. Because the abstract's percentage gains are computed relative to these weak points, the central claim is fragile even if the prompt-example leakage is fixed. The paper introduces a genuinely useful EMED dataset and demonstrates an external-KG retrieval pipeline, and the code is provided, but the evidence for the headline SOTA claim is not sufficient. I therefore agree with the REJECT verdict, though for a different primary reason than the reader's weakest assumption; hence partial agreement. The proposed concrete test, re-running SMAT under its intended evaluation protocol, would settle whether the gap is real or an artifact of baseline misconfiguration.","tokens_in":27799,"tokens_out":8986,"duration_ms":88427,"concrete_test":"Run the released SMAT code on the Synthea dataset using its original OMAP evaluation protocol, including validation-set threshold selection, and compare its P/R/F1 with the KG-RAG4SM row in Table 6 (P=20.00, R=36.36, F1=25.80). If SMAT's F1 exceeds 25.80 at any threshold, or if its precision is materially above the reported 11.82, then the claimed 69.20%/21.97% improvement over SMAT is an artifact of a weak baseline operating point. Similarly, re-run Jellyfish-8B on MIMIC using its official schema-matching prompt and threshold calibration; if its F1 exceeds 16.66, the MIMIC claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that KG-RAG4SM beats SOTA methods depends on the baseline operating points in Tables 5 and 6, and those operating points look degenerate. SMAT-trained achieves R=100 with P=11.82 on Synthea and P=1.25 on EMED; Unicorn-trained reaches R≈99.99 with P≈0.04-0.46; Jellyfish-8B reaches R=100 with P=6.81 on MIMIC. A model that obtains 100% recall by predicting almost every candidate as positive is not a credible SOTA schema matcher under the extreme class imbalance of these benchmarks (e.g., 129 positives among 64,080 MIMIC pairs). The paper reports no validation-based threshold selection for these baselines and no error bars for the stochastic LLM runs, so the claimed 35.89%/30.50% and 69.20%/21.97% relative gains are gains over weak operating points rather than evidence of SOTA-level performance. Moreover, the abstract's MIMIC precision improvement (35.89%) is not reproducible from Table 5, where 9.52 versus 6.81 is a 39.8% relative increase, indicating the headline numbers are not tightly coupled to the reported tables. If the baselines were operated at their intended thresholds, the claimed SOTA advantage could disappear.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes KG-RAG4SM, a knowledge-graph-based retrieval-augmented generation framework for schema matching. It retrieves relevant subgraphs from Wikidata using vector-based, BFS-based, LLM-based, and query-based methods, prunes them with ranking schemes, and injects the retrieved knowledge into LLM prompts. The authors introduce a new EMED benchmark, report experiments on MIMIC, Synthea, CMS, and EMED, and claim that KG-RAG4SM outperforms state-of-the-art LLM-based (Jellyfish-8B, GPT-4o-mini) and PLM-based (SMAT, Unicorn) matchers in precision and F1, while also mitigating LLM hallucinations. The paper includes ablation studies on retrieval strategies, backbone LLMs, and ranking, plus efficiency comparisons.","tokens_in":28073,"tokens_out":7347,"duration_ms":67304,"significance":"If the stated results held, the paper would offer a practical way to inject structured world knowledge into LLM-based schema matching without retraining, and the EMED dataset would be a useful resource for the community. The paper ships code and data, and the ablation of four retrieval paradigms is informative. The case studies give concrete illustrations of how KG context can correct LLM errors. However, the headline claims are not currently supported by the evidence: the comparison baselines are operated at degenerate working points, the configuration is selected using test data, the few-shot examples overlap with the evaluated benchmark, and the absolute precision values are too low for real deployment. These issues materially weaken the significance of the empirical claims.","major_comments":[{"comment":"The SOTA comparison rests on degenerate baseline operating points. Jellyfish-8B reaches 100% recall with precision 6.81 on MIMIC, SMAT reaches 100% recall with precision 11.82 on Synthea, and Unicorn-trained reaches roughly 99.99% recall with precision below 1% on all datasets. The paper does not tune decision thresholds for these baselines; it simply compares F1 and precision at these extreme points. Under the heavy class imbalance of these benchmarks (e.g., 129 positive pairs out of 64,080 on MIMIC), a model that predicts almost all pairs as positive trivially obtains high recall, and the reported relative gains over such baselines do not constitute evidence of SOTA-level schema matching. The evaluation should either tune each baseline's threshold on a validation set or report precision-recall curves.","section":"§4.2.1, Tables 5 and 6"},{"comment":"The evaluation protocol has no train/validation/test split for KG-RAG4SM, and the system configuration is selected after observing CMS test results. The paper states that cosine similarity was chosen 'after extensive experimentation' (§3.2.1), that top-2 ranked subgraphs are 'generally more effective' (§3.2.3), and that 3-hop search is a 'trade-off solution' (§3.2.2); all of these decisions are validated on the CMS dataset (§4.3.1). Reporting results on the same dataset used to select the pipeline, top-k, BFS depth, and similarity metric inflates the reported performance and makes the transfer to other datasets difficult to assess. The authors should hold out a validation set for configuration selection or report results under a predefined configuration.","section":"§4.1.1, §3.2, §4.3.1"},{"comment":"The few-shot examples in the prompts are not guaranteed to be disjoint from the evaluation pairs. Figure 4 and Example 1 of Figure 8 use the pair death-personid and beneficiarysummary-desynpufid, which are CMS attributes, and CMS is one of the evaluated datasets. The paper says there is no train/validation/test split for KG-RAG4SM (§4.1.1) but does not state that these prompt examples are excluded from the evaluation. If any of the few-shot pairs coincide with an evaluated pair, the reported gains include answer memorization rather than retrieval benefit. The authors must explicitly confirm the disjointness of prompt examples and test pairs, or re-design the evaluation with examples from an unrelated domain.","section":"§3.1, Figure 4, §4.1.1"},{"comment":"The headline improvement numbers in the abstract are not reproducible from the reported tables. The abstract claims a 35.89% precision improvement on MIMIC over Jellyfish-8B, but Table 5 shows 9.52% versus 6.81%, which is a 39.8% relative increase. The F1 improvement (30.50%) is close to the computed 30.56% (16.66 versus 12.76), but the precision figure is inconsistent. This discrepancy suggests that the abstract numbers are not tightly coupled to the experimental results and must be corrected.","section":"Abstract and §4.2.1, Table 5"},{"comment":"The experiments report a single run for each LLM-based configuration, with no error bars or repeated runs. Although the decoding hyperparameters (top-k=1, top-p=0.9, temperature 0.6) likely make the outputs nearly deterministic, the paper does not establish this or report any variance. Given the small metric differences among some settings (e.g., CMS precision 52.38 versus 44.44 in Table 8), a single run is insufficient to demonstrate that the differences are stable. The paper should report multiple runs and variance, or explicitly justify that the decoding settings remove sampling stochasticity.","section":"§4.2, Table 4"}],"minor_comments":[{"comment":"The text contains a typo: 'MIMC' should be 'MIMIC' in the results analysis.","section":"§4.2.1"},{"comment":"The figure title reads 'Prompts for LLMs as Schema Mather' and the prompt text contains 'atttribute'; both are typos.","section":"Figure 4"},{"comment":"The conclusion refers to the method as 'KG-RAG4M' instead of 'KG-RAG4SM'.","section":"§6"},{"comment":"The symbol G is used both for the large knowledge graph and for the retrieved subgraph, which is confusing; the paper should use distinct notation for the full graph and the subgraph.","section":"§2.1.2 and §2.2"},{"comment":"The hyperparameter table lists top-k=1 for LLM generation; with top-k=1 the temperature 0.6 has no effect on sampling. The authors should clarify whether sampling is enabled and how the selection of top-k=1 interacts with the temperature setting.","section":"Table 4"}],"recommendation":"major_revision","confidential_remarks":"The paper addresses a relevant problem and has reproducible artifacts, but the empirical evaluation needs substantial rework before the claims can be considered. The lack of threshold tuning for baselines and the absence of a validation split are the most serious issues; both are fixable in revision. The abstract discrepancy with Table 5 is also concerning and should be corrected. I would encourage the editor to request a major revision rather than a rejection, since the core idea is sound and the identified problems are within the scope of a careful rewrite."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The one genuinely new thing here is retrieving subgraphs from a large external KG (Wikidata) to augment LLMs for schema matching, and the new EMED benchmark (e-MedSolution to OMOP CDM) is a useful resource. The design space of retrieval variants is reasonable, the ablations are informative, and the paper ships code and data. Credit where due: the Figure 10 case study shows two concrete errors where injected KG context flips the LLM answer to the ground truth, and the ranking-based refinement clearly helps in Table 9.\n\nThe soft spots are all in the evaluation, and they are load-bearing. The stress-test note lands: SMAT-trained and Unicorn-trained hit near-100% recall by labeling almost every pair positive, and Jellyfish-8B does the same on MIMIC (R=100, P=6.81). Claiming a 35.89% precision improvement over that operating point is not evidence of SOTA behavior. The paper itself acknowledges the baselines look overfit, yet still uses them for the headline comparison. There is also test-set tuning: top-k, BFS depth, similarity metric, and ranking are selected after seeing CMS results, reported without error bars despite temperature 0.6, and the prompt few-shot examples come from the same benchmark family with no held-out split. The abstract's precision number does not match Table 5 (9.52 vs 6.81 is ~39.8%, not 35.89%), which suggests sloppiness in the headline figures.\n\nI could not find deliberate deception; the method and dataset are presented transparently. But the submitted evidence does not support the SOTA claims. A revised version with baselines operated at sensible thresholds, repeated LLM runs, a proper hyperparameter selection protocol, and at least an attempt at the closest RAG baselines (ReMatch is the obvious one) could make this a solid paper.\n\nWho for: data integration researchers working on LLM-based schema matching and anyone interested in evaluation pitfalls. It deserves a serious referee — the idea is worth examining and the EMED dataset is valuable, but the revision bar is high.","headline":"KG-RAG4SM's idea is fresh but the evaluation's baseline operating points are degenerate, so the SOTA claims don't hold — still worth reviewing for the dataset and mechanism.","tokens_in":28604,"tokens_out":2884,"would_cite":false,"duration_ms":30213,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A system that retrieves relevant subgraphs from Wikidata and inserts them into LLM prompts improves schema-matching precision and F1 over LLM and PLM baselines, with no retraining of the backbone model.","keywords":["schema matching","knowledge-graph retrieval","retrieval-augmented generation","large language models","data integration","Wikidata","healthcare data","subgraph ranking"],"falsifier":"Hold out the exact prompt-example pairs from the CMS ground truth, rerun the pipeline, and compare F1; if the gain over the no-RAG baseline concentrates on those pairs and collapses on the remaining ones, the reported improvement is prompt memorization rather than retrieval benefit.","tokens_in":1579,"feed_emoji":"🔗","tokens_out":2733,"duration_ms":96776,"temperature":0.7,"pith_summary":"KG-RAG4SM is a retrieval-augmented generation pipeline for schema matching: given a pair of attributes from a source and a target database, it retrieves the most relevant triples and paths from Wikidata, ranks them, and pastes the chosen subgraph into the prompt of an LLM that answers a binary question about whether the two attributes are semantically matched. The paper's central claim is that this external knowledge injection resolves semantic ambiguities that similarity-based and plain-LLM matchers miss, and that it does so without fine-tuning or training. On healthcare benchmarks, the reported effect is sizable: with Jellyfish-8B as the backbone, precision and F1 on MIMIC improve by 35.89% and 30.50% over the same model without retrieval; with GPT-4o-mini as the backbone, precision and F1 on Synthea improve by 69.20% and 21.97% over the trained SMAT baseline. The paper also introduces EMED, a new real-world healthcare schema-matching benchmark, and argues that the retrieval is efficient enough to scale to Wikidata-scale graphs.","feed_headline":"Knowledge-graph context lifts LLM schema matching by up to 69%","feed_subtitle":"Retrieved Wikidata subgraphs resolve ambiguous attribute pairs like doctor versus provider, with no retraining.","key_machinery":"KG-RAG4SM (knowledge graph-based retrieval-augmented generation for schema matching) is a pipeline that combines retrieval, ranking, and prompt augmentation. It forms a binary question from the source attribute, target attribute, and their textual descriptions; retrieves relevant knowledge from Wikidata; and then ranks the retrieved paths before injecting the top-1 or top-2 subgraphs into the LLM prompt. Two retrieval routes carry the argument: vector-based KG-triple retrieval, which embeds questions and KG triples and selects by cosine similarity, and vector-based entity retrieval followed by 3-hop BFS traversal between retrieved entity pairs. A ranking scheme scores paths by how many of the top retrieved relations they contain, normalized by path length, so that concise, relevant subgraphs are preferred over long or noisy ones. The load-bearing step is the subgraph itself: it gives the LLM a verifiable external semantic relationship instead of relying only on parametric memory.","core_discovery":"KG-RAG4SM treats each candidate attribute pair as a binary schema-matching question and augments the prompt with a small, ranked subgraph retrieved from Wikidata. The central claim is that the retrieved subgraph supplies the missing commonsense or domain knowledge: for example, knowing that a beneficiary is a subclass of a customer and a customer is a subclass of a patient helps the model separate measurement time from intervention date, while knowing that a physical-status classification is a medical classification helps it align two identifier columns. Across MIMIC, Synthea, CMS, and the new EMED dataset, the method reports higher precision and F1 than Jellyfish-8B, GPT-4o-mini without retrieval, SMAT, and Unicorn, with the two vector-based retrievers (KG-triple retrieval and entity retrieval followed by 3-hop BFS) as the best-performing pipelines. The paper further claims that the retrieved context mitigates LLM hallucination on complex real-world mapping cases.","pith_inferences":["The paper does not establish a clean separation between prompt examples and evaluated pairs, so the CMS results are the least protected against memorization; MIMIC and Synthea numbers are more informative as tests of the retrieval claim.","The reported improvements are relative gains over baselines whose absolute precision and F1 are low, so the practical significance depends on whether a deployment can tolerate many false positives or would need a human-in-the-loop.","A stronger test of the mechanism would compare KG-RAG4SM against a retrieval-augmented baseline that retrieves from the schema's own textual descriptions rather than from Wikidata, isolating the value of external structured knowledge from the value of any retrieved context."],"forward_implications":["Schema matching can be improved without a label-hungry training phase: KG-RAG4SM needs only prompt examples and an external KG, not a supervised training set.","Smaller LLMs can outperform larger ones when equipped with retrieved KG context, because retrieved knowledge can conflict with the parametric knowledge of large models.","Noise control is central: keeping only top-1 or top-2 ranked subgraphs prevents context poisoning, and on the CMS dataset ranking-based refinement improved F1 by 62.59% over unranked retrieval for the KG-triples pipeline.","The paradigm is designed to transfer to other LLM-based data integration tasks such as entity matching and data fusion, which the paper names as future work.","A commonsense KG such as Wikidata can support domain-specific healthcare matching for ambiguities involving acronyms, abbreviations, relatedness, and hierarchical roles."],"supporting_citations":[{"why":"Supplies Wikidata as the external knowledge graph whose dumps and API provide the entities, relations, and triples used in retrieval.","marker":"[18]"},{"why":"Jellyfish is the instruction-tuned LLM baseline and the backbone used for the MIMIC comparison.","marker":"[6]"},{"why":"SMAT is the PLM baseline and the source of the OMAP benchmark datasets MIMIC, Synthea, and CMS.","marker":"[21]"},{"why":"Unicorn is the second PLM baseline compared in the effectiveness evaluation.","marker":"[11]"},{"why":"RoBERTa is the embedding model used to vectorize questions, entities, relations, and triples for vector-based retrieval.","marker":"[26]"},{"why":"Wikidata5M provides the KG subset whose triples are used to train and support vector-based KG-triple retrieval.","marker":"[34]"}],"fun_headline_variants":["LLM schema matching boosted 69% via knowledge graph retrieval","No retraining: KG-augmented LLM beats SOTA schema matching by 69%","Knowledge graph retrieval fixes LLM hallucination in schema matching","Schema matching gains 69% precision with knowledge graph RAG","Retrieved Wikidata subgraphs lift LLM schema matching accuracy"],"cache_read_input_tokens":30720,"weakest_assumption_plain":"The load-bearing premise is that the few-shot examples in the prompt are disjoint from the attribute pairs being evaluated; the paper does not enforce a train/test split, and the illustrative example pairs come from the CMS benchmark, which is itself one of the evaluated datasets.","fun_headline_variants_meta":{"raw":{"variants":["LLM schema matching boosted 69% via knowledge graph retrieval","No retraining: KG-augmented LLM beats SOTA schema matching by 69%","Knowledge graph retrieval fixes LLM hallucination in schema matching","Schema matching gains 69% precision with knowledge graph RAG","Retrieved Wikidata subgraphs lift LLM schema matching accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00073,"raw_usage":{"total_tokens":3323,"prompt_tokens":1056,"completion_tokens":2267,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":672,"completion_tokens_details":{"reasoning_tokens":2176}},"tokens_in":672,"tokens_out":2267,"duration_ms":15204,"temperature":1.0,"reasoning_tokens":2176,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:20:00.384369+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out the exact prompt-example pairs from the CMS ground truth, rerun the pipeline, and compare F1; if the gain over the no-RAG baseline concentrates on those pairs and collapses on the remaining ones, the reported improvement is prompt memorization rather than retrieval benefit.","supporting_citations":[{"cited_title":"Wikidata: A Free Collaborative Knowledgebase,","cited_arxiv_id":null,"evidence_quote":"Supplies Wikidata as the external knowledge graph whose dumps and API provide the entities, relations, and triples used in retrieval."},{"cited_title":"Jellyfish: Instruction-Tuning Local Large Language Models for Data Pre- processing,","cited_arxiv_id":null,"evidence_quote":"Jellyfish is the instruction-tuned LLM baseline and the backbone used for the MIMIC comparison."},{"cited_title":"SMAT: An Attention- based Deep Learning Solution to the Automation of Schema Matching,","cited_arxiv_id":null,"evidence_quote":"SMAT is the PLM baseline and the source of the OMAP benchmark datasets MIMIC, Synthea, and CMS."},{"cited_title":"Uni- corn: A Unified Multi-tasking Model for Supporting Matching Tasks in Data Integration,","cited_arxiv_id":null,"evidence_quote":"Unicorn is the second PLM baseline compared in the effectiveness evaluation."},{"cited_title":"An Analysis of Links in Wikidata,","cited_arxiv_id":null,"evidence_quote":"Wikidata5M provides the KG subset whose triples are used to train and support vector-based KG-triple retrieval."}],"review_version":1}