REVIEW 3 major objections 6 minor 31 references
Ontology matching can treat equivalence and subsumption as one joint task, and a staged multi-agent LLM search finds either the exact match or the finest broader concept.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · grok-4.5
2026-07-30 11:12 UTC pith:UQ55DNFO
load-bearing objection Useful joint OM task and a solid multi-agent pipeline, but the subsumption SOTA claim rests on derived labels that sit one hop above a deleted near-synonym—exactly where their search is tuned. the 3 major comments →
AgentMap: Joint Equivalence and Subsumption Discovery for Ontology Matching
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
Hybrid Ontology Matching is a well-defined joint task, and AgentMap’s equivalence-first multi-agent workflow—semantic retrieval, hierarchy-guided candidate updates, and specialised LLM agents for screening, verification, and subsumption search—can identify either the equivalent concept or the finest subsumer, outperforming single-step LLM prompts on HOM and outperforming prior specialised systems on equivalence-only and subsumption-only evaluation.
What carries the argument
AgentMap: a three-agent pipeline (initial equivalence screening, equivalence verification against the candidate’s nearest parent and child, iterative subsumption discovery by expanding to parents) plus lexical matching with LLM conflict resolution, all operating on dual top-k embedding candidate sets.
Load-bearing premise
Subsumption ground truth is manufactured by deleting each equivalence target from the ontology and treating its direct parent as the intended match, which may not match how missing equivalents appear in real deployments.
What would settle it
Rebuild the four test sets with independently human-annotated finest subsumers on unmodified target ontologies (no deleted equivalents); if AgentMap’s hybrid and subsumption accuracies then fall below the staged baselines or below BERTSub-style rankers under the same candidate budget, the central performance claim fails.
If this is right
- Knowledge integration pipelines can run one matcher instead of separate equivalence and subsumption systems when the relation type is unknown a priori.
- Staged hierarchy walk plus specialised agents is a reusable pattern for other LLM tasks over taxonomies and concept lattices.
- Subsumption remains the harder half: absolute accuracies stay below 0.5 on three of four sets, so hierarchical search still needs tighter targeting.
- The same dual-candidate design (small set for LLM, larger for lexical) can be reused whenever costly reasoning must be paired with cheap string evidence.
Where Pith is reading between the lines
- If the manufactured-parent construction systematically understates real ambiguity, reported HOM gains may shrink once benchmarks include true missing-leaf and cross-branch subsumption cases.
- The large gap over embedding rankers on near-synonym-heavy pharma data suggests LLM agents mainly win by disambiguating fine lexical near-misses that cosine ranking cannot separate.
- Extending the same agent handoff pattern beyond named concepts to roles, restrictions, or multi-relation alignments is a direct next experiment the architecture already supports.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces Hybrid Ontology Matching (HOM), a task in which a system must decide, per source concept, whether an equivalence or a subsumption mapping to the target ontology holds, and output both the target concept and the relation type. The authors propose AgentMap, a multi-agent LLM framework: embedding retrieval builds candidate sets; AgentES screens for equivalence, AgentEV verifies against the candidate's most similar parent/child, and AgentSD iteratively expands candidates upward through the hierarchy (d_max=2) to find a subsumer; a lexical matcher with LLM conflict resolution runs in parallel. Four existing equivalence benchmarks (three Bio-ML tasks, HeLiS-FoodOn) are converted to HOM by splitting source concepts into equivalence and subsumption subsets; for the latter, the equivalence target is deleted from the target ontology, its children reattached to its parent, and the parent declared the subsumption ground truth. AgentMap outperforms four prompting baselines on HOM, and beats LogMap/AML/BERTMap/GenOM on equivalence and BERTSub/HiT/OnT/embedding rankers on subsumption, with ablations over components, backbones, embeddings, and top-k.
Significance. If the results hold, the HOM formulation is a genuinely useful reframing — in practice the existence of an exact match is not known in advance — and the staged decomposition with hierarchy-guided search is a sensible, transferable design. The evaluation is more careful than typical for this genre: matched candidate coverage for BERTSub (§4.2), dual backbones for the GenOM comparison (Table 3), component ablation (Table 5), and backbone/embedding/top-k sensitivity (Tables 6–8, Fig. 2). Full prompt templates and a fixed split seed are provided (Appendices A–B). The authors are also commendably transparent in §4.1 about the synthetic nature of the subsumption ground truth. However, the headline subsumption results (Table 4, and the "new state of the art" claim in the Conclusion) rest entirely on this derived ground-truth distribution, whose construction aligns closely with the method's search configuration, and no retrieval-ceiling or external-validation evidence is supplied.
major comments (3)
- [§4.1, Tables 2 and 4] The subsumption ground truth is constructed by deleting the equivalence target Y of source concept cs and declaring parent(Y) the GT subsumer. This has two load-bearing consequences the paper does not quantify. (i) Depth: since cs was equivalent to Y, the GT subsumer is always exactly one hierarchy level above cs's granularity; real missing-equivalence cases have no such guarantee (the closest named subsumer may be several levels up or absent). (ii) Retrievability: cs is near-synonymous with the deleted Y, so top-5 embedding retrieval will surface Y's former siblings, and a single Parents() expansion reaches the GT — i.e., the benchmark's difficulty is calibrated to AgentSD's exact configuration (top-5 C0, d_max=2). The paper should report (a) oracle reachability: the fraction of T_sub whose GT lies in C0 ∪ Parents(C0) ∪ Parents²(C0); (b) the distribution of AgentSD successes across iter
- [Table 4, Conclusion] The Conclusion claims AgentMap 'sets a new state of the art on subsumption matching,' but this is measured only on the derived distribution. Two observations make external validation necessary. First, the embedding-ranker baselines score near zero on SNOMED–NCIT–Pharm (OnT 0.012, HiT 0.003, SBERT 0.004, OpenAI 0.009), far below what these encoders achieve on Bio-ML's own independently annotated subsumption task [10]; this suggests the node-removal construction (siblings of the deleted near-synonym outrank the GT parent in embedding space) drives much of the apparent gap, rather than method superiority alone. Second, Bio-ML already provides independently annotated subsumption reference mappings, which the paper cites — evaluating AgentMap and BERTSub on those original subsumption test sets would directly test whether the Table 4 gains transfer off the derived distribution. Either that exp
- [§3.2, Appendix B.2] AgentEV's prompt mandates 'Always select exactly one candidate ID' from C_ref = {ĉt, p*, ch*} — it cannot reject a false positive from AgentES. If AgentEV replaces ĉt with p* (the parent), p* is a subsumer of cs, not an equivalent; what relation does AgentMap then emit? As written (§3.2), the output appears to remain an equivalence mapping, which would be a systematic relation-labeling error counted as wrong in the HOM setting but potentially inflating EqvAcc if evaluated leniently, and silently distorting the equivalence-first logic. Please clarify the relation assigned in this branch and report an error breakdown: how often AgentEV swaps to p*/ch*, and the false-equivalence rate on T_sub instances (where the correct answer is subsumption).
minor comments (6)
- [§4.2 vs §4.4] Inconsistency: §4.2 states BERTSub receives C0 plus parents expanded up to two levels (matching AgentSD's search range), while the note after Table 4 says BERTSub 'is given the initial retrieved candidates C0 as AgentMap.' Please reconcile — the candidate set materially affects the fairness interpretation of Table 4.
- [Table 2, HeLiS–FoodOn rows] Check the bold/underline markup: on HeLiS–FoodOn, LLM+Neighbour+CoT has the best OverallAcc (0.461 vs AgentMap 0.452) and EqvAcc (0.793 vs 0.753), but the typesetting in the provided text is ambiguous about which cells are bolded. The text discussion is honest about this loss; the table should reflect it.
- [§4.4, HeLiS–FoodOn] The subsumption gain over BERTSub (0.269 → 0.278) corresponds to roughly 2 of 190 test instances; with n=190 this is well within sampling noise. A caveat (or confidence intervals) is warranted, as done in Appendix C.2 for the embedding reversal.
- [Appendix D] 'Agent C's subsumption search' — the agents are named AgentES/AgentEV/AgentSD throughout §3.2; unify the naming.
- [General] Typos: 'mathches' (§4.1), 'hirerachy' (§4.2), 'Superisingly' and 'archives' (§4.6), 'the the' (§4.4), 'the corrected predicted source concepts' (§4.1), 'Textural Data' in Fig. 1 (should be 'Textual').
- [§4 / Appendix A] No runtime or LLM-call cost analysis is given. AgentMap issues multiple sequential LLM calls per source concept (screening, verification, iterated subsumption search, conflict resolution) versus one for the prompting baselines; average calls per concept and wall-clock/token cost would help readers assess deployability. Also, 'code and data will be released' — given the benchmark-construction concern above, releasing the modified ontologies and split files is important for verification.
Circularity Check
No derivation circularity: empirical OM system paper; reported accuracies are held-out metrics, not quantities forced by fitted inputs or self-citation chains.
full rationale
AgentMap’s central claims are empirical performance numbers (OverallAcc, EqvAcc, SubAcc, Accuracy_eq, Accuracy_sub) on four constructed HOM splits and on equivalence-/subsumption-alone settings (Tables 2–4). Those metrics are ratios of correct predictions against fixed ground-truth mappings; they are not algebraic consequences of a fitted objective, nor uniqueness theorems imported from the authors. Self-citations (BERTMap, BERTSub, GenOM, Bio-ML) supply prior systems and dataset lineage used as baselines and source material; they do not force AgentMap’s reported scores. The dual candidate sets, d_max=2 hierarchy walk, and LLM agent prompts are design choices evaluated against external labels, not definitions of the claimed accuracies. Benchmark construction (deriving T_sub parents by deleting equivalence targets) raises external-validity questions about how hard subsumption is, but that is evaluation bias, not circular reduction of a ‘prediction’ to its inputs. No self-definitional loop, fitted-input-as-prediction, load-bearing self-cited uniqueness, or renamed known law appears in the derivation chain. Honest finding: score 0; steps empty.
Axiom & Free-Parameter Ledger
free parameters (4)
- C0 top-k (agent candidates) =
5
- C+ top-k (lexical candidates) =
20
- d_max hierarchy expansions =
2
- backbone LLM and embedding model =
GPT-4.1-mini; text-embedding-3-small
axioms (5)
- domain assumption Equivalence is prioritized over subsumption; if an equivalent named concept exists it is the unique correct HOM answer.
- domain assumption When no equivalence exists, the correct answer is the most specific named subsumer in the (possibly modified) target ontology.
- ad hoc to paper Subsumption reference mappings may be derived as direct parents of equivalence targets, after removing those targets and reattaching children.
- domain assumption Labels and synonyms (plus local parents/children) are sufficient evidence for LLM agents; IRIs must not be used.
- domain assumption Cosine similarity of frozen text embeddings yields a candidate pool that contains the true match often enough for top-5 reasoning.
invented entities (2)
-
Hybrid Ontology Matching (HOM) task
no independent evidence
-
AgentMap multi-agent workflow (AgentES, AgentEV, AgentSD + LM&CR)
no independent evidence
read the original abstract
Ontology matching (OM) has traditionally been formulated as either equivalence discovery or subsumption matching. The existing OM systems identify only one type of semantic correspondence and cannot simultaneously discover equivalence and subsumption mappings. In this paper, we introduce Hybrid Ontology Matching (HOM), a new OM task that unifies equivalence and subsumption discovery, and accordingly propose a Large Language Model (LLM)-based multi-agent OM framework AgentMap that is implemented by a series of interdependent semantic decisions. Given a concept in the source ontology, AgentMap integrates semantic retrieval, hierarchical search, and collaborative multi-agent LLM reasoning to progressively explore the target ontology, identifying either the equivalent concept, if one exists, or the most fine-grained subsumer. We further extend four OM datasets for a HOM benchmark and evaluate AgentMap under hybrid, equivalence-only, and subsumption-only settings. Experimental results show that AgentMap achieves promising performance on the hybrid setting, and at the same time outperforms equivalence matching and subsumption matching baselines on the equivalence-only and subsumption-only settings, respectively.
Figures
Reference graph
Works this paper leans on
-
[1]
Ontology matching using convolutional neural networks
Alexandre Bento, Amal Zouaq, and Michel Gagnon. Ontology matching using convolutional neural networks. In Nicoletta Calzolari, Frédéric Béchet, Philippe Blache, Khalid Choukri, 12 Christopher Cieri, Thierry Declerck, Sara Goggi, Hitoshi Isahara, Bente Maegaard, Joseph Mariani, Hélène Mazo, Asuncion Moreno, Jan Odijk, and Stelios Piperidis, editors,Proceed...
2020
-
[2]
Snomed-ct: The advanced terminology and coding system for ehealth
L Bos and K Donnelly. Snomed-ct: The advanced terminology and coding system for ehealth. Stud Health Technol Inform, 121:279–290, 2006
2006
-
[3]
Contextual semantic embeddings for ontology subsumption prediction.World Wide Web, 26(5):2569–2591, September 2023
Jiaoyan Chen, Yuan He, Yuxia Geng, Ernesto Jiménez-Ruiz, Hang Dong, and Ian Horrocks. Contextual semantic embeddings for ontology subsumption prediction.World Wide Web, 26(5):2569–2591, September 2023
2023
-
[4]
Foodon: a harmonized food ontology to increase global food traceability, quality control and data inte- gration.npj Science of Food, 2(1):23, 2018
Damion M Dooley, Emma J Griffiths, Gurinder S Gosal, Pier L Buttigieg, Robert Hoehndorf, Matthew C Lange, Lynn M Schriml, Fiona SL Brinkman, and William WL Hsiao. Foodon: a harmonized food ontology to increase global food traceability, quality control and data inte- gration.npj Science of Food, 2(1):23, 2018
2018
-
[5]
The llama 3 herd of models, 2024
Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Amy Yang, Angela Fan, et al. The llama 3 herd of models, 2024
2024
-
[6]
Cruz, and Fran- cisco M
Daniel Faria, Catia Pesquita, Emanuel Santos, Matteo Palmonari, Isabel F. Cruz, and Fran- cisco M. Couto. The AgreementMakerLight Ontology Matching System. In Robert Meers- man, Hervé Panetto, Tharam Dillon, Johann Eder, Zohra Bellahsene, Norbert Ritter, Pieter De Leenheer, and Deijing Dou, editors,On the Move to Meaningful Internet Systems: OTM 2013 Conf...
2013
-
[7]
LLMs4OM: Matching Ontologies with Large Language Models, April 2024
Hamed Babaei Giglou, Jennifer D’Souza, Felix Engel, and Sören Auer. LLMs4OM: Matching Ontologies with Large Language Models, April 2024. arXiv:2404.10317
Pith/arXiv arXiv 2024
-
[8]
BERTMap: A BERT-Based Ontology Alignment System.Proceedings of the AAAI Conference on Artificial Intelligence, 36(5):5684–5691, June 2022
Yuan He, Jiaoyan Chen, Denvar Antonyrajah, and Ian Horrocks. BERTMap: A BERT-Based Ontology Alignment System.Proceedings of the AAAI Conference on Artificial Intelligence, 36(5):5684–5691, June 2022. Number: 5
2022
-
[9]
Exploring Large Language Models for Ontology Alignment, September 2023
Yuan He, Jiaoyan Chen, Hang Dong, and Ian Horrocks. Exploring Large Language Models for Ontology Alignment, September 2023. arXiv:2309.07172
Pith/arXiv arXiv 2023
-
[10]
Machine Learning-Friendly Biomedical Datasets for Equivalence and Subsumption Ontology Matching
Yuan He, Jiaoyan Chen, Hang Dong, Ernesto Jiménez-Ruiz, Ali Hadian, and Ian Horrocks. Machine Learning-Friendly Biomedical Datasets for Equivalence and Subsumption Ontology Matching. In Ulrike Sattler, Aidan Hogan, Maria Keet, Valentina Presutti, João Paulo A. Almeida, Hideaki Takeda, Pierre Monnin, Giuseppe Pirrò, and Claudia d’Amato, editors,The Semanti...
2022
-
[11]
Language models as hierarchy encoders
Yuan He, Zhangdie Yuan, Jiaoyan Chen, and Ian Horrocks. Language models as hierarchy encoders. InProceedings of the 38th International Conference on Neural Information Processing Systems, NIPS ’24, Red Hook, NY, USA, 2024. Curran Associates Inc
2024
-
[12]
OLaLa: Ontology Matching with Large Language Models
Sven Hertling and Heiko Paulheim. OLaLa: Ontology Matching with Large Language Models. InProceedings of the 12th Knowledge Capture Conference 2023, K-CAP ’23, pages 131–139, New York, NY, USA, December 2023. Association for Computing Machinery. 13
2023
-
[13]
Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al
Albert Q. Jiang, Alexandre Sablayrolles, Antoine Roux, Arthur Mensch, Blanche Savary, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Emma Bou Hanna, Florian Bressand, et al. Mixtral of experts, 2024
2024
-
[14]
Logmap: Logic-based and scalable ontology matching
Ernesto Jiménez-Ruiz and Bernardo Cuenca Grau. Logmap: Logic-based and scalable ontology matching. InThe Semantic Web–ISWC 2011, volume 7031 ofLecture Notes in Computer Science, pages 273–288. Springer, 2011
2011
-
[15]
Magneto: Combining Small and Large Language Models for Schema Matching, June 2025
Yurong Liu, Eduardo Pena, Aecio Santos, Eden Wu, and Juliana Freire. Magneto: Combining Small and Large Language Models for Schema Matching, June 2025. arXiv:2412.08194 [cs]
Pith/arXiv arXiv 2025
-
[16]
Large language models as oracles for ontology alignment
Sviatoslav Lushnei, Dmytro Shumskyi, Severyn Shykula, Ernesto Jiménez-Ruiz, and Ar- tur d’Avila Garcez. Large language models as oracles for ontology alignment. In Vera Demberg, Kentaro Inui, and Lluís Marquez, editors,Proceedings of the 19th Conference of the European Chapter of the Association for Computational Linguistics (Volume 1: Long Papers), pages...
2026
-
[17]
Self-refine: Iterative refine- ment with self-feedback.Advances in neural information processing systems, 36:46534–46594, 2023
Aman Madaan, Niket Tandon, Prakhar Gupta, Skyler Hallinan, Luyu Gao, Sarah Wiegreffe, Uri Alon, Nouha Dziri, Shrimai Prabhumoye, Yiming Yang, et al. Self-refine: Iterative refine- ment with self-feedback.Advances in neural information processing systems, 36:46534–46594, 2023
2023
-
[18]
BioSTransformers for Biomedical Ontologies Alignment:
Safaa Menad, Wissame Laddada, Saïd Abdeddaïm, and Lina Soualmia. BioSTransformers for Biomedical Ontologies Alignment:. InProceedings of the 15th International Joint Conference on Knowledge Discovery, Knowledge Engineering and Knowledge Management, pages 73–84, Rome, Italy, 2023. SCITEPRESS - Science and Technology Publications
2023
-
[19]
Openai agents sdk: Handoffs.https://openai.github.io/openai-agents-python/ handoffs/, 2025
OpenAI. Openai agents sdk: Handoffs.https://openai.github.io/openai-agents-python/ handoffs/, 2025. Accessed: 2026-07-21
2025
-
[20]
Ontol- ogy matching: A literature review.Expert Systems with Applications, 42(2):949–971, 2015
Lorena Otero-Cerdeira, Francisco J Rodríguez-Martínez, and Alma Gómez-Rodríguez. Ontol- ogy matching: A literature review.Expert Systems with Applications, 42(2):949–971, 2015
2015
-
[21]
BioGITOM: Matching Biomedical Ontologies with Graph Isomorphism Transformer.The VLDB Journal, 34(6):65, September 2025
Samira Oulefki, Lamia Berkani, Nassim Boudjenah, Ladjel Bellatreche, and Aicha Mokhtari. BioGITOM: Matching Biomedical Ontologies with Graph Isomorphism Transformer.The VLDB Journal, 34(6):65, September 2025
2025
-
[22]
Genom: ontology matching with descrip- tion generation and large language models.World Wide Web, 29(3):29, 2026
Yiping Song, Jiaoyan Chen, and Renate A Schmidt. Genom: ontology matching with descrip- tion generation and large language models.World Wide Web, 29(3):29, 2026
2026
-
[23]
Springer Science & Business Media, 2013
Steffen Staab and Rudi Studer.Handbook on ontologies. Springer Science & Business Media, 2013
2013
-
[24]
Chi, Quoc Le, and Denny Zhou
Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Ed H. Chi, Quoc Le, and Denny Zhou. Chain-of-thought prompting elicits reasoning in large language models. InAdvances in Neural Information Processing Systems (NeurIPS), volume 35, pages 24824–24837, 2022
2022
-
[25]
Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024
An Yang et al. Qwen2.5 technical report.arXiv preprint arXiv:2412.15115, 2024
Pith/arXiv arXiv 2024
-
[26]
Language models as ontology encoders
Hui Yang, Jiaoyan Chen, Yuan He, Yongsheng Gao, and Ian Horrocks. Language models as ontology encoders. InThe Semantic Web – ISWC 2025: 24th International Semantic Web Conference, Nara, Japan, November 2–6, 2025, Proceedings, Part I, page 443–461, Berlin, Heidelberg, 2025. Springer-Verlag. 14
2025
-
[27]
React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022
Shunyu Yao, Jeffrey Zhao, Dian Yu, Nan Du, Izhak Shafran, Karthik Narasimhan, and Yuan Cao. React: Synergizing reasoning and acting in language models.arXiv preprint arXiv:2210.03629, 2022. A Implementation Details The benchmark is split into equivalence and subsumption evaluation subsets using a fixed ran- dom seed of 42 to ensure reproducibility. All op...
Pith/arXiv arXiv 2022
-
[28]
Identify whether each plausible candidate is broader than the source
-
[29]
Exclude candidates that are equivalent, narrower, sibling concepts, parts, attributes, or merely related 16 concepts
-
[30]
Among the remaining broader candidates, choose the closest and most specific one
-
[31]
If no candidate is a perfect direct parent, choose the best available broader candidate. Return exactly this format: Reasoning: <short reasoning> Selected: <candidate_id> This prompt is applied once, afterdmax upward expansions, over the complete search trajectoryC0∪C1∪· · ·∪Cdmax. B.4 Conflict Resolution Prompt: Equivalence Arbitration Conflict Resolutio...
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.