Pith. sign in

REVIEW 4 major objections 5 minor 1 cited by

MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph

T0 review · 4 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read MEDMKG is a multimodal medical knowledge graph linking chest X-ray images to UMLS concepts, with a Neighbor-aware Filtering algorithm and benchmarks showing mixed but often positive downstream gains.

desk verdict The resource is genuinely new and worth having, but the downstream experiments don't actually show MEDMKG drove the gains—the augmentation methods use their own knowledge sources and no protocol or ablation links Tables 3–4 to MEDMKG. read the letter →

arxiv 2505.17214 v1 pith:AFKDVRVI submitted 2025-05-22 cs.AI

Xiaochen Wang , Yuan Zhong , Lingwei Zhang , Lisong Dai , Ting Wang , Fenglong Ma This is my paper
classification cs.AI
keywords knowledgemedicalmultimodalmedmkgclinicalgraphgraphstasks
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper introduces MEDMKG, a knowledge graph that connects chest X-ray images from MIMIC-CXR to clinical concepts from UMLS. A two-stage pipeline first uses MetaMap to find candidate UMLS concepts in radiology reports, then uses ChatGPT-4o to pick the concept that best fits the report context and to label the image-concept relation as positive, negative, or uncertain. A new filtering step, Neighbor-aware Filtering (NaF), ranks images by how many distinct concept-relation pairs they connect and how rare those pairs are, keeping only images needed to cover all concepts.

The authors then benchmark the graph with 24 baselines and 4 vision-language backbones on link prediction, text-image retrieval, and visual question answering. Many configurations improve over the backbone, especially on retrieval at small K and on VQA accuracy with MR-MKG. However, some configurations hurt performance, for example CLIP with KRISP on SLAKE drops from 65.07 to 56.90 accuracy. All results come from a single run with a fixed seed; the paper's own checklist says no error bars or significance tests were computed.

The graph quality rests on assumptions that are checked only lightly: the human evaluation covers 30 images and 30 concepts, and the filtering step is not compared against keeping the unfiltered graph or random filtering. The resource is still a useful contribution because code, graph metadata, and reconstruction instructions are public, but the strength of the downstream claims should be read with caution.

Extended reading notes

Core claim

The central claim appears in the abstract: 'Results show that MEDMKG not only improves performance in downstream medical tasks but also offers a strong foundation for developing adaptive and robust strategies for multimodal knowledge integration in medical artificial intelligence.' If the paper is correct, integrating MEDMKG into vision-language models improves medical text-image retrieval and visual question answering, and the NaF filtering step preserves graph quality while reducing redundancy.

Load-bearing premise

The construction assumes that the Impression and Findings sections of each MIMIC-CXR report accurately describe what is visible in the paired X-ray image, so concepts extracted from that text are valid labels for the image (Sections 3.1 and 3.2, Appendix F.1). Any historical, comparative, or non-visual statement in a report would create a false cross-modal edge. This assumption is spot-checked only by a human evaluation on 30 images, so its failure rate at scale is unknown.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper introduces MEDMKG, a multimodal medical knowledge graph built by extending UMLS with chest X-ray images and concepts extracted from MIMIC-CXR reports using MetaMap, ChatGPT-4o-based disambiguation, and a relation-stance labeling step. To reduce redundancy, the authors propose Neighbor-aware Filtering (NaF), which scores images by the informativeness of their concept neighbors and greedily selects images until all concepts are covered. The graph is evaluated in two settings: link prediction on the graph itself (17 embedding baselines) and knowledge-augmented text-image retrieval and VQA, where existing KG-augmented methods (KnowledgeCLIP, FashionKLIP, KRISP, MKBN, K-PathVQA, EKGRL, MR-MKG) are combined with four vision-language backbones and evaluated on six datasets. The paper reports quality scores from a small human evaluation and claims that MEDMKG improves downstream medical tasks.

Significance. A well-curated multimodal medical knowledge graph would be a valuable community resource, and the paper provides a concrete construction pipeline plus released code and data. The breadth of the benchmark (17 link-prediction models, 4 backbones, 6 datasets) is a genuine strength, and the human evaluation, though small, is a reasonable first step toward quality assessment. However, the central utility claim is not currently supported: the downstream experiments do not show that any baseline actually consumed MEDMKG, no ablation isolates the effect of the graph or of the NaF filter, and the paper itself admits (checklist item 7) that no error bars or significance tests are reported. If the attribution issue is resolved, the resource still has value, but as written the evidence for the abstract's downstream-performance claim is incomplete.

major comments (4)
  1. [Section 4, especially 4.2 and 4.3] The paper asserts that 'MEDMKG is employed as auxiliary knowledge' in the augmentation setting, but no reported protocol establishes that any of the evaluated methods actually consume MEDMKG. Appendix J.1 describes KnowledgeCLIP as using ConceptNet or VisualGenome and FashionKLIP as using FashionMMKG; Appendix K.2 describes KRISP, MKBN, K-PathVQA, EKGRL, and MR-MKG with their own task-specific knowledge sources. The paper never specifies a graph-injection or adaptation module for MEDMKG, never reports an ablation comparing 'method with MEDMKG' versus 'method without MEDMKG' or 'method with UMLS-only graph,' and never states which graph each table row used. Consequently Tables 3 and 4 cannot attribute the observed gains to MEDMKG; they could reflect the methods' original knowledge sources.
  2. [Section 4.3, Table 4] The claim that 'incorporating external knowledge from our multimodal medical knowledge graph consistently improves model performance' is contradicted by multiple entries in Table 4, for example CLIP+SLAKE accuracy drops from 65.07 to 56.90 with +KRISP, and BioMedCLIP+SLAKE accuracy drops from 70.14 to 57.18 with +KRISP; numerous Precision and Recall values also decrease. Similar drops appear in Table 3 (e.g., CLIP+MIMIC-CXR Recall@100 falls from 58.26 to 56.06 with +FashionKLIP). The paper's own checklist item 7 states that no error bars or significance tests are provided, so the 'consistent improvement' claim is not supported by the reported evidence and should be substantially qualified.
  3. [Section 3.4 and Section 4] NaF is presented as a core contribution, but no experiment evaluates its effect. The filtering algorithm is never ablated: the paper does not compare link prediction, retrieval, or VQA performance on the unfiltered graph versus the NaF-filtered graph, nor does it compare NaF against a random or degree-based filter. The human evaluation in Appendix G assesses only the final filtered graph, so the claims that NaF 'preserves graph quality while reducing redundancy' and improves downstream utility are not empirically tested.
  4. [Sections 3.1-3.2 and Appendix F.1] The construction pipeline relies on the assumption that the Impression and Findings sections of each MIMIC-CXR report describe what is actually visible in the paired image, so concepts extracted from the text become labels for the image. Historical, comparative, or non-visual statements in a report would create false cross-modal edges. The human evaluation in Appendix G samples only 30 images, with no reported error bars or inter-rater agreement, so the failure rate of this assumption at scale is unknown; the authors should quantify this risk or provide a substantially larger validation.
minor comments (5)
  1. [Appendix E] The Limitations section addresses data access and privacy but does not mention the label-noise risk of using report text as image labels or the lack of statistical significance testing, despite the checklist claiming limitations are discussed.
  2. [Figure 1 and Appendix G] The human assessment reports 'approximately 80%' scores but does not state the number of raters, the instruction details, or any measure of agreement; this information should be added.
  3. [Appendix F.1 and Section 4.2] MedCSPCLIP is used both to select representative reports during graph construction and as an evaluation backbone in retrieval and VQA experiments; although the retrieval split excludes MIMIC-CXR pairs used in curation, this overlap should be discussed as a possible source of favorable bias.
  4. [Table 2 and Appendix F.5] There are a few typographical errors: 'SimpIE' should be 'SimplE' in the Table 2 discussion, and 'Illutration' in Appendix F.5 is misspelled.
  5. [Section 4.3] The statement that MR-MKG 'demonstrates superior generalization' is based on a single fixed-seed run on three datasets; without variance or statistical tests, this conclusion is stronger than the evidence warrants.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: construction parameters are not fit to downstream metrics, link prediction uses standard held-out splits, and the only self-citation (MedCSPCLIP) is not load-bearing.

full rationale

MEDMKG's construction pipeline is fixed before any downstream evaluation: MetaMap concept identification, ChatGPT-4o disambiguation, UMLS intra-modality edges, and NaF image scoring do not optimize any retrieval, VQA, or link-prediction target metric. NaF's informativeness score in Eq. (1) is a hand-specified heuristic over graph-internal neighbor counts, and the selection rule stops when all concepts are covered, so the filtering step does not reduce to any later performance quantity. The link prediction experiments (Section 4.1) train and evaluate seventeen models on an 8:1:1 split of the graph, which is standard model evaluation rather than a fitted-input prediction. The knowledge-augmentation experiments (Sections 4.2 and 4.3) report existing KG-augmented methods rather than a method whose parameters are fit to MEDMKG; the paper's failure to detail how each baseline ingested MEDMKG is an evidential gap in the utility claim, not a circular reduction. The one self-citation intersection is MedCSPCLIP [11], which is used in Appendix F.1 for report clustering and also appears as a downstream backbone; because MedCSPCLIP is a fixed pretrained model and the retrieval evaluation explicitly excludes MIMIC-CXR pairs used in curation, this intersection does not cause any predicted quantity to be defined in terms of itself. No equation, fitted parameter, or evaluation metric in the paper is equivalent by construction to the graph construction inputs, so the derivation chain is self-contained with respect to circularity.

Assumptions & free parameters 3 free parameters · 5 assumptions · 0 invented entities

The graph's validity depends on unvalidated choices: DBSCAN clustering and centroid selection in preprocessing, the hand-authored UMLS semantic type filter, the NaF informativeness assumption, and the assumed fidelity of report text to image content. No free parameters are fitted to downstream task labels, but the construction choices are not sensitivity-tested. No new physical entities are proposed.

free parameters (3)
  • DBSCAN hyperparameters (eps, min_samples)
    Appendix F.1 uses DBSCAN on MedCSPCLIP embeddings to select representative reports; the hyperparameters are not reported and influence which reports enter the graph.
  • Excluded UMLS semantic type list
    Table 5 is a hand-authored filter determining which concepts may appear; its effect on graph quality and downstream utility is not tested.
  • Retrieval evaluation sample size = 10,000
    Section 4.2 samples 10,000 MIMIC-CXR pairs from data not used in curation; results depend on this fixed sample, and its representativeness is not analyzed.
assumptions (5)
  • domain assumption UMLS provides correct and sufficiently complete clinical concepts and relations for the radiology domain
    Section 3.1 treats UMLS as the foundational backbone; errors or gaps in UMLS propagate into all intra-modality edges and constrain cross-modal concepts.
  • domain assumption MetaMap plus ChatGPT-4o returns accurate, context-appropriate UMLS concepts for radiology reports
    Section 3.2 relies on this two-stage extraction for every cross-modal edge; only 30 images are human-checked, so accuracy at scale is not established.
  • domain assumption Impression and Findings sections describe what is visible in the paired image
    Section 3.1 links images to concepts extracted from reports, and Appendix F.1 restricts extraction to these sections; non-visual statements would create false positive edges.
  • ad hoc to paper NaF score ranks images by clinical informativeness
    Equation (1) and Algorithm 1 assert that images with rare relation-concept pairs are more informative and that greedy coverage preserves clinical richness; no ablation against unfiltered or random-filtered graphs is given.
  • domain assumption MedCSPCLIP text embeddings support meaningful report clustering
    Appendix F.1 clusters reports with DBSCAN over MedCSPCLIP encodings to select representative reports; the clustering quality is not evaluated.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph." pith.science (2026). https://pith.science/paper/AFKDVRVI

@misc{pith2026250517214,
  author       = {Pith},
  title        = {Pith review of: MEDMKG: Benchmarking Medical Knowledge Exploitation with Multimodal Knowledge Graph},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/AFKDVRVI}},
  note         = {Machine review of arXiv:2505.17214}
}
read the original abstract

Medical deep learning models depend heavily on domain-specific knowledge to perform well on knowledge-intensive clinical tasks. Prior work has primarily leveraged unimodal knowledge graphs, such as the Unified Medical Language System (UMLS), to enhance model performance. However, integrating multimodal medical knowledge graphs remains largely underexplored, mainly due to the lack of resources linking imaging data with clinical concepts. To address this gap, we propose MEDMKG, a Medical Multimodal Knowledge Graph that unifies visual and textual medical information through a multi-stage construction pipeline. MEDMKG fuses the rich multimodal data from MIMIC-CXR with the structured clinical knowledge from UMLS, utilizing both rule-based tools and large language models for accurate concept extraction and relationship modeling. To ensure graph quality and compactness, we introduce Neighbor-aware Filtering (NaF), a novel filtering algorithm tailored for multimodal knowledge graphs. We evaluate MEDMKG across three tasks under two experimental settings, benchmarking twenty-four baseline methods and four state-of-the-art vision-language backbones on six datasets. Results show that MEDMKG not only improves performance in downstream medical tasks but also offers a strong foundation for developing adaptive and robust strategies for multimodal knowledge integration in medical artificial intelligence.

Figures

Figures reproduced from arXiv: 2505.17214 by the authors.

Figure 2
Figure 2. Distribution of entities involved in MEDMKG. The top 10 semantic types are shown individually, and rare types are grouped as “Others.” retrieval and multimodal visual question answering (VQA)–to demonstrate the practical benefits of integrating structured knowledge into diverse and complex clinical tasks. These evaluations highlight MEDMKG’s effectiveness in both structural understanding and knowledge-enhanced model… view at source ↗
Figure 3
Figure 3. shows a subgraph of MEDMKG, provided to facilitate a better understanding of its structure and content. As shown in [PITH_FULL_IMAGE:figures/full_fig_p024_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. KAMR: Grounding Generation via Knowledge-Aligned Multi-hop Retrieval

    cs.IR 2026-07 conditional novelty 6.0 of 10

    Partial-alignment contrastive pretraining plus anchor-then-expand graph retrieval improves multi-hop KG evidence recovery and downstream QA over strong dense and graph RAG baselines.

Reference graph

Works this paper leans on

89 extracted references · 59 canonical work pages · cited by 1 Pith paper

  1. [1]

    Recent advances in predictive modeling with electronic health records

    Jiaqi Wang, Junyu Luo, Muchao Ye, Xiaochen Wang, Yuan Zhong, Aofei Chang, Guanjie Huang, Ziyi Yin, Cao Xiao, Jimeng Sun, et al. Recent advances in predictive modeling with electronic health records. CoRR, 2024

    2024

  2. [2]

    Developing multimodal healthcare foundation model: From data-driven to knowledge-enhanced

    Xiaochen Wang. Developing multimodal healthcare foundation model: From data-driven to knowledge-enhanced. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 39, pages 29305–29306, 2025

    2025

  3. [3]

    Medpath: Augmenting health risk prediction via medical knowledge paths

    Muchao Ye, Suhan Cui, Yaqing Wang, Junyu Luo, Cao Xiao, and Fenglong Ma. Medpath: Augmenting health risk prediction via medical knowledge paths. In Proceedings of the Web Conference 2021, pages 1397–1409, 2021

    2021

  4. [4]

    Adverse drug reaction discovery using a tumor- biomarker knowledge graph

    Meng Wang, Xinyu Ma, Jingwen Si, Hongjia Tang, Haofen Wang, Tunliang Li, Wen Ouyang, Liying Gong, Yongzhong Tang, Xi He, et al. Adverse drug reaction discovery using a tumor- biomarker knowledge graph. Frontiers in genetics, 11:625659, 2021

    2021

  5. [5]

    Prediction of adverse drug reactions based on knowledge graph embedding

    Fei Zhang, Bo Sun, Xiaolin Diao, Wei Zhao, and Ting Shu. Prediction of adverse drug reactions based on knowledge graph embedding. BMC Medical Informatics and Decision Making , 21:1–11, 2021

    2021

  6. [6]

    Knowledge graph prediction of unknown adverse drug reactions and validation in electronic health records

    Daniel M Bean, Honghan Wu, Ehtesham Iqbal, Olubanke Dzahini, Zina M Ibrahim, Matthew Broadbent, Robert Stewart, and Richard JB Dobson. Knowledge graph prediction of unknown adverse drug reactions and validation in electronic health records. Scientific reports, 7(1):16416, 2017

    2017

  7. [7]

    CoRelation: Boosting Automatic ICD Coding Through Contextualized Code Relation Learning

    Junyu Luo, Xiaochen Wang, Jiaqi Wang, Aofei Chang, Yaqing Wang, and Fenglong Ma. Corelation: Boosting automatic icd coding through contextualized code relation learning. arXiv preprint arXiv:2402.15700, 2024

    work page Pith review arXiv 2024

  8. [8]

    Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports

    Alistair EW Johnson, Tom J Pollard, Seth J Berkowitz, Nathaniel R Greenbaum, Matthew P Lungren, Chih-ying Deng, Roger G Mark, and Steven Horng. Mimic-cxr, a de-identified publicly available database of chest radiographs with free-text reports. Scientific data, 6(1):317, 2019

    2019

Show all 89 references
  1. [9]

    The unified medical language system (umls): integrating biomedical terminology

    Olivier Bodenreider. The unified medical language system (umls): integrating biomedical terminology. Nucleic acids research, 32(suppl_1):D267–D270, 2004

    2004

  2. [10]

    Hierarchical pretraining on multimodal electronic health records

    Xiaochen Wang, Junyu Luo, Jiaqi Wang, Ziyi Yin, Suhan Cui, Yuan Zhong, Yaqing Wang, and Fenglong Ma. Hierarchical pretraining on multimodal electronic health records. In Proceedings of the Conference on Empirical Methods in Natural Language Processing. Conference on Empirical ...

    2023

  3. [11]

    Unity in diversity: Collaborative pre-training across multimodal medical sources

    Xiaochen Wang, Junyu Luo, Jiaqi Wang, Yuan Zhong, Xiaokun Zhang, Yaqing Wang, Parminder Bhatia, Cao Xiao, and Fenglong Ma. Unity in diversity: Collaborative pre-training across multimodal medical sources. In Proceedings of the 62nd Annual Meeting of the Association for Computa...

    2024

  4. [12]

    Raim: Recurrent attentive and intensive model of multimodal patient monitoring data

    Yanbo Xu, Siddharth Biswal, Shriprasad R Deshpande, Kevin O Maher, and Jimeng Sun. Raim: Recurrent attentive and intensive model of multimodal patient monitoring data. In Proceedings of the 24th ACM SIGKDD international conference on Knowledge Discovery & Data Mining, pages 25...

    2018

  5. [13]

    Synthesizing multimodal electronic health records via predictive diffusion models

    Yuan Zhong, Xiaochen Wang, Jiaqi Wang, Xiaokun Zhang, Yaqing Wang, Mengdi Huai, Cao Xiao, and Fenglong Ma. Synthesizing multimodal electronic health records via predictive diffusion models. In Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining...

    2024

  6. [14]

    Dcmn: Double core memory network for patient outcome prediction with multimodal data

    Yujuan Feng, Zhenxing Xu, Lin Gan, Ning Chen, Bin Yu, Ting Chen, and Fei Wang. Dcmn: Double core memory network for patient outcome prediction with multimodal data. In 2019 IEEE International Conference on Data Mining (ICDM), pages 200–209. IEEE, 2019. 10

    2019

  7. [15]

    Democratizing ehr analyses with fiddle: a flexible data-driven preprocessing pipeline for structured clinical data

    Shengpu Tang, Parmida Davarmanesh, Yanmeng Song, Danai Koutra, Michael W Sjoding, and Jenna Wiens. Democratizing ehr analyses with fiddle: a flexible data-driven preprocessing pipeline for structured clinical data. Journal of the American Medical Informatics Association, 27(12...

    1921

  8. [16]

    How to leverage multimodal ehr data for better medical predictions? arXiv preprint arXiv:2110.15763, 2021

    Bo Yang and Lijun Wu. How to leverage multimodal ehr data for better medical predictions? arXiv preprint arXiv:2110.15763, 2021

    2021 arXiv

  9. [17]

    padr: Towards personalized adverse drug reaction prediction by modeling multi-sourced data

    Junyu Luo, Cheng Qian, Xiaochen Wang, Lucas Glass, and Fenglong Ma. padr: Towards personalized adverse drug reaction prediction by modeling multi-sourced data. In Proceedings of the 32nd ACM International Conference on Information and Knowledge Management, pages 4724–4730, 2023

    2023

  10. [18]

    Llava-med: Training a large language-and-vision assistant for biomedicine in one day

    Chunyuan Li, Cliff Wong, Sheng Zhang, Naoto Usuyama, Haotian Liu, Jianwei Yang, Tristan Naumann, Hoifung Poon, and Jianfeng Gao. Llava-med: Training a large language-and-vision assistant for biomedicine in one day. Advances in Neural Information Processing Systems, 36, 2024

    2024

  11. [19]

    Med-flamingo: a multimodal medical few-shot learner

    Michael Moor, Qian Huang, Shirley Wu, Michihiro Yasunaga, Yash Dalmia, Jure Leskovec, Cyril Zakka, Eduardo Pontes Reis, and Pranav Rajpurkar. Med-flamingo: a multimodal medical few-shot learner. In Machine Learning for Health (ML4H), pages 353–367. PMLR, 2023

    2023

  12. [20]

    Knowledge-driven deep learning for fast mr imaging: Undersampled mr image reconstruction from supervised to un-supervised learning

    Shanshan Wang, Ruoyou Wu, Sen Jia, Alou Diakite, Cheng Li, Qiegen Liu, Hairong Zheng, and Leslie Ying. Knowledge-driven deep learning for fast mr imaging: Undersampled mr image reconstruction from supervised to un-supervised learning. Magnetic Resonance in Medicine, 92(2):496–...

    2024

  13. [21]

    Fedkim: Adaptive federated knowledge injection into medical foundation models

    Xiaochen Wang, Jiaqi Wang, Houping Xiao, Jinghui Chen, and Fenglong Ma. Fedkim: Adaptive federated knowledge injection into medical foundation models. In Proceedings of the 2024 Conference on Empirical Methods in Natural Language Processing, pages 8141–8154, 2024

    2024

  14. [22]

    Snomed-ct: The advanced terminology and coding system for ehealth

    Kevin Donnelly et al. Snomed-ct: The advanced terminology and coding system for ehealth. Studies in health technology and informatics, 121:279, 2006

    2006

  15. [23]

    Medical subject headings (mesh)

    Carolyn E Lipscomb. Medical subject headings (mesh). Bulletin of the Medical Library Association, 88(3):265, 2000

    2000

  16. [24]

    Drugbank: a comprehensive resource for in silico drug discovery and exploration

    David S Wishart, Craig Knox, An Chi Guo, Savita Shrivastava, Murtaza Hassanali, Paul Stothard, Zhan Chang, and Jennifer Woolsey. Drugbank: a comprehensive resource for in silico drug discovery and exploration. Nucleic acids research, 34(suppl_1):D668–D672, 2006

    2006

  17. [25]

    The human disease network

    Kwang-Il Goh, Michael E Cusick, David Valle, Barton Childs, Marc Vidal, and Albert-László Barabási. The human disease network. Proceedings of the National Academy of Sciences , 104(21):8685–8690, 2007

    2007

  18. [26]

    Learning entity and relation embeddings for knowledge graph completion

    Yankai Lin, Zhiyuan Liu, Maosong Sun, Yang Liu, and Xuan Zhu. Learning entity and relation embeddings for knowledge graph completion. In Proceedings of the AAAI conference on artificial intelligence, volume 29, 2015

    2015

  19. [27]

    Building a knowledge graph to enable precision medicine

    Payal Chandak, Kexin Huang, and Marinka Zitnik. Building a knowledge graph to enable precision medicine. Scientific Data, 10(1):67, 2023

    2023

  20. [28]

    Automated extraction of potential migraine biomarkers using a semantic graph

    Wytze J Vlietstra, Ronald Zielman, Robin M van Dongen, Erik A Schultes, Floris Wiesman, Rein V os, Erik M Van Mulligen, and Jan A Kors. Automated extraction of potential migraine biomarkers using a semantic graph. Journal of biomedical informatics, 71:178–189, 2017

    2017

  21. [29]

    Enriching contextualized language model from knowledge graph for biomedical information extraction

    Hao Fei, Yafeng Ren, Yue Zhang, Donghong Ji, and Xiaohui Liang. Enriching contextualized language model from knowledge graph for biomedical information extraction. Briefings in bioinformatics, 22(3):bbaa110, 2021

    2021

  22. [30]

    Constructing biomedical domain-specific knowledge graph with minimum supervision

    Jianbo Yuan, Zhiwei Jin, Han Guo, Hongxia Jin, Xianchao Zhang, Tristram Smith, and Jiebo Luo. Constructing biomedical domain-specific knowledge graph with minimum supervision. Knowledge and Information Systems, 62:317–336, 2020. 11

    2020

  23. [31]

    Covid-on-the-web: Knowledge graph and services to advance covid-19 research

    Franck Michel, Fabien Gandon, Valentin Ah-Kane, Anna Bobasheva, Elena Cabrio, Olivier Corby, Raphaël Gazzotti, Alain Giboin, Santiago Marro, Tobias Mayer, et al. Covid-on-the-web: Knowledge graph and services to advance covid-19 research. In The Semantic Web–ISWC 2020: 19th In...

    2020

  24. [32]

    Richpedia: a large-scale, compre- hensive multi-modal knowledge graph

    Meng Wang, Haofen Wang, Guilin Qi, and Qiushuo Zheng. Richpedia: a large-scale, compre- hensive multi-modal knowledge graph. Big Data Research, 22:100159, 2020

    2020

  25. [33]

    Multi- modal analogical reasoning over knowledge graphs

    Ningyu Zhang, Lei Li, Xiang Chen, Xiaozhuan Liang, Shumin Deng, and Huajun Chen. Multi- modal analogical reasoning over knowledge graphs. In The Eleventh International Conference on Learning Representations, 2022

    2022

  26. [34]

    Mmkg: multi-modal knowledge graphs

    Ye Liu, Hui Li, Alberto Garcia-Duran, Mathias Niepert, Daniel Onoro-Rubio, and David S Rosenblum. Mmkg: multi-modal knowledge graphs. In The Semantic Web: 16th International Conference, ESWC 2019, Portorož, Slovenia, June 2–6, 2019, Proceedings 16, pages 459–474. Springer, 2019

    2019

  27. [35]

    Tiva-kg: A multimodal knowledge graph with text, image, video and audio

    Xin Wang, Benyuan Meng, Hong Chen, Yuan Meng, Ke Lv, and Wenwu Zhu. Tiva-kg: A multimodal knowledge graph with text, image, video and audio. In Proceedings of the 31st ACM International Conference on Multimedia, pages 2391–2399, 2023

    2023

  28. [36]

    Answering visual-relational queries in web-extracted knowledge graphs

    Daniel Oñoro-Rubio, Mathias Niepert, Alberto García-Durán, Roberto González, and Roberto J López-Sastre. Answering visual-relational queries in web-extracted knowledge graphs. arXiv preprint arXiv:1709.02314, 2017

    2017 arXiv

  29. [37]

    An overview of metamap: historical perspective and recent advances

    Alan R Aronson and François-Michel Lang. An overview of metamap: historical perspective and recent advances. Journal of the American Medical Informatics Association, 17(3):229–236, 2010

    2010

  30. [38]

    Gpt-4 technical report

    J OpenAI Achiam, S Adler, S Agarwal, L Ahmad, I Akkaya, FL Aleman, D Almeida, J Altenschmidt, S Altman, S Anadkat, et al. Gpt-4 technical report. arxiv. arXiv preprint arXiv:2303.08774, 2023

    2023 arXiv

  31. [39]

    Redundancy, redundancy, re- dundancy: the three keys to highly robust anatomical parsing in medical images

    Xiang Sean Zhou, Zhigang Peng, Yiqiang Zhan, Maneesh Dewan, Bing Jian, Arun Krishnan, Yimo Tao, Martin Harder, Stefan Grosskopf, and Ute Feuerlein. Redundancy, redundancy, re- dundancy: the three keys to highly robust anatomical parsing in medical images. In Proceedings of the...

    2010

  32. [40]

    Translating embeddings for modeling multi-relational data

    Antoine Bordes, Nicolas Usunier, Alberto Garcia-Duran, Jason Weston, and Oksana Yakhnenko. Translating embeddings for modeling multi-relational data. Advances in neural information processing systems, 26, 2013

    2013

  33. [41]

    Knowledge graph embedding by translating on hyperplanes

    Zhen Wang, Jianwen Zhang, Jianlin Feng, and Zheng Chen. Knowledge graph embedding by translating on hyperplanes. In Proceedings of the AAAI conference on artificial intelligence, volume 28, 2014

    2014

  34. [42]

    Knowledge graph embedding via dynamic mapping matrix

    Guoliang Ji, Shizhu He, Liheng Xu, Kang Liu, and Jun Zhao. Knowledge graph embedding via dynamic mapping matrix. In Proceedings of the 53rd annual meeting of the association for computational linguistics and the 7th international joint conference on natural language processing...

    2015

  35. [43]

    Rotate: Knowledge graph embed- ding by relational rotation in complex space

    Zhiqing Sun, Zhi-Hong Deng, Jian-Yun Nie, and Jian Tang. Rotate: Knowledge graph embed- ding by relational rotation in complex space. arXiv preprint arXiv:1902.10197, 2019

    1902 arXiv

  36. [44]

    A three-way model for collective learning on multi-relational data

    Maximilian Nickel, V olker Tresp, Hans-Peter Kriegel, et al. A three-way model for collective learning on multi-relational data. In Icml, volume 11, pages 3104482–3104584, 2011

    2011

  37. [45]

    Embedding entities and relations for learning and inference in knowledge bases

    Bishan Yang, Wen-tau Yih, Xiaodong He, Jianfeng Gao, and Li Deng. Embedding entities and relations for learning and inference in knowledge bases. arXiv preprint arXiv:1412.6575, 2014

    2014 arXiv

  38. [46]

    Complex embeddings for simple link prediction

    Théo Trouillon, Johannes Welbl, Sebastian Riedel, Éric Gaussier, and Guillaume Bouchard. Complex embeddings for simple link prediction. In International conference on machine learning, pages 2071–2080. PMLR, 2016. 12

    2016

  39. [47]

    Simple embedding for link prediction in knowledge graphs

    Seyed Mehran Kazemi and David Poole. Simple embedding for link prediction in knowledge graphs. Advances in neural information processing systems, 31, 2018

    2018

  40. [48]

    Tucker: Tensor factorization for knowledge graph completion

    I Balaževic, C Allen, and TM Hospedales. Tucker: Tensor factorization for knowledge graph completion. arxiv 2019. arXiv preprint arXiv:1901.09590

    2019 arXiv

  41. [49]

    Hypernetwork knowledge graph embeddings

    Ivana Balaževi´c, Carl Allen, and Timothy M Hospedales. Hypernetwork knowledge graph embeddings. In Artificial Neural Networks and Machine Learning–ICANN 2019: Workshop and Special Sessions: 28th International Conference on Artificial Neural Networks, Munich, Germany, Septembe...

    2019

  42. [50]

    Convolutional 2d knowledge graph embeddings

    Tim Dettmers, Pasquale Minervini, Pontus Stenetorp, and Sebastian Riedel. Convolutional 2d knowledge graph embeddings. In Proceedings of the AAAI conference on artificial intelligence, volume 32, 2018

    2018

  43. [51]

    Adaptive convolution for multi-relational learning

    Xiaotian Jiang, Quan Wang, and Bin Wang. Adaptive convolution for multi-relational learning. In Proceedings of the 2019 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, Volume 1 (Long and Short Papers), pag...

    2019

  44. [52]

    Low-dimensional hyperbolic knowledge graph embeddings

    Ines Chami, Adva Wolf, Da-Cheng Juan, Frederic Sala, Sujith Ravi, and Christopher Ré. Low-dimensional hyperbolic knowledge graph embeddings. arXiv preprint arXiv:2005.00545, 2020

    2005 arXiv

  45. [53]

    Multi-relational poincaré graph embed- dings

    Ivana Balazevic, Carl Allen, and Timothy Hospedales. Multi-relational poincaré graph embed- dings. Advances in Neural Information Processing Systems, 32, 2019

    2019

  46. [54]

    Reasoning with neural tensor networks for knowledge base completion

    Richard Socher, Danqi Chen, Christopher D Manning, and Andrew Ng. Reasoning with neural tensor networks for knowledge base completion. Advances in neural information processing systems, 26, 2013

    2013

  47. [56]

    Design and development of a multimodal biomedical information retrieval system

    Dina Demner-Fushman, Sameer Antani, Matthew Simpson, and George R Thoma. Design and development of a multimodal biomedical information retrieval system. Journal of Computing Science and Engineering, 6(2):168–177, 2012

    2012

  48. [57]

    Preparing a collection of radiology examinations for distribution and retrieval

    Dina Demner-Fushman, Marc D Kohli, Marc B Rosenman, Sonya E Shooshan, Laritza Ro- driguez, Sameer Antani, George R Thoma, and Clement J McDonald. Preparing a collection of radiology examinations for distribution and retrieval. Journal of the American Medical Informatics Associ...

    2016

  49. [58]

    Learning transferable visual models from natural language supervision

    Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pa...

    2021

  50. [59]

    Sedigheh Eslami, Christoph Meinel, and Gerard De Melo. Pubmedclip: How much does clip benefit visual question answering in the medical domain? In Findings of the Association for Computational Linguistics: EACL 2023, pages 1181–1193, 2023

    2023

  51. [60]

    Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs

    Sheng Zhang, Yanbo Xu, Naoto Usuyama, Hanwen Xu, Jaspreet Bagga, Robert Tinn, Sam Preston, Rajesh Rao, Mu Wei, Naveen Valluri, et al. Biomedclip: a multimodal biomedical foundation model pretrained from fifteen million scientific image-text pairs. arXiv preprint arXiv:2303.00915, 2023

    2023 arXiv

  52. [61]

    Contrastive language- image pre-training with knowledge graphs

    Xuran Pan, Tianzhu Ye, Dongchen Han, Shiji Song, and Gao Huang. Contrastive language- image pre-training with knowledge graphs. Advances in Neural Information Processing Systems, 35:22895–22910, 2022. 13

    2022

  53. [62]

    FashionKLIP: Enhancing E-commerce image-text retrieval with fashion multi-modal conceptual knowledge graph

    Xiaodan Wang, Chengyu Wang, Lei Li, Zhixu Li, Ben Chen, Linbo Jin, Jun Huang, Yanghua Xiao, and Ming Gao. FashionKLIP: Enhancing E-commerce image-text retrieval with fashion multi-modal conceptual knowledge graph. In Sunayana Sitaram, Beata Beigman Klebanov, and Jason D Willia...

  54. [63]

    Decoupled weight decay regularization

    Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  55. [64]

    Overview of imageclef 2018 medical domain visual question answering task

    Sadid A Hasan, Yuan Ling, Oladimeji Farri, Joey Liu, Henning Müller, and Matthew Lungren. Overview of imageclef 2018 medical domain visual question answering task. Proceedings of CLEF 2018 Working Notes, 2018

    2018

  56. [65]

    A dataset of clinically generated visual questions and answers about radiology images

    Jason J Lau, Soumya Gayen, Asma Ben Abacha, and Dina Demner-Fushman. A dataset of clinically generated visual questions and answers about radiology images. Scientific data, 5(1):1–10, 2018

    2018

  57. [66]

    Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering

    Bo Liu, Li-Ming Zhan, Li Xu, Lin Ma, Yan Yang, and Xiao-Ming Wu. Slake: A semantically- labeled knowledge-enhanced dataset for medical visual question answering. In 2021 IEEE 18th International Symposium on Biomedical Imaging (ISBI), pages 1650–1654. IEEE, 2021

    2021

  58. [67]

    Pathvqa: 30000+ questions for medical visual question answering

    Xuehai He, Yichen Zhang, Luntian Mou, Eric Xing, and Pengtao Xie. Pathvqa: 30000+ questions for medical visual question answering. arXiv preprint arXiv:2003.10286, 2020

    2003 arXiv

  59. [68]

    Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa

    Kenneth Marino, Xinlei Chen, Devi Parikh, Abhinav Gupta, and Marcus Rohrbach. Krisp: Integrating implicit and symbolic knowledge for open-domain knowledge-based vqa. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14111–14121, 2021

    2021

  60. [69]

    Medical knowledge-based network for patient-oriented visual question answering

    Jian Huang, Yihao Chen, Yong Li, Zhenguo Yang, Xuehao Gong, Fu Lee Wang, Xiaohong Xu, and Wenyin Liu. Medical knowledge-based network for patient-oriented visual question answering. Information Processing & Management, 60(2):103241, 2023

    2023

  61. [70]

    K-pathvqa: Knowledge- aware multimodal representation for pathology visual question answering

    Usman Naseem, Matloob Khushi, Adam G Dunn, and Jinman Kim. K-pathvqa: Knowledge- aware multimodal representation for pathology visual question answering. IEEE Journal of Biomedical and Health Informatics, 2023

    2023

  62. [71]

    Ekgrl: Entity-based knowledge graph representation learning for fact-based visual question answering

    Yongjian Ren, Xiaotang Chen, and Kaiqi Huang. Ekgrl: Entity-based knowledge graph representation learning for fact-based visual question answering. In Chinese Conference on Pattern Recognition and Computer Vision (PRCV), pages 485–496. Springer, 2023

    2023

  63. [72]

    Multimodal reasoning with multimodal knowledge graph

    Junlin Lee, Yequan Wang, Jing Li, and Min Zhang. Multimodal reasoning with multimodal knowledge graph. arXiv preprint arXiv:2406.02030, 2024

    2024 arXiv

  64. [73]

    A density-based algorithm for discovering clusters in large spatial databases with noise

    Martin Ester, Hans-Peter Kriegel, Jörg Sander, Xiaowei Xu, et al. A density-based algorithm for discovering clusters in large spatial databases with noise. In kdd, volume 96, pages 226–231, 1996. 14 NeurIPS Paper Checklist

    1996

  65. [75]

    Guidelines: • The answer NA means that the abstract and introduction do not include the claims made in the paper

    Claims Question: Do the main claims made in the abstract and introduction accurately reflect the paper’s contributions and scope? Answer: [Yes] Justification: The claims made in the abstract and introduction are supported by Section 3 and Section 4. Guidelines: • The answer NA...

  66. [76]

    Limitations

    Limitations Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes] Justification: Please see the Appendix E. Guidelines: • The answer NA means that the paper has no limitation while the answer No means that the paper has limitation...

  67. [77]

    Guidelines: • The answer NA means that the paper does not include theoretical results

    Theory assumptions and proofs Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof? Answer: [NA] 15 Justification: This paper does not involve theory assumptions or proofs. Guidelines: • The answer NA mean...

  68. [78]

    Addi- tionally, we set the uniform seed (42) for all experiments to ensure reproducibility

    Experimental result reproducibility Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and da...

  69. [79]

    Guidelines: • The answer NA means that paper does not include experiments requiring code

    Open access to data and code Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material? 16 Answer: [Yes] Justification: We have provided links t...

  70. [80]

    • The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them

    Experimental setting/details Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes] Justification: Please check Section 4, Appen...

  71. [81]

    Instead, we set fixed random seed and provide reproduciable source codes to ensure the fair comparison

    Experiment statistical significance Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments? Answer: [No] Justification: Considering that this benchmark study contains quit...

  72. [82]

    Guidelines: • The answer NA means that the paper does not include experiments

    Experiments compute resources Question: For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed to reproduce the experiments? Answer: [Yes] Justification: Please see Appendix B f...

  73. [83]

    Guidelines: • The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics

    Code of ethics Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? Answer: [Yes] Justification: This work complies with the Code of Ethics. Guidelines: • The answer NA means t...

  74. [84]

    Guidelines: • The answer NA means that there is no societal impact of the work performed

    Broader impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [Yes] Justification: Please check Appendix A for the corresponding information. Guidelines: • The answer NA means that there i...

  75. [85]

    Guidelines: • The answer NA means that the paper poses no such risks

    Safeguards Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: This paper does...

  76. [86]

    Guidelines: • The answer NA means that the paper does not use existing assets

    Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] Justification: Please check App...

  77. [87]

    Guidelines: • The answer NA means that the paper does not release new assets

    New assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The documentation is available on the Hugging Face datacard. Guidelines: • The answer NA means that the paper does n...

  78. [88]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Crowdsourcing and research with human subjects Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)? A...

  79. [89]

    Guidelines: • The answer NA means that the paper does not involve crowdsourcing nor research with human subjects

    Institutional review board (IRB) approvals or equivalent for research with human subjects Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals...

  80. [90]

    findings

    Declaration of LLM usage 20 Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact t...

    2025

  81. [2023]

    Association for Computational Linguistics

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.