{"id":"bee913dd-52eb-4036-a8b1-b3f135b7de88","arxiv_id":"2501.01644","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"Combining frozen biomedical language model embeddings with graph contrastive learning produces better initial node embeddings for biomedical knowledge graph link prediction.","lead":"This paper introduces PrimeKG++, a biomedical knowledge graph enriched with protein, gene, drug, and disease attributes, and a pipeline that combines language model embeddings with graph contrastive learning before link prediction. The authors report large gains over random initialization and direct LM embeddings on link prediction for PrimeKG++ and DrugBank drug-target interactions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"If the GCL pretraining stage consumes the full PrimeKG++ graph, validation/test edges leak into the very embeddings being compared, and the Table 2 gains over LM baselines may reflect data leakage rather than the method.","rationale":"The reader's weakest assumption identifies the same load-bearing point, and the text supports it. Section 4.4 describes GCL as operating on the graph with random node/edge masking only as augmentation; no mention is made of withholding the validation/test edge splits during pretraining, and Section 5.1's split description applies to the supervised link prediction stage. The default reading is therefore full-graph pretraining. If that is what the code does, the main comparison is compromised: the proposed method's initial embeddings are graph-informed while the LM baseline is not, so the AP/F1 gaps in Table 2 cannot be attributed to the method. This is not an internal inconsistency in the math; it is an unresolved procedural risk in the evaluation. The fix is straightforward and cheap: inspect the code or run the ablation. Given that, the conditional verdict is appropriate rather than a rejection, because the paper may well have performed the masking and simply failed to state it. I also note the absolute gains are small and reported without error bars or significance tests, but that is secondary to the leakage question.","tokens_in":17064,"tokens_out":4452,"duration_ms":44988,"concrete_test":"Check the released code and data at https://github.com/HySonLab/BioMedKG to see whether the GCL pretraining stage constructs its adjacency from the full PrimeKG++ edge set or only from the 60% training edges. If pretraining uses only training edges, the concern is resolved. If it uses full edges, rerun the best configuration (GRACE + ReDAF, 1:1) with GCL pretraining restricted to the training split and report AP/F1 on the untouched test split; if the gap over Direct LM-derived embeddings (Table 2: 0.996/0.983 vs 0.993/0.975) disappears or shrinks below noise, the headline claim fails. Additionally, evaluate on a node-disjoint split of DrugBank DTI to test the 'unseen nodes' wording.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that GCL-pretrained multimodal embeddings improve link prediction over random and direct LM initialization. For that claim to be meaningful, the graph information used to produce those embeddings must not include the validation or test edges whose prediction is later scored. Section 5.1 describes a 60/20/20 random split of edges for the supervised link prediction task, but nowhere states that the GCL pretraining stage (Section 4.4) masks these edges. The GCL models operate on the graph's adjacency structure (e.g., DGI, GGD, GRACE all consume A and X), and the full-graph adjacency is the default implementation. If GCL pretraining uses all PrimeKG++ edges, then test edges contribute to the node embeddings that initialize RGCN/DistMult; the supervised model can exploit test-edge topology through its initialization. Direct LM-derived embeddings contain no graph topology, so the comparison in Table 2 would be biased in favor of the proposed method by an amount that is unavailable from the reported numbers. The 'unseen nodes' claim is likewise unsupported by a random edge split: nodes in the test edges are present in the full node set and in GCL pretraining. This is a procedural premise, but it is load-bearing because the entire empirical case rests on it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a multimodal pretraining framework for biomedical knowledge graph link prediction. For each entity type, modality-specific language models (ProtBERT, DNABERT, MolFormer, BioBERT) produce attribute embeddings; a fusion module (attention or ReDAF) combines them; graph contrastive learning (DGI, GGD, or GRACE) pretrains representations on within-type subgraphs; and an RGCN encoder with a DistMult decoder performs supervised link prediction. The authors introduce PrimeKG++, a version of PrimeKG enriched with biological sequences and textual descriptions, and evaluate the pipeline on PrimeKG++ and the DrugBank drug-target interaction dataset, comparing random initialization, direct LM-derived embeddings, and nine GCL/fusion configurations at negative sampling ratios of 1:1, 1:3, and 1:5.","tokens_in":17336,"tokens_out":8599,"duration_ms":80841,"significance":"If the experimental comparison is leakage-free, the paper offers a useful and reproducible recipe: code, pretrained models, and the PrimeKG++ dataset are promised, and the configuration study (three GCL objectives, two learned fusion modules, multiple negative ratios, per-relation precision, embedding-size ablation, and a downstream XGBoost probe) is more thorough than is typical for a short empirical paper. The main contribution is not a new architecture but a pretrained-initialization strategy, and the paper is honest that it is not targeting state-of-the-art link prediction. However, the central claim rests on an unstated procedural premise: that GCL pretraining does not see validation or test edges. The text does not state this, and Section 5.3 says that PrimeKG++ was used during GCL pretraining. The 'unseen nodes' claim is also not established by the random edge split, and the DrugBank summary overstates Table 3. These issues make the current evidence conditional.","major_comments":[{"comment":"Section 4.4 defines the GCL module on the graph adjacency A and describes only random node/edge masking as augmentation; Section 5.1 describes a 60/20/20 random split of edges for the supervised link prediction model but never states that the same split is applied to the GCL pretraining stage. Since DGI, GGD, and GRACE all consume the adjacency structure, the default implementation would use all PrimeKG++ edges, including those that later appear in validation and test. The pretrained embeddings z_i would then encode test-edge topology before RGCN/DistMult is trained, biasing Table 2 in favor of the proposed method and making the comparison with random and LM-derived initialization misleading. The authors must state explicitly whether GCL pretraining masks validation/test edges; if it does not, the experiments need to be rerun with GCL pretraining on the training-only subgraph, and the abstract and conclusion claims revised accordingly.","section":"§4.4 and §5.1"},{"comment":"Section 5.1 randomly splits edges into 60/20/20, so every node present in a test edge is also present in the training graph and in GCL pretraining; no node is held out. Section 5.2.2 nevertheless claims the DrugBank experiments evaluate 'handling unseen nodes.' With a random edge split, nodes in the test set are seen at both the GCL and KGE stages, and only individual edges are held out, so the claim is not supported. To substantiate generalizability to unseen nodes, the authors need either to define 'unseen' precisely or to run an inductive split in which all edges incident to held-out nodes are removed from training and to report performance separately for those held-out nodes.","section":"§5.1 and §5.2.2"},{"comment":"The abstract and Section 6 state that the approach consistently outperforms baseline methods on DrugBank, but Table 3 contradicts this. At 1:1 negative sampling, Direct LM-derived embeddings reach AP 0.994 and F1 0.957, while GGD with None, Attention, and ReDAF reach only AP 0.985, 0.9862, and 0.9865 with F1 0.948, 0.951, and 0.954, respectively, and all DGI configurations have lower AP and F1. The text in Section 5.2.2 acknowledges that some configurations do not surpass LM-derived embeddings, so the abstract and conclusion should be revised to identify GRACE as the configuration family that consistently matches or exceeds the LM baseline, or should report an aggregate result with a significance test.","section":"Abstract, §5.2.2, Table 3"},{"comment":"All results in Tables 2 and 3 are reported as single point estimates, with no standard deviations, confidence intervals, number of seeds, or significance tests. The headline improvements at 1:1 are small, for example GRACE with ReDAF reaches AP 0.996 and F1 0.983 versus Direct LM AP 0.993 and F1 0.975, and configuration-to-configuration differences are even smaller. Without repeated runs, the claim of significant performance improvements is not established. Please report means and standard deviations over at least three random seeds, ideally with paired significance tests across configurations.","section":"Tables 2–3"},{"comment":"Section 2.3 cites several contrastive learning methods for knowledge graphs, including KGCL, KE-GCL, and MCLEA, and the paper claims novelty in applying multimodal contrastive learning to biomedical knowledge graphs, but Section 5.2 evaluates only random and LM initialization baselines under a single RGCN-DistMult backbone. The contribution would be better contextualized by comparing against at least one existing contrastive knowledge graph representation learning method, or a strong graph-pretraining baseline, using the same downstream protocol, or by narrowing the novelty claim to the specific pretraining-initialization setting. This is a scope issue rather than a correctness error, but it affects how a reader can interpret the novel framing.","section":"§2.3 and §5.2"}],"minor_comments":[{"comment":"The caption says 'DPI benchmark' while the text and Section 5.2.2 say 'DTI'; correct the acronym.","section":"Table 1"},{"comment":"The fused embedding is denoted u_i in the equation and then immediately written as u_i in R^D, while Table 5 defines h_i as the unified embedding; use one symbol consistently.","section":"§4.3"},{"comment":"The regularization term Lreg uses X for the encoder output and Z for relation embeddings, but Table 5 defines Z as the latent node representation; disambiguate the notation.","section":"§4.5"},{"comment":"Reference [18] is cited for BioGPT, but the listed paper is a different one by Lewis et al.; please provide the correct BioGPT citation or revise the sentence.","section":"References"},{"comment":"References [14] and [15] both cite DrugBank 6.0; merge them or clearly distinguish the benchmark dataset from the knowledgebase.","section":"References"},{"comment":"The equation for omega_m uses V, tanh, and zeta_r, but the following sentence introduces X_i, S_i, W_t, W_r, and C that do not appear in the displayed equations; align the notation.","section":"Appendix C.2"},{"comment":"The method is introduced as GGD but is later referred to as GD; use GGD consistently.","section":"Appendix D.2"},{"comment":"The caption refers to left and right panels, but the figure as included does not show panel labels; add (a) and (b) labels.","section":"Figure 3"},{"comment":"The Introduction says PrimeKG has more than 4 million relationships, while Table 1 reports 3,527,861 triples for PrimeKG++; clarify whether the difference is due to filtering, entity coverage, or counting of triples.","section":"Introduction"},{"comment":"The text says the most significant gains appear over baseline methods, but at 1:1 the gain over Direct LM is at most 0.003 in AP and 0.008 in F1; temper the wording or support it with significance evidence.","section":"§5.2.1"}],"recommendation":"major_revision","confidential_remarks":"The main risk is the unstated masking assumption in the GCL pretraining stage. Because the code and data are publicly released, this is checkable; I would ask the editor to require the authors to confirm or fix the masking first. If GCL used the full-graph adjacency, the PrimeKG++ results need to be rerun with a training-only subgraph. The DrugBank overstatement and the 'unseen nodes' claim are text-level fixes once the experimental protocol is clarified."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the useful thing here is PrimeKG++, and the pipeline is a competent assembly of known pieces. But the central comparison is undercut by an unaddressed leakage risk: GCL pretraining appears to operate on the full PrimeKG++ graph while val/test edges are only held out from the supervised link-prediction stage.\n\nWhat's actually new: PrimeKG++, an enriched version of PrimeKG with sequences and text for genes/drugs/diseases, is a concrete resource. The integration of frozen LMs, attention fusion, GCL, and RGCN/DistMult for biomedical link prediction is, as far as the citations show, new. The paper is honest that it is not chasing SOTA, and it ships code and data.\n\nWhat it does well: the ablation across GCL methods and fusion choices is thorough; the DrugBank transfer setup is a sensible generalization test; the appendix has more details than most.\n\nSoft spots: first, the leakage issue. Section 5.1 describes the 60/20/20 edge split for the KGE fine-tuning, but Section 4.4 never says GCL pretraining masks those edges. DGI, GGD, and GRACE all use the adjacency matrix, so unless explicitly hidden, val/test edges participate in the embeddings that initialize the supervised model. That biases the comparison against random and LM-only init. This is the load-bearing point and the paper needs to state and implement the masking, or the Table 2 gains are not interpretable. Second, 'unseen nodes' is unsupported by a random edge split; nodes in test edges are in the full node set. Third, Table 3 contradicts the text: several GGD/DGI configurations score below Direct LM, so 'consistently outperforms' is wrong, and the paper should say so. Fourth, no error bars; the margins in Table 2 are often 0.003–0.005, which without variance could be noise.\n\nThese are all fixable. The dataset and pipeline deserve a serious look, but the main empirical claim needs a clean evaluation first. I'd send to peer review and require the leakage question be resolved, error bars added, and the overclaims trimmed.","headline":"PrimeKG++ is a genuinely useful resource and the pipeline is plausible, but the main comparison is compromised by an unaddressed GCL test-edge leakage risk and overclaimed 'unseen nodes' generality.","tokens_in":17871,"tokens_out":2136,"would_cite":true,"duration_ms":20627,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretrained multimodal embeddings improve biomedical link prediction, including for unseen nodes, by combining language-model vectors with graph contrastive learning.","keywords":["biomedical knowledge graphs","link prediction","graph contrastive learning","language model embeddings","multimodal fusion","PrimeKG++","drug-target interaction","drug repurposing"],"falsifier":"Run the same GRACE/ReDAF and RGCN/DistMult pipeline with validation and test edges explicitly removed from the graph used for GCL pretraining; if PrimeKG++ AP and F1 fall to the direct-LM baseline levels, the reported gain is leakage rather than representation quality.","tokens_in":16853,"feed_emoji":"🧬","tokens_out":6541,"duration_ms":60482,"temperature":0.7,"pith_summary":"The paper tries to show that initializing a biomedical link-prediction model with pretrained node embeddings, rather than random vectors or plain language-model vectors, makes prediction measurably better. The embeddings are built by taking modality-specific language-model representations of each entity (protein sequences, gene sequences, SMILES strings, and text descriptions), fusing them, and refining them with graph contrastive learning before supervised link prediction. If true, this gives a practical recipe: pretrain once on a rich graph like the new PrimeKG++ resource, then reuse the embeddings to get higher average precision and F1 on biomedical link prediction, including on entities not seen during supervised training. The paper also contributes PrimeKG++, an augmented version of PrimeKG with added sequences and descriptions for genes, proteins, and drugs.","feed_headline":"Pretrained embeddings lift biomedical link prediction to 0.996 AP","feed_subtitle":"Graph-contrastive pretraining beats random and direct language-model baselines, and carries to unseen DrugBank nodes.","key_machinery":"The load-bearing mechanism is a two-stage representation pipeline. First, each entity's modalities are encoded by frozen domain-specific language models—protein sequences, nucleotide sequences, SMILES strings, and text descriptions—and combined by an attention-based or relation-guided dual adaptive fusion module into one embedding per node. Second, graph contrastive learning, a self-supervised technique that maximizes agreement between two augmented views of the same graph, aligns embeddings of nodes of the same type; the paper instantiates this with DGI, GGD, and GRACE. The resulting vectors initialize a relational graph convolutional network whose DistMult scoring function is trained for link prediction. PrimeKG++ supplies the multimodal node attributes that make this pipeline possible.","core_discovery":"The central claim is that the fusion of LM-derived semantic embeddings with graph contrastive learning produces node representations that outperform both random initialization and direct LM embeddings when used as input to an RGCN encoder with a DistMult decoder for link prediction. On PrimeKG++ the best configuration (GRACE with ReDAF) reaches 0.996 AP and 0.983 F1 at a 1:1 negative sampling ratio, and the gap over baselines widens at harder 1:3 and 1:5 ratios. On the DrugBank drug-target interaction dataset the same pretrained representations generalize to a different graph with novel nodes, again beating random initialization and generally matching or exceeding direct LM embeddings. The paper further claims that the combined semantic-plus-relational embeddings are accurate for unseen nodes, a property the authors attribute to graph contrastive learning capturing topology that standalone LM vectors lack.","pith_inferences":["The paper does not state whether the graph-contrastive pretraining stage masks validation and test edges; if it does not, the pretrained embeddings could have seen test structure, so an explicit masking protocol is the first check on the reported gains.","The claim of generalization to unseen nodes rests on DrugBank sharing entity identities with PrimeKG++; a sharper test would remove all overlapping entities and measure whether the performance advantage survives.","Because the language models are frozen and the contrastive module adds only about 164K parameters, the recipe suggests that cheap self-supervised pretraining on a rich graph can substitute for training embeddings from scratch in low-resource biomedical settings."],"forward_implications":["Users of biomedical link prediction can replace random or LM-only initializations with these pretrained embeddings and expect higher AP and F1 at no change to the downstream RGCN/DistMult model.","The benefit is largest when negative sampling is hard at 1:3 and 1:5, where the direct-LM baseline degrades more sharply than the pretrained configurations.","The same pretrained embeddings transfer to a second dataset, DrugBank, whose nodes and relations differ from PrimeKG++, supporting the paper's claim of generalization to unseen nodes.","PrimeKG++ becomes a public resource for evaluating multimodal biomedical knowledge graph methods, since it adds sequences and text to PrimeKG's existing structure.","The framework is easily extended to other relation types such as drug-disease or protein-disease interactions, according to the authors."],"supporting_citations":[{"why":"Supplies PrimeKG, the base knowledge graph that PrimeKG++ augments with sequences and descriptions.","marker":"[2]"},{"why":"Provides ProteinBERT, the encoder for amino-acid sequences of proteins.","marker":"[1]"},{"why":"Provides DNABERT, the encoder for gene and nucleotide sequences.","marker":"[12]"},{"why":"Provides MoLFormer, the encoder for SMILES strings of small molecules.","marker":"[27]"},{"why":"Provides BioBERT, the encoder for textual descriptions of all entity types.","marker":"[17]"},{"why":"Provides the Relational Graph Convolutional Network used as the link-prediction encoder.","marker":"[28]"},{"why":"Provides the DistMult scoring function used to rank candidate triples.","marker":"[36]"},{"why":"Provides GRACE, the graph contrastive learning model that produces the best-performing configurations.","marker":"[43]"},{"why":"Provides the Relation-guided Dual Adaptive Fusion (ReDAF) method used to combine modalities.","marker":"[40]"},{"why":"The DrugBank drug-target interaction dataset used to test generalization to unseen nodes.","marker":"[14]"}],"fun_headline_variants":["Contrastive LM fusion hits 0.996 AP on biomedical links","Graph-contrastive pretraining yields 0.996 AP and generalizes to unseen nodes","0.996 AP: Multimodal contrastive learning on biomedical KGs","Graph-contrastive language fusion beats baselines on drug-target links"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that the graph-contrastive pretraining stage never trains on the edges that later appear in validation and test splits, so those edges cannot leak into the pretrained node embeddings.","fun_headline_variants_meta":{"raw":{"variants":["Contrastive LM fusion hits 0.996 AP on biomedical links","Graph-contrastive pretraining yields 0.996 AP and generalizes to unseen nodes","0.996 AP: Multimodal contrastive learning on biomedical KGs","Graph-contrastive language fusion beats baselines on drug-target links"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001216,"raw_usage":{"total_tokens":4981,"prompt_tokens":901,"completion_tokens":4080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":3995}},"tokens_in":517,"tokens_out":4080,"duration_ms":27842,"temperature":1.0,"reasoning_tokens":3995,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T22:23:36.492904+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same GRACE/ReDAF and RGCN/DistMult pipeline with validation and test edges explicitly removed from the graph used for GCL pretraining; if PrimeKG++ AP and F1 fall to the direct-LM baseline levels, the reported gain is leakage rather than representation quality.","supporting_citations":[{"cited_title":"Building a knowl- edge graph to enable precision medicine","cited_arxiv_id":null,"evidence_quote":"Supplies PrimeKG, the base knowledge graph that PrimeKG++ augments with sequences and descriptions."},{"cited_title":"ProteinBERT: a universal deep-learning model of protein se- quence and function","cited_arxiv_id":null,"evidence_quote":"Provides ProteinBERT, the encoder for amino-acid sequences of proteins."},{"cited_title":"DNABERT: pre-trained Bidirectional Encoder Representations from Transformers model for DNA-language in genome","cited_arxiv_id":null,"evidence_quote":"Provides DNABERT, the encoder for gene and nucleotide sequences."},{"cited_title":"Large-scale chemical language repre- sentations capture molecular structure and properties.Nature Machine Intelligence, 4(12):1256–1264, 2022","cited_arxiv_id":null,"evidence_quote":"Provides MoLFormer, the encoder for SMILES strings of small molecules."},{"cited_title":"Biobert: a pre-trained biomedical language representation model for biomedical text mining","cited_arxiv_id":null,"evidence_quote":"Provides BioBERT, the encoder for textual descriptions of all entity types."},{"cited_title":"Embedding entities and relations for learning and inference in knowl- edge bases, 2015","cited_arxiv_id":null,"evidence_quote":"Provides the DistMult scoring function used to rank candidate triples."},{"cited_title":"Deep graph contrastive representation learning, 2020","cited_arxiv_id":null,"evidence_quote":"Provides GRACE, the graph contrastive learning model that produces the best-performing configurations."},{"cited_title":"Native: Multi-modal knowledge graph completion in the wild","cited_arxiv_id":null,"evidence_quote":"Provides the Relation-guided Dual Adaptive Fusion (ReDAF) method used to combine modalities."},{"cited_title":"Drugbank 6.0: the drugbank knowledgebase for","cited_arxiv_id":null,"evidence_quote":"The DrugBank drug-target interaction dataset used to test generalization to unseen nodes."}],"review_version":1}