Pith. sign in

REVIEW 4 major objections 5 minor 2 references

Predicting Drug-Drug Interactions Using Heterogeneous Graph Neural Networks: HGNN-DDI

T0 review · 4 major / 5 minor · reviewed 2026-08-05 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read HGNN-DDI claims that a heterogeneous graph of drugs, proteins, and their interactions, with language-model embeddings, predicts drug-drug interaction types with up to 96.9% F1 on six grouped DrugBank classes.

desk verdict The headline SOTA claim evaporates under the paper's own realistic-negative evaluation; the architecture is a reasonable but incremental combination of existing components. read the letter →

arxiv 2508.18766 v1 pith:DDU7PQVJ submitted 2025-08-26 cs.LG

Hongbo Liu , Siyi Li , Zheng Yu This is my paper · ORCID
classification cs.LG
keywords drug-druginteractionpredictionheterogeneousgraphneuralnetworkChemBERTaESM-1blinkDrugBankPrimeKGmulti-classclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper proposes HGNN-DDI, a heterogeneous graph neural network that predicts drug-drug interaction types by combining ChemBERTa embeddings of drug SMILES strings with ESM-1b embeddings of protein sequences, then jointly modelling drug-drug, drug-protein, and protein-protein edges plus drug similarity edges in one graph. The authors try to establish that adding drug-protein and protein-protein context to a GCN/GAT link-prediction pipeline improves multi-class DDI accuracy beyond earlier graph, embedding, and language-model baselines on the DrugBank benchmark. On test data where only 10% of pairs are negatives, the best variant (HGCN) reaches 96.9% F1 for six grouped interaction classes and about 90% F1 for 86 fine classes. The paper's own appendix reports that when all negative pairs are included, F1 falls to 87.1%, and it notes that 93% of all pairs in the data have no interaction, so the headline numbers depend on the negative-sampling choice.

What carries the argument

The central object is a heterogeneous graph with two node types, drugs and proteins, and four edge types: drug-drug interactions, drug-protein interactions, protein-protein interactions, and a similarity edge between drugs whose Morgan-fingerprint similarity exceeds 0.7. Drug node features come from ChemBERTa encoding of SMILES strings, protein node features from ESM-1b encoding of amino-acid sequences, and a three-layer GCN (or GAT) propagates information across node and edge types; a final MLP classifies the interaction type for each drug pair. The key mechanism is that indirect paths through proteins and protein-protein interactions provide additional evidence for whether and how two drug

What would settle it

Run the same trained model on a test set that includes all negative pairs, or samples negatives at the natural 93% rate, and recompute F1 and accuracy; the paper's appendix already reports F1 falling from 0.969 to 0.871, and if under this distribution the model no longer outperforms the best baseline, the claim that HGNN-DDI beats state-of-the-art baselines is refuted.

Watch

Extended reading notes

Core claim

The authors claim that a heterogeneous knowledge graph containing drugs and proteins as nodes, with DDI, DPI, and PPI edges plus a drug-similarity edge from SMILES-based Morgan fingerprints, lets a three-layer GCN followed by an MLP link predictor achieve state-of-the-art DDI type prediction. On the six-class DrugBank task, they report F1 96.91, precision 97.01, recall 94.23, and accuracy 96.86 for the HGCN variant, and roughly 90% on the 86-class task; the HGAT variant performs nearly the same. They also report that including DPI and PPI information improves accuracy over using only DDI edges, and that the remaining performance bottleneck appears to be the dataset rather than the network ar

Load-bearing premise

The evaluation assumes that a test set with only 10% negative (no-interaction) pairs, randomly drawn from the complete test graph, represents the real DDI prediction problem; in the underlying data 93% of pairs have no interaction, so this sampling inflates all reported metrics.

Editorial extensions

If this is right

  • If the claimed result holds, adding target and protein interaction context is a direct way to boost DDI prediction without needing expensive 3D molecular conformations.
  • The architecture can be retrained as new drug-protein interaction data arrive, which could help hospital alert systems cover newer and less-studied drugs.
  • The near-identical performance of HGAT and HGCN suggests that heterogeneous graph structure, not the attention mechanism, is the main driver of the reported gains.
  • Grouping the 86 fine-grained DDI types into six mechanistic classes yields much higher accuracy, implying that a six-class screening layer may be the more practical clinical target.
  • The model's simple MLP classifier leaves room for richer decoders, so further gains may come from replacing the final classifier rather than the graph encoder.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • My reading: the reported 96.9% F1 is not directly comparable to real-world DDI screening because the test set contains only 10% negative pairs; the appendix value of 87.1% F1 on all negatives is a more honest estimate of performance on the natural 93%-negative distribution.
  • My reading: the similarity edges above 0.7 likely let the model generalize to unseen drugs by transductive similarity, but the paper does not isolate this effect; evaluating on held-out drug scaffolds would test that mechanism directly.
  • My reading: the paper's claim that the bottleneck is the dataset implies that harder negative sampling or a balanced, larger dataset should push F1 above 90% on the full distribution; a curriculum negative-sampling extension would test this.
  • My reading: because the six-class grouping obscures differences between specific interaction mechanisms, clinical use would require reporting per-class reliability in addition to the aggregate F1.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes HGNN-DDI, a heterogeneous graph neural network for drug-drug interaction (DDI) type prediction. Drug and protein nodes are embedded via ChemBERTa and ESM-1b; edges encode DDIs, drug-protein interactions, protein-protein interactions, and Morgan-fingerprint similarity above 0.7. A GCN or GAT encoder plus an MLP classifier predicts interaction types, either in six author-defined classes or the original 86 DrugBank types. On the DrugBank benchmark the authors report a maximum F1 of 0.969 (six-class HGCN) and claim state-of-the-art accuracy and robustness. The appendix acknowledges that when all negative samples are included, F1 drops to 0.871 and the 'no interaction' class is poorly predicted.

Significance. If the reported performance held under realistic test conditions, the model would be a useful contribution to DDI prediction. The architectural idea of combining ChemBERTa/ESM-1b features with a heterogeneous graph is reasonable, and the appendix's additional experiments with full negatives are a step toward honest evaluation. However, the headline claim is currently based on an artificial test distribution with only 10% negative samples, and the baseline comparisons are not controlled. The paper also provides no code, trained models, or processed data. The significance as stated is therefore not established; the merits of the architecture remain plausible but unverified.

major comments (4)
  1. [§3, negative-sampling paragraph; Eqs. (6)–(9)] The test protocol uses only 10% negative samples. Because Eqs. (6)–(9) weight per-class metrics by class frequency in the test set, this choice directly inflates the reported weighted F1 and accuracy. Appendix A states that 93% of drug pairs in the underlying graph are negative and that with all 400,000 negatives the model's F1 and accuracy drop to 0.871 and 0.854. The abstract and Section 3 claim that HGNN-DDI 'outperforms state-of-the-art baselines in prediction accuracy and robustness' without qualifying this artificial negative-sampling protocol. This is the central load-bearing problem because it invalidates the headline comparison.
  2. [§3, Table 2] Table 2 reports baseline numbers with no statement that the baselines use the same negative sampling ratio, same six-class taxonomy, same train/test split, or same graph structure. The six-class grouping is introduced in this paper, while published baselines such as KGNN were designed for 86-type DDI prediction. Without a controlled protocol, the 'outperforms' margin in Table 2 is uninterpretable; it may reflect evaluation setup rather than model quality.
  3. [Appendix A, Figures A.1 and A.3] The authors explicitly state that the model's predictions are 'not accurate enough for category 0' and that the no-interaction class is the majority class in reality. Since the practical task is to decide whether a drug pair interacts, a model that cannot detect the majority no-interaction class is not robust to the actual class imbalance. The claimed 'robustness' in the abstract is contradicted by the appendix. The paper should report per-class precision/recall at natural prevalence, not only weighted averages, and discuss the real operating regime for the model.
  4. [§2.4 and overall reproducibility] No code, processed graphs, hyperparameters, or data splits are provided. The key experiments cannot be reproduced from the text. For a claim of state-of-the-art performance, the evaluation protocol (negative sampling, split, class mapping) must be specified in full and ideally supported by released artifacts.
minor comments (5)
  1. [§2.2, Figure 4] The text says type 6 (effects) interactions are most frequent, but the Figure 4 caption says 'Type 49 interactions appear most frequently.' Please reconcile.
  2. [§2.3.3] The tool is called RDKit, not 'RDKits' or 'RDKits'; correct the term throughout.
  3. [References] Reference [15] appears to cite an ESM-2 bioRxiv paper ('Evolutionary-scale prediction of atomic level protein structure with a language model'); ESM-1b is introduced in Rives et al., PMLR 2021. Verify the correct reference.
  4. [§2.1.1 and §2.2] DrugBank counts are inconsistent: 192,284 DDIs / 191,878 drug pairs in Section 2.1.1, but 191,808 distinct DDIs in Section 2.2. Clarify which number is correct and how they were computed.
  5. [Equations (1)–(9)] The equations are typeset with stray symbols (e.g., 'accuracy!="#!$%"'). Please re-typeset all equations with standard LaTeX.

Circularity Check

0 steps flagged · score 0.0 of 10

No derivation-chain circularity; evaluation-protocol issues are validity concerns, not reduction-to-input.

full rationale

The paper's claimed derivation chain—from SMILES/protein features through ChemBERTa/ESM-1b embeddings, heterogeneous graph construction, GCN/GAT layers, and MLP classification—does not contain a step that reduces to its own inputs by construction. There is no self-citation serving as load-bearing support, no uniqueness theorem imported from the authors' prior work, and no ansatz smuggled in via citation. The headline SOTA claim is weakened by the 10% negative-sampling test protocol and the 6-class label grouping, as the paper's own Appendix A shows F1=0.871 and accuracy=0.854 when all 400,000 negatives are included; however, this is an experimental-design/validity limitation (non-representative test distribution, coarse label taxonomy, and undocumented baseline comparability) rather than circularity. Equations (6)–(9) compute weighted metrics on the test set as constructed, but the model's predictions are not fitted to those metrics, nor are the reported values equivalent to a fitted parameter by definition. The appendix's candid admission that category 0 is poorly predicted further confirms the issue is data imbalance and test-set composition, not a circular derivation.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The central performance claim rests on two author-controlled evaluation choices (10% negative sampling and the 6-class relabeling) and on uncritical acceptance of external database labels and pretrained embeddings. None of these are derived or validated independently in the paper.

free parameters (3)
  • negative_sample_ratio_test = 0.10
    Test set includes only 10% negative edges, far below the 93% true negative rate in the full graph, inflating accuracy, precision, recall, and F1.
  • molecular_similarity_threshold = 0.7
    Drug-drug similarity edges are kept only when RDKit Morgan similarity exceeds 0.7; this threshold is asserted without tuning or citation and changes the graph structure.
  • six_class_grouping = Absorption, distribution, metabolism, excretion, toxicity, effects
    The 86 DrugBank interaction types are coarsened into six categories defined by the authors, making the classification task easier and changing the meaning of the benchmark.
assumptions (4)
  • domain assumption DrugBank Multi-Typed DDI labels are correct and exhaustive for the 86 interaction types.
    The model is trained and evaluated against these labels; label errors or omissions propagate directly into the reported metrics.
  • domain assumption PrimeKG drug-protein and protein-protein relationships are complete and accurate for the 1,544 proteins and 1,706 drugs included.
    The heterogeneous graph relies on these edges for message passing; missing or incorrect relationships would distort drug representations.
  • domain assumption ChemBERTa SMILES embeddings and ESM-1b protein sequence embeddings capture enough molecular and functional information for DDI prediction.
    Node features are entirely generated by these pretrained models; if the embeddings miss key biological properties, the classifier cannot recover them.
  • ad hoc to paper Drugs with Morgan fingerprint similarity above 0.7 should be connected by an edge in the graph.
    The threshold is chosen by the authors without evidence and directly determines the adjacency structure used by the GNN.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Predicting Drug-Drug Interactions Using Heterogeneous Graph Neural Networks: HGNN-DDI." pith.science (2026). https://pith.science/paper/DDU7PQVJ

@misc{pith2026250818766,
  author       = {Pith},
  title        = {Pith review of: Predicting Drug-Drug Interactions Using Heterogeneous Graph Neural Networks: HGNN-DDI},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DDU7PQVJ}},
  note         = {Machine review of arXiv:2508.18766}
}
read the original abstract

Drug-drug interactions (DDIs) are a major concern in clinical practice, as they can lead to reduced therapeutic efficacy or severe adverse effects. Traditional computational approaches often struggle to capture the complex relationships among drugs, targets, and biological entities. In this work, we propose HGNN-DDI, a heterogeneous graph neural network model designed to predict potential DDIs by integrating multiple drug-related data sources. HGNN-DDI leverages graph representation learning to model heterogeneous biomedical networks, enabling effective information propagation across diverse node and edge types. Experimental results on benchmark DDI datasets demonstrate that HGNN-DDI outperforms state-of-the-art baselines in prediction accuracy and robustness, highlighting its potential to support safer drug development and precision medicine.

Figures

Figures reproduced from arXiv: 2508.18766 by the authors.

Figure 1
Figure 1. Meta path for Drug-Drug Interaction. Drug-Protein Interaction and Protein-Protein Interactions, which illustrates that there may be docking happening between drug-protein interactions. As illustrated in [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Meta path in dataset We used the DrugBank Multi-Typed DDI (TDcommons) dataset to predict potential drug-drug interactions. In addition, the Drug-Protein and Protein-Protein interactions data extracted from the PrimeKG database are used as a supplement for improved prediction. As portrayed in Figure 2a and Figure 2b, a brief illustration of DDIs, DPIs, and PPIs is provided. Brompheniramine, megestrol acetate, and met… view at source ↗
Figure 3
Figure 3. The architecture of the heterogeneous graph for two nodes and four edges with Drug-Drug interactions, Drug- Protein Interactions and Protein-Protein interactions. The heterogeneous graph is constructed as the general structure depicted in [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗
Figures from the paper (3 more)
Figure 4
Figure 4. Figure 4: Distribution of Interaction Types of Drug-Drug Interactions [PITH_FULL_IMAGE:figures/full_fig_p005_4.png]
Figure 5
Figure 5. Figure 5: Model Architecture of the HGNN-DDI for Drug-Drug Interactions Prediction Proceedings of the 4th International Conference on Signal Processing and Machine Learning DOI: 10.54254/2755-2721/79/20241329 82 [PITH_FULL_IMAGE:figures/full_fig_p006_5.png]
Figure 6
Figure 6. Figure 6: The f1 score, loss, precision and recall of HGNN-DDI model During the testing phase, negative samples constituted 10% of the entire test set, obtained by randomly sampling negative edges from the complete test graph. Commonly used evaluation metrics such as accuracy, p…

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

2 extracted references · 1 canonical work pages

  1. [1]

    #!$"%!"#!$

    Introduction Drug-drug interactions (DDIs), a prominent aspect of adverse drug reactions, occur when the combined effects of multiple medications diverge from the expected results if these drugs were used separately [1]. These interactions are broadly categorized into pharmacokinetic and pharmacodynamic types. Pharmacokinetic interactions involve changes ...

    work page arXiv 2024

  2. [14]

    No Interaction

    S. Chithrananda, G. Grand, B. Ramsundar, Chemberta: Large-scale self-supervised pretraining for molecular prop- erty prediction (2020). arXiv:2010.09885. [15] Z. Lin, H. Akin, R. Rao, B. Hie, Z. Zhu, et al, Evolutionary-scale prediction of atomic level protein structure with a language model, bioRxiv (2022). arXiv:https://www.biorxiv.org/content/early/202...

    arXiv 2020

Pith tools

Reviewed August 5, 2026 · model on record in the stance chip above.