{"id":"09975d11-3b27-427e-997e-9a8ffe1db869","arxiv_id":"2508.18766","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"HGNN-DDI claims state-of-the-art drug-drug interaction prediction, but its skewed test set and missing SOTA baselines make the performance claim unsupported.","lead":"This paper describes a machine learning model, HGNN-DDI, that predicts how drug pairs interact by combining drug and protein data in a heterogeneous graph with GNN layers and pretrained language models. The reported near-97% F1 score is undermined by a test set with only 10% negative pairs and by the absence of the strongest competitor models in the comparison.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Claimed SOTA rests on 10%-negative test set; at natural 93% negative prevalence Appendix A shows F1 drops to 0.871 and no-interaction class is poorly predicted.","rationale":"The reader's weakest_assumption correctly identifies the evaluation's negative-sampling design as the central vulnerability. The paper's own Appendix A acknowledges the issue: only 10% negative samples are used, the true negative rate is 93%, and the model misclassifies no-interaction cases. The reported 0.969 F1 is therefore an artifact of the artificially easy test distribution, not evidence of real-world predictive superiority. The abstract's unqualified 'outperforms state-of-the-art' claim cannot be accepted on this evidence. I considered whether another issue—missing baselines or lack of code—is more load-bearing, but those are secondary: the negative-prevalence mismatch alone invalidates the headline comparison. The concrete test would settle it by reproducing the evaluation at natural prevalence and comparing baselines under identical conditions. Since this is exactly the reader's concern and supports the reader's REJECT verdict, no verdict adjustment is needed.","tokens_in":8940,"tokens_out":4071,"duration_ms":39870,"concrete_test":"Re-run the full evaluation on the same train/validation/test split but with negative edges sampled at natural prevalence (93% non-interactions) from the candidate drug-pair space, and compute macro-F1, precision/recall for the no-interaction class, and AUPRC. Apply the identical protocol to MUFFIN, 3DGT-DDI, and DeepDDI. If HGNN-DDI's F1 falls toward the 0.871 reported in Figure A.3, or any baseline exceeds it under this protocol, the abstract's state-of-the-art claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that HGNN-DDI 'outperforms state-of-the-art baselines in prediction accuracy and robustness' is supported only by metrics computed on a test set in which negatives are 10% of samples, whereas the underlying DrugBank graph has 93% non-interacting drug pairs. With so few negatives, the six-class F1 of 0.969 does not measure the model's ability to detect the majority no-interaction class; the paper's own Appendix A states that the no-interaction class is poorly predicted and reports F1=0.871, accuracy=0.854 when all 400,000 negatives are included. The abstract and Section 3 do not qualify the claim with this artificial negative-sampling protocol. In addition, Table 2's baseline numbers are not documented as using the same negative prevalence, train/test split, or class taxonomy (6 vs 86 classes), so the 'outperforms' margin may be an artifact of evaluation setup rather than model quality. This is the load-bearing weakness because if the evaluation were run at realistic prevalence, the headline result disappears; Appendix A's own numbers demonstrate it. No code or processed data is provided, making independent verification impossible.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes HGNN-DDI, a heterogeneous graph neural network for drug-drug interaction (DDI) type prediction. Drug and protein nodes are embedded via ChemBERTa and ESM-1b; edges encode DDIs, drug-protein interactions, protein-protein interactions, and Morgan-fingerprint similarity above 0.7. A GCN or GAT encoder plus an MLP classifier predicts interaction types, either in six author-defined classes or the original 86 DrugBank types. On the DrugBank benchmark the authors report a maximum F1 of 0.969 (six-class HGCN) and claim state-of-the-art accuracy and robustness. The appendix acknowledges that when all negative samples are included, F1 drops to 0.871 and the 'no interaction' class is poorly predicted.","tokens_in":9222,"tokens_out":4483,"duration_ms":46108,"significance":"If the reported performance held under realistic test conditions, the model would be a useful contribution to DDI prediction. The architectural idea of combining ChemBERTa/ESM-1b features with a heterogeneous graph is reasonable, and the appendix's additional experiments with full negatives are a step toward honest evaluation. However, the headline claim is currently based on an artificial test distribution with only 10% negative samples, and the baseline comparisons are not controlled. The paper also provides no code, trained models, or processed data. The significance as stated is therefore not established; the merits of the architecture remain plausible but unverified.","major_comments":[{"comment":"The test protocol uses only 10% negative samples. Because Eqs. (6)–(9) weight per-class metrics by class frequency in the test set, this choice directly inflates the reported weighted F1 and accuracy. Appendix A states that 93% of drug pairs in the underlying graph are negative and that with all 400,000 negatives the model's F1 and accuracy drop to 0.871 and 0.854. The abstract and Section 3 claim that HGNN-DDI 'outperforms state-of-the-art baselines in prediction accuracy and robustness' without qualifying this artificial negative-sampling protocol. This is the central load-bearing problem because it invalidates the headline comparison.","section":"§3, negative-sampling paragraph; Eqs. (6)–(9)"},{"comment":"Table 2 reports baseline numbers with no statement that the baselines use the same negative sampling ratio, same six-class taxonomy, same train/test split, or same graph structure. The six-class grouping is introduced in this paper, while published baselines such as KGNN were designed for 86-type DDI prediction. Without a controlled protocol, the 'outperforms' margin in Table 2 is uninterpretable; it may reflect evaluation setup rather than model quality.","section":"§3, Table 2"},{"comment":"The authors explicitly state that the model's predictions are 'not accurate enough for category 0' and that the no-interaction class is the majority class in reality. Since the practical task is to decide whether a drug pair interacts, a model that cannot detect the majority no-interaction class is not robust to the actual class imbalance. The claimed 'robustness' in the abstract is contradicted by the appendix. The paper should report per-class precision/recall at natural prevalence, not only weighted averages, and discuss the real operating regime for the model.","section":"Appendix A, Figures A.1 and A.3"},{"comment":"No code, processed graphs, hyperparameters, or data splits are provided. The key experiments cannot be reproduced from the text. For a claim of state-of-the-art performance, the evaluation protocol (negative sampling, split, class mapping) must be specified in full and ideally supported by released artifacts.","section":"§2.4 and overall reproducibility"}],"minor_comments":[{"comment":"The text says type 6 (effects) interactions are most frequent, but the Figure 4 caption says 'Type 49 interactions appear most frequently.' Please reconcile.","section":"§2.2, Figure 4"},{"comment":"The tool is called RDKit, not 'RDKits' or 'RDKits'; correct the term throughout.","section":"§2.3.3"},{"comment":"Reference [15] appears to cite an ESM-2 bioRxiv paper ('Evolutionary-scale prediction of atomic level protein structure with a language model'); ESM-1b is introduced in Rives et al., PMLR 2021. Verify the correct reference.","section":"References"},{"comment":"DrugBank counts are inconsistent: 192,284 DDIs / 191,878 drug pairs in Section 2.1.1, but 191,808 distinct DDIs in Section 2.2. Clarify which number is correct and how they were computed.","section":"§2.1.1 and §2.2"},{"comment":"The equations are typeset with stray symbols (e.g., 'accuracy!=\"#!$%\"'). Please re-typeset all equations with standard LaTeX.","section":"Equations (1)–(9)"}],"recommendation":"major_revision","confidential_remarks":"The paper reads as a lightly revised conference proceeding. The central superiority claim rests on a nonstandard test distribution and uncontrolled baselines. These issues are fixable by rerunning the evaluation at natural prevalence with matched baselines and by adjusting the abstract and conclusions accordingly. If the journal requires the headline claim to hold under controlled comparison, the revision is substantial, but it is within the scope of a serious revision. I therefore recommend major revision rather than outright rejection, despite the current evidence being insufficient."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Dear Colleague,\n\nThe thing to know: the paper claims SOTA DDI prediction with F1 0.969, but that number comes from a test set where negatives are only 10% of samples, while the actual DrugBank graph is 93% negative. The paper's own Appendix A reports F1=0.871 at realistic prevalence and admits the no-interaction class is poorly predicted. So the abstract's 'outperforms state-of-the-art' is unsupported.\n\nWhat's worth credit: the model sensibly combines ChemBERTa drug embeddings, ESM-1b protein embeddings, and a heterogeneous graph with DDI/DPI/PPI edges plus a similarity threshold. That's a workable pipeline. The authors also deserve credit for including the negative-sampling ablation in the appendix, even if the main text continues to lead with the favorable numbers.\n\nThe soft spots are load-bearing. The 10% negative sampling makes the task artificial; the 6-class grouping of the 86 interaction types is a custom relabeling that isn't validated; and Table 2 gives no indication that the baselines were evaluated under the same split, prevalence, or class taxonomy. No code or processed data is released, so the comparisons are not independently checkable. The novelty is incremental: GCN/GAT with pretrained embeddings on a standard dataset, with the contributions mostly in feature choice rather than a new method.\n\nWho is this for? A reader looking for a cautionary example of how evaluation setup can manufacture performance. The paper is honest enough in the appendix to be useful for teaching, but it does not support its central claim.\n\nRecommendation: I would not send this to peer review as-is. The flaw is acknowledged in the paper itself, and the headline claim is not defensible. If the authors redo the evaluation at realistic prevalence and benchmark against MUFFIN/3DGT-DDI with matched protocols, there might be a modest empirical note worth publishing. As it stands, the paper is a workshop-level report with a serious evaluation gap.","headline":"The headline SOTA claim evaporates under the paper's own realistic-negative evaluation; the architecture is a reasonable but incremental combination of existing components.","tokens_in":9708,"tokens_out":2446,"would_cite":false,"duration_ms":24534,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"HGNN-DDI claims that a heterogeneous graph of drugs, proteins, and their interactions, with language-model embeddings, predicts drug-drug interaction types with up to 96.9% F1 on six grouped DrugBank classes.","keywords":["drug-drug interaction prediction","heterogeneous graph neural network","ChemBERTa","ESM-1b","link prediction","DrugBank","PrimeKG","multi-class classification"],"falsifier":"Run the same trained model on a test set that includes all negative pairs, or samples negatives at the natural 93% rate, and recompute F1 and accuracy; the paper's appendix already reports F1 falling from 0.969 to 0.871, and if under this distribution the model no longer outperforms the best baseline, the claim that HGNN-DDI beats state-of-the-art baselines is refuted.","tokens_in":8813,"feed_emoji":"💊","tokens_out":4803,"duration_ms":47651,"temperature":0.7,"pith_summary":"This paper proposes HGNN-DDI, a heterogeneous graph neural network that predicts drug-drug interaction types by combining ChemBERTa embeddings of drug SMILES strings with ESM-1b embeddings of protein sequences, then jointly modelling drug-drug, drug-protein, and protein-protein edges plus drug similarity edges in one graph. The authors try to establish that adding drug-protein and protein-protein context to a GCN/GAT link-prediction pipeline improves multi-class DDI accuracy beyond earlier graph, embedding, and language-model baselines on the DrugBank benchmark. On test data where only 10% of pairs are negatives, the best variant (HGCN) reaches 96.9% F1 for six grouped interaction classes and about 90% F1 for 86 fine classes. The paper's own appendix reports that when all negative pairs are included, F1 falls to 87.1%, and it notes that 93% of all pairs in the data have no interaction, so the headline numbers depend on the negative-sampling choice.","feed_headline":"Heterogeneous drug-protein graph predicts DDI types at 96.9% F1","feed_subtitle":"Combining SMILES, protein sequences, and PPI/DPI edges beats earlier baselines on the six-class DrugBank benchmark.","key_machinery":"The central object is a heterogeneous graph with two node types, drugs and proteins, and four edge types: drug-drug interactions, drug-protein interactions, protein-protein interactions, and a similarity edge between drugs whose Morgan-fingerprint similarity exceeds 0.7. Drug node features come from ChemBERTa encoding of SMILES strings, protein node features from ESM-1b encoding of amino-acid sequences, and a three-layer GCN (or GAT) propagates information across node and edge types; a final MLP classifies the interaction type for each drug pair. The key mechanism is that indirect paths through proteins and protein-protein interactions provide additional evidence for whether and how two drug","core_discovery":"The authors claim that a heterogeneous knowledge graph containing drugs and proteins as nodes, with DDI, DPI, and PPI edges plus a drug-similarity edge from SMILES-based Morgan fingerprints, lets a three-layer GCN followed by an MLP link predictor achieve state-of-the-art DDI type prediction. On the six-class DrugBank task, they report F1 96.91, precision 97.01, recall 94.23, and accuracy 96.86 for the HGCN variant, and roughly 90% on the 86-class task; the HGAT variant performs nearly the same. They also report that including DPI and PPI information improves accuracy over using only DDI edges, and that the remaining performance bottleneck appears to be the dataset rather than the network ar","pith_inferences":["My reading: the reported 96.9% F1 is not directly comparable to real-world DDI screening because the test set contains only 10% negative pairs; the appendix value of 87.1% F1 on all negatives is a more honest estimate of performance on the natural 93%-negative distribution.","My reading: the similarity edges above 0.7 likely let the model generalize to unseen drugs by transductive similarity, but the paper does not isolate this effect; evaluating on held-out drug scaffolds would test that mechanism directly.","My reading: the paper's claim that the bottleneck is the dataset implies that harder negative sampling or a balanced, larger dataset should push F1 above 90% on the full distribution; a curriculum negative-sampling extension would test this.","My reading: because the six-class grouping obscures differences between specific interaction mechanisms, clinical use would require reporting per-class reliability in addition to the aggregate F1."],"forward_implications":["If the claimed result holds, adding target and protein interaction context is a direct way to boost DDI prediction without needing expensive 3D molecular conformations.","The architecture can be retrained as new drug-protein interaction data arrive, which could help hospital alert systems cover newer and less-studied drugs.","The near-identical performance of HGAT and HGCN suggests that heterogeneous graph structure, not the attention mechanism, is the main driver of the reported gains.","Grouping the 86 fine-grained DDI types into six mechanistic classes yields much higher accuracy, implying that a six-class screening layer may be the more practical clinical target.","The model's simple MLP classifier leaves room for richer decoders, so further gains may come from replacing the final classifier rather than the graph encoder."],"supporting_citations":[{"why":"Supplies the DrugBank Multi-Typed DDI gold-standard dataset used for training and evaluation.","marker":"[11]"},{"why":"Supplies the PrimeKG graph from which drug-protein and protein-protein interactions are extracted.","marker":"[13]"},{"why":"Supplies ChemBERTa, the pretrained language model that encodes SMILES strings into drug node embeddings.","marker":"[14]"},{"why":"Supplies ESM-1b, the protein language model used to create protein node embeddings.","marker":"[15]"},{"why":"Supplies RDKit's Morgan fingerprint and similarity calculation used to build drug similarity edges.","marker":"[12]"},{"why":"Provides the GCN-and-GAT-on-heterogeneous-graph methodology that the model adapts for DDI prediction.","marker":"[17]"},{"why":"Introduces the knowledge-graph GNN approach for DDI prediction that this work extends with DPI and PPI edges.","marker":"[7]"},{"why":"Serves as a state-of-the-art baseline combining molecular graphs and knowledge graphs for multi-class DDI prediction.","marker":"[8]"},{"why":"Serves as a baseline combining 3D molecular structure and text features for DDI prediction.","marker":"[9]"},{"why":"Supplies an attention-based GNN on drug molecular graphs, a direct architectural predecessor for the graph encoder.","marker":"[16]"}],"fun_headline_variants":["Heterogeneous graph model predicts drug-drug interactions at 96.9% F1","HGNN-DDI: Graph neural nets beat baselines on drug interaction prediction","Integrating drug-protein edges boosts DDI prediction to 96.9% F1","Three-layer GCN on drug-protein graph achieves SOTA DDI prediction","Drug-protein heterogeneous graph improves DDI type classification"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The evaluation assumes that a test set with only 10% negative (no-interaction) pairs, randomly drawn from the complete test graph, represents the real DDI prediction problem; in the underlying data 93% of pairs have no interaction, so this sampling inflates all reported metrics.","fun_headline_variants_meta":{"raw":{"variants":["Heterogeneous graph model predicts drug-drug interactions at 96.9% F1","HGNN-DDI: Graph neural nets beat baselines on drug interaction prediction","Integrating drug-protein edges boosts DDI prediction to 96.9% F1","Three-layer GCN on drug-protein graph achieves SOTA DDI prediction","Drug-protein heterogeneous graph improves DDI type classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000849,"raw_usage":{"total_tokens":3494,"prompt_tokens":675,"completion_tokens":2819,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":419,"completion_tokens_details":{"reasoning_tokens":2721}},"tokens_in":419,"tokens_out":2819,"duration_ms":19077,"temperature":1.0,"reasoning_tokens":2721,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T16:12:35.451399+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the same trained model on a test set that includes all negative pairs, or samples negatives at the natural 93% rate, and recompute F1 and accuracy; the paper's appendix already reports F1 falling from 0.969 to 0.871, and if under this distribution the model no longer outperforms the best baseline, the claim that HGNN-DDI beats state-of-the-art baselines is refuted.","supporting_citations":[],"review_version":1}