{"id":"2de7e180-39ae-4a5c-8554-494fedcda6ac","arxiv_id":"2501.12309","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A hybrid supervised and self-supervised graph neural network with node-edge attention matches or exceeds existing methods on protein-protein interaction, gene ontology, and compound similarity prediction.","lead":"This paper describes a graph neural network that predicts properties of connections between pairs of nodes, such as whether two proteins interact. It combines attention over node and edge features with a loss that mixes supervised and self-supervised terms, and reports competitive results on protein interaction, gene function, and compound similarity tasks.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The compound-similarity result relies on Tanimoto labels for pairs whose fingerprints cannot be computed; the paper never defines those labels, so MAE 0.013 is not a valid validation of the unknown-structure claim.","rationale":"I read the paper's central claim as having three pillars: PPI ranking, GO competitiveness, and unknown-compound similarity. The PPI experiments are the most solid: 10 repeats of 5-fold CV, shared folds, Friedman p=7.8e-30 and a CD diagram support the ranking claim; the GO experiments have proper statistical tests and are honestly worded as competitive rather than superior. The unknown-compound pillar is where the evaluation breaks down. Section III-C fixes edge weights to Tanimoto coefficients and reports 21,528 pairs including 5,390 with an unknown compound; Section IV-C / Figure 9 says fingerprints cannot be created for those compounds. A Tanimoto coefficient is computed from fingerprints, so the target label for these pairs is undefined. No statement anywhere explains how the MAE was computed for those pairs. This is an internal inconsistency in the experimental setup, not a matter of outside consensus. The reader's weakest_assumption identifies exactly this flaw, so I agree. A revision could fix it by reporting label construction and either removing those pairs from the MAE and clearly separating unsupervised embedding training from supervised evaluation, or providing a principled surrogate label. Thus the existing CONDITIONAL verdict is appropriate and remains unchanged. Secondary issues (the GO sentence selecting the lowest validation F1, no code, no error bars on MAE) reinforce the need for revision but do not change the verdict.","tokens_in":11290,"tokens_out":4596,"duration_ms":45358,"concrete_test":"Obtain the data-construction code and the label tensor used for the 5,390 pairs containing at least one unknown compound. Redo the 5-fold CV twice: (a) using only the 16,138 known-known pairs as labeled targets and (b) using all 21,528 pairs with the exact label handling in the released code. If the MAE under (a) differs from 0.013, or if (b) reproduces 0.013 only by assigning Tanimoto values to pairs with missing fingerprints, then the paper's unknown-structure similarity claim is not quantitatively validated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that the model 'provides a solution for the previously unsolved problem of predicting similarity between compounds with unknown structures' (Abstract; Section IV-C). To validate this by MAE, every pair in the 21,528-pair dataset needs a Tanimoto ground-truth label. But Section III-C says edge weights are Tanimoto coefficients computed from MACCS fingerprints, and Section IV-C / Figure 9 states that for compounds with generic '-R' substituents 'it is impossible to create a fingerprint.' Therefore the 5,390 pairs involving at least one unknown compound (Section III-C) cannot have a well-defined Tanimoto label. The paper does not state whether these pairs are excluded from the MAE calculation, given a placeholder label, or treated as unlabeled and used only through the self-supervised term. If excluded, the reported MAE=0.013 validates only known-known pairs and the unknown-compound claim rests on the qualitative Figure 9 examples and PCA plot, not on a quantitative benchmark. If included with placeholder or imputed labels, the target itself is undefined and the number is meaningless. Either way, the strongest advertised contribution—predicting similarity without structure—is not supported by the reported experiment. This is an internal inconsistency in the evaluation setup, not a disagreement with consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes NEAConv, a message-passing graph neural network for edge-centric tasks. The architecture consists of a tokenizer MLP, a single attention-based convolution layer that uses both node and edge features, and a prediction MLP; training uses a hybrid loss that combines a supervised prediction term, a supervised cosine-similarity term on the central node embeddings, and a self-supervised term that aligns the prediction with the embedding cosine similarity. Experiments are reported on PPI prediction for five organisms, GO term prediction for three species, and compound similarity on six KEGG pathways. The authors report state-of-the-art or competitive results on PPI and GO, an MAE of 0.013 on compound similarity, and claim that the model provides a solution for predicting similarity between compounds with unknown structures.","tokens_in":11511,"tokens_out":4393,"duration_ms":47759,"significance":"If the results hold, the paper makes a useful contribution: a single architecture with attention over node and edge features plus a hybrid supervised/self-supervised loss is competitive across three bioinformatics benchmarks. The PPI evaluation is a strength, using 10x5-fold cross-validation, repeated runs, and statistical tests; the GO comparison uses standard CAFA-style evaluation; and the ablation study on attention and loss components directly probes the contribution of each design element. However, the strongest advertised claim, namely quantitative similarity prediction for compounds with unknown structures, is not supported by the reported evaluation because the ground-truth Tanimoto labels for the 5,390 pairs involving at least one unknown compound are never defined. The architecture and the PPI/GO results may still be defensible, but the compound-similarity experiment needs to be substantially clarified or its claims restricted.","major_comments":[{"comment":"The headline MAE of 0.013 is not a valid validation of the unknown-structure claim as reported. Section III-C states that edge weights are Tanimoto coefficients computed from MACCS fingerprints, while Section IV-C and Figure 9 state that for compounds with generic '-R' substituents 'it is impossible to create a fingerprint.' The dataset contains 21,528 pairs, including 5,390 pairs involving at least one unknown compound, but the paper never defines what ground-truth label is used for those pairs, nor whether they are excluded from the MAE calculation. If they are excluded, the MAE covers only known-known pairs and the unknown-structure claim rests on the qualitative Figure 9 examples and the PCA plot; if they are included with placeholder or imputed labels, the target is undefined and the number is not meaningful. The authors should state the labeling rule, report MAE separately for known-known and unknown-involved pairs, or temper the abstract claim accordingly.","section":"III-C and IV-C"},{"comment":"The GO model-selection rule is stated as 'Five runs were performed with different initialization seeds, and the run with the lowest validation F1 score was selected.' If literal, this is the opposite of standard model selection and would select the worst validation result, which would make the reported F1max comparisons difficult to interpret. Please clarify whether this is a typo (e.g., lowest validation loss or highest validation F1 was intended), report the criterion actually used, and provide the variance across the five seeds so the Nemenyi test in Figure 7 can be meaningfully assessed.","section":"IV-B"},{"comment":"Baseline provenance is unstated for the GO comparison. It is not clear whether exp2GO, deepGOplus, NMFGO, and BLAST were rerun under the same training/validation folds and the same 2016/2017 GOA files, or whether their F1max values were taken directly from the original publications. No error bars or seed-level variability are reported for any GO method. Without this information, the claim of no statistically significant difference between the proposed model and exp2GO cannot be independently checked, because the Friedman/Nemenyi procedure requires comparable samples from the same experimental protocol.","section":"IV-B"}],"minor_comments":[{"comment":"The notation in 'Let G = {v, e} be a graph, where v is the set of |v| nodes' reuses the same symbol for a set and its cardinality; using uppercase V for the node set and E for the edge set would avoid ambiguity.","section":"II-A"},{"comment":"In Equation (3), the message vector m_{k,j} is used before it is formally defined; please define it explicitly in the text that introduces Equation (2), or state that it is the value vector V of node k weighted by the attention scalar w.","section":"II-C"},{"comment":"The caption of Figure 4 says 'the line inside the box denotes the mean F1 of each k-fold,' but the boxplots aggregate 10 repetitions of 5-fold cross-validation; the line should more precisely be described as the mean over repetitions for each fold, and the text would benefit from reporting the numeric mean and standard deviation of F1 for each model.","section":"IV-A"},{"comment":"There are small language and consistency issues: 'the proposed loss function is build' should be 'is built'; 'exp2go' is sometimes written with lowercase 'go' (e.g., in the Section IV-B text) while the reference and figure use 'exp2GO'; and the claim that the model 'outperforms state-of-the-art methods' should be phrased more carefully for the GO task, where the authors themselves report no significant difference with exp2GO.","section":"II-E"}],"recommendation":"major_revision","confidential_remarks":"The central blocker is the undefined ground-truth label for compound-similarity pairs involving unknown structures. I do not see this as an irreparable flaw: the PPI and GO experiments and the ablation study are potentially publishable, and the authors can likely address the issue by clarifying whether the 5,390 unknown-involved pairs were excluded or labeled, or by restricting the claim to known-known pairs and presenting the unknown-structure evidence as qualitative. The GO model-selection rule also needs a clear correction or justification before I would be comfortable with the F1max comparison. I would encourage the editor to request a revision rather than a rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The real news here is the architecture plus the evaluation discipline on PPI: a single MPNN with attention over both node and edge features, trained with a supervised-plus-cosine-consistency loss, matches or beats specialized predictors on five PPI datasets. The 10x5 cross-validation, Friedman test, and critical-difference diagram are exactly the kind of evidence this area usually lacks. GO results are weaker but honestly reported as statistically tied with exp2GO. That part of the paper deserves a serious referee.\n\nThe compound-similarity section does not hold up as written. The paper says 5,390 of the 21,528 pairs involve at least one compound with unknown structure, and it also says those compounds cannot have fingerprints. The edge labels are Tanimoto coefficients computed from fingerprints. So for those pairs there is no well-defined ground truth. The paper never says whether those pairs are excluded from the reported MAE of 0.013, given placeholder labels, or used only in the self-supervised term. If they are excluded, the number validates only known-known pairs and the advertised \"unknown structure\" solution rests on qualitative examples. If they are included, the target itself is undefined and the number is meaningless. This is not a nitpick; it is the load-bearing claim of Section IV-C and the abstract.\n\nAlso worth fixing: the GO model selection rule—choosing the run with the lowest validation F1—is unusual and needs an explanation, not a sentence. The headline MAE has no error bars. No code, data, or hyperparameters are released, which makes the PPI results harder to verify. The novelty is real but incremental: graph attention with edge features is well established, and the compound-similarity task extends the authors' own prior work. The self-citations are not a problem per se, but they do narrow the claimed contribution.\n\nWho is this for? Researchers working on edge-level prediction in bioinformatics and anyone designing GNN losses for partially labeled graphs. The PPI comparison is worth reading; the compound-similarity claim needs a major revision before it can be accepted.\n\nRecommendation: send to peer review. A good referee should push for a precise definition of the labels for unknown-structure pairs, error bars, and released code. If the authors can clarify or drop the unsupported claim, the PPI and GO results carry the paper.","headline":"A genuinely useful edge-centric GNN with solid PPI/GO experiments, but the compound-similarity headline rests on an undefined ground truth for unknown-structure pairs.","tokens_in":12067,"tokens_out":1247,"would_cite":false,"duration_ms":15831,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims that a single graph neural network with node-and-edge attention and a hybrid supervised/self-supervised loss matches or beats specialized predictors for protein-protein interaction, gene function, and compound similarity.","keywords":["graph neural networks","edge-centric prediction","node-edge attention","hybrid loss","self-supervised learning","protein-protein interaction","gene function prediction","compound similarity"],"falsifier":"Recompute the compound-similarity mean absolute error after removing the 5,390 pairs that involve at least one compound with unknown structure; if the reported 0.013 disappears or degrades, the claim that the model solves unknown-structure similarity is not supported by the stated experiment. A second check is to train the same model without the self-supervised term and compare the embedding organization of unknown-structure compounds; if the PCA map is essentially unchanged, that loss component is not doing the work the paper attributes to it.","tokens_in":11036,"feed_emoji":"🧬","tokens_out":14744,"duration_ms":133615,"temperature":0.7,"pith_summary":"The paper argues that one graph architecture can handle edge-centric tasks — predicting properties of pairs of nodes — across biology. The architecture is a message-passing layer that attends over both node and edge features, followed by an MLP predictor, trained end-to-end with a hybrid loss that joins a supervised prediction term with two cosine-similarity terms. This lets the model learn from unlabeled subgraphs and from sparse one-hot node features, which is why the authors claim it can score compounds whose structures are unknown. A reader should care because edge-centric predictions are common in bioinformatics, and most graph networks under-use edge attributes and require dense labels.","feed_headline":"Edge-attention GNN matches or beats protein and gene predictors","feed_subtitle":"Self-supervised cosine loss lets the same architecture score compounds with unknown structures","key_machinery":"The central object is the NodeEdgeAttentionConv (NEAConv) layer, a message-passing update in which the attention weight is $w = \\mathrm{sum}(Q \\odot K)$ across the feature dimension, with $Q$ formed from the target node and edge features and $K,V$ formed from source node features; the update concatenates the weighted aggregation with the original tokenized features and applies a Tanh activation. The second load-bearing object is the loss $L = \\alpha L_{\\text{supervised}} + \\beta L_{\\text{cosine}} + \\gamma L_{\\text{cosine pred}}$. The cosine term pushes the cosine similarity of the two central-node embeddings toward the label, and the self-supervised prediction term aligns the model's output with that embedding similarity, which is what lets one-hot placeholder nodes and unlabeled pairs still organize the latent space.","core_discovery":"The central claim is that the proposed NEAConv architecture with the hybrid loss of Eq. 5 is a competitive general-purpose predictor for relations between node pairs. NEAConv computes attention weights as the summed Hadamard product of query and key vectors, builds the query from the target node together with edge features, and updates a central node by concatenating the aggregated neighbor messages with its tokenized features. The hybrid loss couples supervised prediction error, error between the cosine similarity of the two node embeddings and the label, and a self-supervised term that aligns the network's prediction with that embedding similarity. Empirically, the model achieves the highest average rank across five protein-protein interaction datasets, ties with exp2GO on gene function prediction with no statistically significant difference, and reaches a compound-similarity mean absolute error of 0.013 including pairs whose compounds lack structures.","pith_inferences":["An extension not explored in the paper: the same architecture should transfer to non-biological edge-regression problems such as drug-target affinity or social tie strength, because neither the attention update nor the cosine-alignment loss is biology-specific.","One testable extension would be to replace cosine similarity in the loss with other pairwise measures, such as Euclidean distance or dot product, and see whether the latent-space organization and error behavior change accordingly.","A further test the paper leaves open: mask a random subset of known compound structures during training and ask whether their one-hot embeddings still land near structural neighbors in the PCA map, which would directly quantify the claimed solution for unknown structures."],"forward_implications":["On the five PPI benchmarks, the model ranks first among the four compared methods, and the gap to the second-place method is statistically significant.","On gene-function prediction, the model is statistically tied with exp2GO and ahead of two other deep learners, indicating that expression-similarity graphs can drive function transfer.","On compound similarity, the model reaches a mean absolute error of 0.013 and produces a PCA embedding map in which similarity to a reference compound fades smoothly with distance.","The ablation study shows that the node-and-edge attention and the hybrid loss each reduce error on their own, and together they tighten the relationship between embedding distance and predicted similarity.","Because the self-supervised term is active for unlabeled subgraphs, the training procedure can potentially exploit pairs that have no ground-truth label."],"supporting_citations":[{"why":"Supplies the five PPI benchmark datasets and the signed-graph embedding baseline that the model is compared against.","marker":"[18]"},{"why":"Defines the EXP2GO protocol and the gene-function prediction baseline with which the model is statistically tied.","marker":"[22]"},{"why":"Provides the Siamese Residual RCNN baseline, which the PPI ranking places second.","marker":"[31]"},{"why":"Provides the EnAmDNN ensemble-attention baseline used in the PPI comparison.","marker":"[32]"},{"why":"Supplies the recent benchmark that identifies EnAmDNN as a leading sequence-based predictor.","marker":"[33]"},{"why":"Supplies the attention mechanism that the NEAConv layer adapts to graph edges.","marker":"[17]"},{"why":"Defines the evaluation rules and F1max metric used for the gene-function prediction experiments.","marker":"[26]"},{"why":"Defines the fingerprint-based similarity coefficient used as the compound-similarity label and edge weight.","marker":"[30]"},{"why":"Supplies the sequence-derived node features used for the PPI experiments.","marker":"[21]"}],"fun_headline_variants":["Hybrid GNN with self-supervision matches protein and gene predictors","Edge-centric attention GNN beats predictors on protein and gene links","Self-supervised GNN handles protein, gene, and unknown-structure compounds","Attention GNN unifies supervised and self-supervised edge prediction","One GNN for protein, gene, and compound relations with self-supervision"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The broadest compound-similarity claim assumes that a fingerprint-based similarity label exists for every pair in the 21,528-pair dataset, including the 5,390 pairs where at least one compound has no usable structure; the paper itself says those fingerprints cannot be created, so the target for those pairs is undefined.","fun_headline_variants_meta":{"raw":{"variants":["Hybrid GNN with self-supervision matches protein and gene predictors","Edge-centric attention GNN beats predictors on protein and gene links","Self-supervised GNN handles protein, gene, and unknown-structure compounds","Attention GNN unifies supervised and self-supervised edge prediction","One GNN for protein, gene, and compound relations with self-supervision"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001162,"raw_usage":{"total_tokens":4779,"prompt_tokens":881,"completion_tokens":3898,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":497,"completion_tokens_details":{"reasoning_tokens":3803}},"tokens_in":497,"tokens_out":3898,"duration_ms":26149,"temperature":1.0,"reasoning_tokens":3803,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:17:18.392972+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute the compound-similarity mean absolute error after removing the 5,390 pairs that involve at least one compound with unknown structure; if the reported 0.013 disappears or degrades, the claim that the model solves unknown-structure similarity is not supported by the stated experiment. A second check is to train the same model without the self-supervised term and compare the embedding organization of unknown-structure compounds; if the PCA map is essentially unchanged, that loss component is not doing the work the paper attributes to it.","supporting_citations":[{"cited_title":"Attention is all you need,","cited_arxiv_id":null,"evidence_quote":"Supplies the attention mechanism that the NEAConv layer adapts to graph edges."},{"cited_title":"Graph-based prediction of protein-protein interactions with attributed signed graph embedding,","cited_arxiv_id":null,"evidence_quote":"Supplies the five PPI benchmark datasets and the signed-graph embedding baseline that the model is compared against."},{"cited_title":"exp2go: Improving prediction of functions in the gene ontology with expression data,","cited_arxiv_id":null,"evidence_quote":"Defines the EXP2GO protocol and the gene-function prediction baseline with which the model is statistically tied."},{"cited_title":"Multifaceted protein–protein interaction prediction based on siamese residual rcnn,","cited_arxiv_id":null,"evidence_quote":"Provides the Siamese Residual RCNN baseline, which the PPI ranking places second."},{"cited_title":"Protein interaction network reconstruction through ensemble deep learning with attention mechanism,","cited_arxiv_id":null,"evidence_quote":"Provides the EnAmDNN ensemble-attention baseline used in the PPI comparison."},{"cited_title":"Benchmark evaluation of protein–protein interaction prediction algorithms,","cited_arxiv_id":null,"evidence_quote":"Supplies the recent benchmark that identifies EnAmDNN as a leading sequence-based predictor."},{"cited_title":"The cafa challenge reports improved protein function prediction and new functional annotations for hundreds of genes through experimental screens,","cited_arxiv_id":null,"evidence_quote":"Defines the evaluation rules and F1max metric used for the gene-function prediction experiments."},{"cited_title":"Why is tanimoto index an appropriate choice for fingerprint-based similarity calculations?","cited_arxiv_id":null,"evidence_quote":"Defines the fingerprint-based similarity coefficient used as the compound-similarity label and edge weight."},{"cited_title":"Predicting protein-protein interactions based only on sequences infor- mation,","cited_arxiv_id":null,"evidence_quote":"Supplies the sequence-derived node features used for the PPI experiments."}],"review_version":1}