{"id":"cb9e5a01-c3ae-4384-a9dc-c4b97e32de77","arxiv_id":"2412.15790","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"GraphSeqLM fuses LLM sequence embeddings with multi-omic features on a KEGG-based knowledge graph, but its reported accuracy gains over GNN baselines are inconsistent and statistically untested.","lead":"GraphSeqLM adds DNA, RNA, and protein sequence embeddings from large language models to the node features of a signaling-pathway graph, then runs a graph neural network to predict cancer patient survival. The paper reports mixed results across six cancer types, and its claim of consistent superiority over existing GNNs is not supported by its own table.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"GraphSeqLM's central claim is unsupported: Table 2 shows it is worse than baselines on LUAD and STAD, no significance tests are reported, and no ablation isolates the LLM sequence embeddings from the rest of the architecture.","rationale":"The paper's central contribution is the fusion of LLM sequence embeddings with a GNN over a multi-omic signaling graph, and the headline claim is that this yields superior predictive accuracy. The most decisive weakness is not the biological plausibility of the KEGG graph but the absence of any experiment that isolates the proposed mechanism. Table 2 itself undermines the word 'consistently': on LUAD and STAD the proposed model is numerically worse than several baselines in both metrics. Without significance tests, even the numerical wins on the other four cohorts are weak evidence, because the reported cross-validation standard deviations (e.g., 0.35 on BLCA F1) are comparable to or larger than the reported gaps. A controlled ablation replacing LLM embeddings with random or generic fixed vectors, combined with paired significance testing, would directly settle whether the sequence embeddings carry predictive signal. If random vectors perform equally well, the mechanism claimed in the abstract is not supported; if no variant beats the best baseline significantly, the broader 'outperforms existing methods' claim fails. This is why the current evidence does not support acceptance, and the reader's rejection remains appropriate.","tokens_in":6638,"tokens_out":6183,"duration_ms":51847,"concrete_test":"Run paired 5-fold cross-validation on the same six cancer cohorts with three GraphSeqLM variants: (1) original LLM embeddings, (2) fixed random vectors of the same dimension, and (3) PCA or gene-identity features. Report per-dataset accuracy and F1 with paired bootstrap or McNemar tests comparing each variant to the best baseline. If variant (2) matches or beats variant (1), the LLM sequence embeddings are not the source of any gain; if no GraphSeqLM variant beats the best baseline with p<0.05 after multiple-testing correction, the 'superior predictive accuracy' claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing claim is that adding LLM biological-sequence embeddings makes GraphSeqLM consistently outperform existing GNN methods. The paper's own Table 2 fails this claim: on LUAD, GraphSeqLM reaches 0.6746 accuracy and 0.5918 F1, below GIN's 0.6867 accuracy and 0.6770 F1; on STAD, GraphSeqLM reaches 0.6734 accuracy and 0.3319 F1, below UniMP's 0.6831 accuracy and GIN's 0.4132 F1. No significance tests accompany any comparison, and several baseline F1 entries have standard deviations above 0.2, so the apparent wins on BLCA, GBM, LUSC, and SKCM are not shown to be more than fold noise. More fundamentally, GraphSeqLM is not a minimal augmentation of the baselines: it adds a CrossModalityEncoder, a two-stage message-passing scheme, and high-dimensional static sequence features. Because none of these components is ablated, the reported improvements cannot be attributed to the LLM embeddings, which is the paper's stated mechanism. The central claim therefore rests on a confounded and statistically untested comparison.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes GraphSeqLM, a framework that augments a multi-omic signaling knowledge graph (MOS-KG) with DNA, RNA, and protein sequence embeddings from GPT-based sequence language models (DNAGPT and ProtGPT2), then applies a two-stage GNN message-passing scheme (internal then global) and an MLP readout to predict binary overall-survival status. The method is evaluated on 826 samples across six TCGA cancer types with 5-fold cross-validation, comparing accuracy and F1 against GCN, GAT, GIN, and UniMP. The paper claims that GraphSeqLM consistently outperforms these baselines and that the LLM sequence embeddings are responsible for the improvement.","tokens_in":6847,"tokens_out":3757,"duration_ms":30681,"significance":"If the stated claims were established, the work would offer a plausible route to incorporate biological sequence information into interpretable multi-omic graph models, and the use of pretrained DNA/RNA/protein encoders as node-feature augmentations is timely. The manuscript also has strengths: it uses public benchmark data, standard baselines, 5-fold cross-validation, provides a code repository link, and makes the graph-construction pipeline explicit via MOS-KG/KEGG. However, the current evidence does not support the central claim: the paper's own Table 2 shows that GraphSeqLM is not consistently better than the baselines, no significance tests are reported, and no ablation isolates the contribution of the sequence embeddings. As a result the contribution, while potentially interesting, is not established.","major_comments":[{"comment":"The central claim that GraphSeqLM 'consistently outperforms other graph-based encoder models' is contradicted by Table 2. On LUAD, GraphSeqLM's accuracy is 0.6746±0.0352 and F1 is 0.5918±0.1688, while GIN achieves 0.6867±0.0714 accuracy and 0.6770±0.0335 F1; on STAD, GraphSeqLM's accuracy is 0.6734±0.0766 and F1 is 0.3319±0.1847, while UniMP achieves 0.6831±0.0619 accuracy and GIN achieves 0.4132±0.2433 F1. The wording in Sections 1, 3.2, and 4 therefore overstates the results as reported.","section":"Table 2 / Section 3.2"},{"comment":"No statistical significance testing accompanies any comparison. Many F1 estimates have standard deviations above 0.2 (e.g., SKCM F1: GraphSeqLM 0.5332±0.3706 vs GIN 0.4633±0.2931; BLCA F1: GAT 0.7138±0.263), so the apparent improvements on BLCA, GBM, LUSC, and SKCM could arise from fold-level noise. Paired tests across the five folds or repeated-seed comparisons are required before any claim of superiority can be made.","section":"Section 3.2 / Table 2"},{"comment":"The design does not isolate the effect of LLM sequence embeddings. Compared with the GNN baselines, GraphSeqLM changes several factors simultaneously: the node features (LLM embeddings), the CrossModalityEncoder fusion, the two-stage message-passing scheme (GNNin then GNNglobal), and the final readout. Without an ablation that runs the same two-stage architecture with and without the sequence embeddings, or a baseline that uses sequence embeddings without the graph, the paper's stated mechanism — that LLM embeddings enhance expressivity — is confounded with the architectural changes.","section":"Section 2.2"},{"comment":"The sample exclusion criteria are unspecified: the text says samples with 'imbalanced or insufficient data' were excluded, but it does not define these thresholds, report how many samples were removed per cancer type, or state whether exclusion was performed before or after label assignment. In addition, training hyperparameters (learning rate, number of layers, hidden dimensions d_l, d_l', and d, dropout, early stopping) and the train/validation split procedure are not reported, preventing replication and making it impossible to assess whether the comparison is fair to all methods.","section":"Section 3.1"}],"minor_comments":[{"comment":"The heading 'Experiment Settings and Evaludation' contains a typo; 'Evaludation' should be 'Evaluation'.","section":"Section 3.2"},{"comment":"The AVG aggregation in Equation (9) is not defined precisely; it should specify whether the average is taken over all node embeddings or over a particular subset, and how the resulting patient-level vector is formed before the MLP.","section":"Equations (9)-(10)"},{"comment":"Some references are informal or not clearly connected to the method as used; for example, reference [17] describes mosGraphGen, but the manuscript does not state whether that tool generated the exact MOS-KG used here, and reference [3] is a preprint that is not cited in the main text.","section":"References"}],"recommendation":"reject","confidential_remarks":"The paper is a 4-page companion-style manuscript whose central claim is directly contradicted by its own Table 2. While a revised version with significance tests and ablations could, in principle, provide a more defensible empirical study, the current evidence does not support the abstract's assertion of superior predictive accuracy. I would not consider a minor revision appropriate; a major rework of the evaluation and claims would be needed."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a reasonable feature-augmentation idea, but the headline claim of consistent superiority is unsupported by the paper's own Table 2. On LUAD and STAD the proposed model loses on both accuracy and F1, and no significance tests or ablations are provided. The paper deserves a serious look, but only after the evaluation is made honest.\n\nWhat's actually new is the specific combination: DNAGPT and ProtGPT2 embeddings for DNA/RNA/protein sequences, fused with multi-omic features into a KEGG-derived signaling graph, followed by a two-stage message-passing GNN. That's a sensible instantiation of feature augmentation, and the separate encoders per sequence type are biologically reasonable. The methodology section is clear enough to reproduce the idea.\n\nThe problems are in the evaluation. The abstract and Section 3.2 say the model 'consistently outperforms' baselines, but Table 2 shows LUAD accuracy 0.6746 vs GIN's 0.6867 and F1 0.5918 vs 0.6770; STAD accuracy 0.6734 vs UniMP's 0.6831 and F1 0.3319 vs GIN's 0.4132. Those are not small gaps. No significance tests accompany any comparison, and several baseline F1 entries have standard deviations above 0.2, so the wins elsewhere could be fold noise. More fundamentally, there's no ablation isolating the LLM embeddings. The model adds a CrossModalityEncoder and two-stage propagation on top of the baselines, so you can't attribute the gains to the sequence features. The sample exclusion criteria are also unspecified, and there's no comparison to simpler feature-augmentation baselines like PCA or expression-derived features. The self-citations to mosGraphGen and DeepSignalingFlow are legitimate; those are the group's own tools, and this paper extends that line.\n\nFor a reader: this is a workshop-style short paper that shows one way to inject sequence embeddings into multi-omic graph learning. It's not a definitive benchmark. With significance tests, ablations, and a more honest claim, it could be a useful practical note.\n\nI'd send it to review if the authors fix the overclaim and add the missing analyses. As is, I wouldn't trust the superiority claim, but the idea is worth engaging with.","headline":"Reasonable feature-augmentation idea, but the 'consistently outperforms' claim is contradicted by the paper's own Table 2, and missing ablations and significance tests leave the main result unverified.","tokens_in":7380,"tokens_out":3016,"would_cite":false,"duration_ms":23331,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A unified graph-language framework that enriches multi-omic signaling graphs with DNA, RNA, and protein sequence embeddings from LLMs reports improved patient-outcome prediction across six cancer types.","keywords":["Large language models","Graph neural networks","Multi-omic data","Biological sequences","Precision medicine","Cancer survival prediction","Knowledge graphs","Sequence embeddings"],"falsifier":"Retrain GraphSeqLM with the same node features but randomly permuted edge sets, preserving node degrees, and check whether accuracy drops relative to the true graph. If the randomized graph matches the true-graph performance, the reported gains come from the sequence embeddings and MLP rather than the biological topology; a second check is to replace the GNN with a sequence-feature-only MLP and compare.","tokens_in":6423,"feed_emoji":"🧬","tokens_out":8278,"duration_ms":65911,"temperature":0.7,"pith_summary":"GraphSeqLM aims to show that enriching a multi-omic signaling knowledge graph with DNA, RNA, and protein sequence embeddings from large language models improves graph-neural-network prediction of cancer patient survival. The framework builds a KEGG-derived graph whose nodes carry fused omic features and sequence embeddings, then applies two-stage message passing to classify patients as survivors or non-survivors. On a cohort of 826 samples across six cancer types, the authors report accuracy and F1 scores that beat four standard GNN baselines on most datasets. If the claim holds, the approach provides a simple, general feature-augmentation recipe for omic graph models, with direct relevance to precision medicine.","feed_headline":"Fusing LLM sequences into omic graphs improves survival prediction","feed_subtitle":"GraphSeqLM reports accuracy and F1 gains over four GNN baselines on 826 samples across six cancer types.","key_machinery":"The central object is the sequence-attributed Multi-Omic Signaling Knowledge Graph (MOS-KG), where each gene, transcript, or protein node holds a multi-omic feature vector fused with a fixed LLM sequence embedding. The argument runs on two-stage message passing: an internal GNN propagates information along the translation subgraph connecting gene, transcript, and protein nodes, and a global GNN then propagates over the full KEGG regulatory and protein-protein interaction network; node embeddings are average-pooled and passed to an MLP for binary survival classification. The sequence embeddings are the new ingredient because they are the only node features not derived from the sample-specific omic measurements.","core_discovery":"The paper's central claim is that sequence-derived embeddings—computed by DNAGPT for DNA and RNA and by ProtGPT2 for proteins—carry structural and biological information that complements multi-omic node features in a signaling graph, and that fusing the two before message passing makes the graph encoder more expressive. This fusion, combined with internal-then-global message propagation over the Multi-Omic Signaling Knowledge Graph, yields the best overall predictive accuracy and F1 across the evaluated cancer datasets. The authors interpret this as evidence that LLM sequence representations can augment GNNs in a way that overcomes their expressivity limits on large, dense biological networks.","pith_inferences":["A natural ablation the paper does not report is to compare GraphSeqLM against a non-graph MLP using the same omic features and sequence embeddings; if the MLP matches the graph model, the biological topology contributes little beyond the features.","Because the reported standard deviations overlap across folds for some cancer types (e.g., LUAD and STAD), a permutation test or repeated cross-validation would clarify whether the improvements are statistically reliable.","The fusion of sequence and omic features is done by simple concatenation after projection; an attention-based cross-modality layer could be a direct extension and might yield larger gains."],"forward_implications":["Sequence embeddings can be dropped into existing GNN-based omic models as node-level feature augmentation without changing the graph construction pipeline.","Across the six cancer types analyzed, the largest reported accuracy gains appear in GBM and SKCM, where the fused model surpasses the strongest baseline by several percentage points.","The same two-stage message-passing design could be applied to any disease with multi-omic data and a known signaling or interaction network.","GraphSeqLM's reliance on frozen, pretrained biological LLMs means a wide range of encoders can be swapped in as better sequence models become available."],"supporting_citations":[{"why":"Supplies the gene regulatory and signaling pathway structure that defines the edges of the Multi-Omic Signaling Knowledge Graph.","marker":"[7]"},{"why":"Provides the DNA sequence language model used to embed gene and transcript sequences, with T-to-U substitution applied for RNA.","marker":"[16]"},{"why":"Provides the ProtGPT2 protein language model that embeds protein sequences on the graph's protein nodes.","marker":"[4]"},{"why":"Supplies the patient multi-omic feature matrices used as sample-specific node features.","marker":"[5]"},{"why":"Supplies the DNA, RNA, and protein sequences attached to graph nodes for sequence embedding.","marker":"[6]"},{"why":"Serves as the graph convolutional baseline and message-passing backbone that the framework extends with sequence-augmented features.","marker":"[8]"},{"why":"Serves as the graph attention network baseline against which GraphSeqLM is compared.","marker":"[12]"},{"why":"Serves as the graph isomorphism network baseline that sets a comparison bar for expressive graph classification.","marker":"[13]"},{"why":"Serves as a strong unified message-passing baseline used for comparison.","marker":"[11]"}],"fun_headline_variants":["LLM sequence embeddings sharpen omic graph predictions","GraphSeqLM: Fusing LLM embeddings with GNNs tops baselines","Sequence-aware GNNs improve multi-omic cancer analysis","AI sequences boost GNNs for precision medicine","Omic graphs get a lift from LLM protein embeddings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the KEGG-derived signaling graph captures the biological mechanisms that actually determine patient outcomes; if its edges are incomplete, noisy, or irrelevant to survival, message passing over that graph can dilute rather than sharpen the predictive signal.","fun_headline_variants_meta":{"raw":{"variants":["LLM sequence embeddings sharpen omic graph predictions","GraphSeqLM: Fusing LLM embeddings with GNNs tops baselines","Sequence-aware GNNs improve multi-omic cancer analysis","AI sequences boost GNNs for precision medicine","Omic graphs get a lift from LLM protein embeddings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000396,"raw_usage":{"total_tokens":2007,"prompt_tokens":812,"completion_tokens":1195,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":428,"completion_tokens_details":{"reasoning_tokens":1113}},"tokens_in":428,"tokens_out":1195,"duration_ms":10876,"temperature":1.0,"reasoning_tokens":1113,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:05:08.904005+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain GraphSeqLM with the same node features but randomly permuted edge sets, preserving node degrees, and check whether accuracy drops relative to the true graph. If the randomized graph matches the true-graph performance, the reported gains come from the sequence embeddings and MLP rather than the biological topology; a second check is to replace the GNN with a sequence-feature-only MLP and compare.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the DNA sequence language model used to embed gene and transcript sequences, with T-to-U substitution applied for RNA."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the ProtGPT2 protein language model that embeds protein sequences on the graph's protein nodes."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the patient multi-omic feature matrices used as sample-specific node features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the DNA, RNA, and protein sequences attached to graph nodes for sequence embedding."}],"review_version":1}