{"id":"d9923c5a-b4ef-4a93-89dd-f5d9d97b78f3","arxiv_id":"2506.13119","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"PhenoKG, a knowledge-graph-enhanced GNN and transformer model, achieves 24.64% MRR on the MyGene2 rare disease cohort versus 19.02% for SHEPHERD, and works without a candidate gene list.","lead":"A new model, PhenoKG, ranks which genes are likely causing a patient's rare disease using only their symptoms and a medical knowledge graph, and it outperforms the previous best system on a real patient dataset. It can work even when no candidate gene list is provided, which could help doctors narrow down genetic causes faster during the diagnostic odyssey.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The MyGene2 MRR gain over SHEPHERD is not statistically established: it is computed on a filtered 121/146-patient subset with KG-defined candidate sets, and the 5.62-point gap is only about 1.2 times the reported seed-to-seed standard deviation.","rationale":"The reader's weakest assumption correctly identifies the two-hop filtering and the exclusion of 25 MyGene2 patients as the main risk to generalizability. My stress-test agrees with that concern and extends it in two concrete ways. First, the filter changes the task itself: the candidate set is derived from the same KG that supplies the model's embeddings, so the model is evaluated only where the KG already provides a short phenotype-to-gene path. A fair comparison to SHEPHERD requires either the same candidate-generation procedure or a report of results on the full 146-patient set with a clearly defined failure penalty. Second, the reported variance makes the advantage statistically fragile. The best configuration's MRR is 24.64 +/- 4.57, so the 5.62-point gap over SHEPHERD's 19.02 is within the noise of three training runs, especially since SHEPHERD's standard deviation is reported as exactly zero, suggesting it was not run with the same seed variation. A paired bootstrap across patients would settle whether the gap is real. I would not reject the paper: the simulated-data results are more carefully controlled, the method is clearly described, and the limitation that the model cannot rank genes absent from the KG is acknowledged in Section 4.1 and Section 5. The issue is specifically the strength and framing of the MyGene2 headline, which should be conditional pending a paired, full-cohort evaluation. Since the reader already assigned CONDITIONAL, my stress-test does not move the verdict.","tokens_in":12565,"tokens_out":4482,"duration_ms":55319,"concrete_test":"Recompute the MyGene2 comparison as a paired study over all 146 patients using identical candidate sets for PhenoKG and SHEPHERD, assigning excluded patients a reciprocal rank of 0 instead of dropping them, and bootstrap the per-patient MRR difference with 10,000 resamples. If the 95% confidence interval for PhenoKG minus SHEPHERD includes zero, the reported 24.64 versus 19.02 advantage is not established and the headline claim should be revised.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the MyGene2 MRR of 24.64% versus SHEPHERD's 19.02%, stated in the Abstract and Section 4.4. The load-bearing condition is that both models rank the same gene set for the same patients under the same unanswerable-case rule. Section 4.1 shows this condition is not met: evaluation is restricted to the 121 of 146 patients whose causative gene is reachable within a two-hop PrimeKG neighborhood, and the candidate set is exactly the genes in that neighborhood. For the 25 excluded patients the model has zero chance by construction, so the reported MRR is conditional on the two-hop filter and cannot be compared with a genome-wide ranking unless SHEPHERD is given the same recovery advantage. In addition, Table 2 reports SHEPHERD as 19.02 +/- 0.00 with no variance, while the best PhenoKG configuration is 24.64 +/- 4.57 across three seeds; a 5.62-point gap with a 4.57 standard deviation does not support 'substantial improvement' without paired testing. The paper describes the simulation study well, but the real-data superiority claim is not yet demonstrated.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces PhenoKG, a phenotype-only gene prioritization model that builds patient-specific subgraphs from the PrimeKG knowledge graph, encodes them with GATv2 layers and a transformer gene encoder, and ranks candidate genes via contrastive and patient-similarity losses. The model is trained on simulated rare-disease patients and evaluated on a simulated test set and on the real MyGene2 dataset. The central claim is that PhenoKG outperforms the SHEPHERD baseline: MRR of 24.64% versus 19.02% on MyGene2 and MRR of 91.08% versus 79.35% on the simulated test set, with additional ablations across loss functions and embedding initializations.","tokens_in":12840,"tokens_out":3699,"duration_ms":41586,"significance":"If the real-data comparison were sound, this would be a practically useful contribution: phenotype-only gene prioritization that does not require an expert-curated candidate list could serve as an upstream triage step in rare-disease diagnostics. The paper has genuine strengths: it uses an external real-world benchmark (MyGene2), trains on simulated data rather than on the test labels, reports multiple seeds for its own model, and provides a fairly detailed ablation of loss components and embedding initializations. However, the headline real-data result is not yet convincingly established because the evaluation protocol differs from the baseline in a way that can only favor PhenoKG, and the reported gap is not statistically supported.","major_comments":[{"comment":"The MyGene2 evaluation is restricted to the 121 of 146 patients whose causative gene is reachable within a two-hop neighborhood in PrimeKG, and the candidate set is exactly the genes in that two-hop neighborhood. This means the reported MRR is conditional on a favorable reachability assumption, and for the 25 excluded patients the model has zero chance by construction. The comparison with SHEPHERD is therefore not apples-to-apples unless SHEPHERD is evaluated on the same 121 patients with the same candidate set, or PhenoKG is evaluated genome-wide. This issue is load-bearing for the abstract's claim of 'substantial improvements' on MyGene2.","section":"Section 4.1, Table 2"},{"comment":"The claimed MyGene2 improvement over SHEPHERD is not statistically established. The best PhenoKG configuration reports MRR 24.64 ± 4.57 across three seeds, while SHEPHERD is reported as 19.02 ± 0.00. The 5.62-point gap is only about 1.2 standard deviations of the PhenoKG estimate, and no paired per-patient comparison or significance test is provided. The authors should report paired metrics on the same patients, confidence intervals, or a permutation test before claiming superiority.","section":"Section 4.4, Table 2"},{"comment":"The paper selects the best of nine PhenoKG configurations (three loss combinations times three embedding strategies) on each dataset and compares that single best configuration against one SHEPHERD number. No model-selection procedure on a validation set is described for the MyGene2 results, and no multiple-comparison correction is applied. The comparison should either specify a pre-registered or validation-based selection rule, or report all configurations collectively (e.g., with error bars that include selection variability).","section":"Section 4.4, Tables 1 and 2"}],"minor_comments":[{"comment":"The sentence 'The set G with the goal of identifying the causative gene g∗ ∈ G' is grammatically incomplete, and the notation G is used both for the knowledge graph and for the candidate gene set, which is confusing.","section":"Section 3.1"},{"comment":"The regularization term is written as λ|∥p∥2 + mean(∥G∥2) − 2|, but the intended norm is not clear; this should be a squared L2 norm or the formula should be clarified, especially since the text says 'close to unit norm'.","section":"Section 3.3"},{"comment":"The text says 'nDCG@111.57%' and 'nDCG@115.15±3.33%' where the intended metric is nDCG@1; the repeated '1' is a typo that should be corrected.","section":"Section 4.4"},{"comment":"The phrase 'The patient neighbourhood was defined using k=2 nearest neighbours' is ambiguous: k is used earlier for the k-hop neighborhood in the knowledge graph, but here it appears to describe something else; please clarify the notation.","section":"Section 4.2"},{"comment":"The x-axis labels are not fully legible and the figure caption does not specify the exact metric plotted ('Match percentage' is defined only implicitly); please add a clear description of the evaluation protocol for this figure.","section":"Figure 3"},{"comment":"The memory bank M used in the patient similarity loss is described only as 'maintain[ing] a memory bank', with no details on its initialization, size, or update schedule beyond the circular buffer; please provide these details for reproducibility.","section":"Section 3.3"}],"recommendation":"major_revision","confidential_remarks":"This is a solid applied-ML paper with a clear method and an interesting external evaluation, but the headline real-data comparison currently rests on an unequal evaluation setup and a statistically fragile gap. The revision needed is substantial but well-scoped: rerun the comparison on the full MyGene2 cohort with matched candidate sets, add paired significance testing, and clarify the model-selection rule. If those points are addressed, the paper could be a good fit for the journal."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Plainly: this is a competent, honest paper that extends SHEPHERD to a no-candidate-list setting and shows a clear win on a simulated cohort. The MyGene2 headline gap, however, is not established by the evidence as presented.\n\nWhat's new: the model combines GATv2, a transformer gene encoder, and a patient-similarity memory bank, and it evaluates gene prioritization when no candidate list is provided, which is a genuine step beyond SHEPHERD. The simulation study is well handled: training on synthetic UDN-like data and testing on a held-out set, PhenoKG reaches 91.08 MRR vs 79.35 for SHEPHERD, and the ablations are sensible. The authors are transparent about the two-hop restriction on MyGene2 and the exclusion of 25/146 patients, and they state clearly that the method is not ready for standalone clinical use.\n\nThe soft spots are real. The MyGene2 comparison is the load-bearing result, and it currently rests on a filtered 121-patient subset with candidate sets defined by the same KG used to pretrain the embeddings. SHEPHERD's variance is reported as ±0.00, while PhenoKG's best config is 24.64 ± 4.57 over three seeds; the 5.62-point gap is about 1.2 standard deviations, so without paired significance testing the 'substantial improvement' claim is premature. The evaluation also mixes two different setups: simulated data uses candidate lists, MyGene2 uses k-hop neighborhoods, so the reader has to be careful not to treat the two as the same task. And the usual closed-world caveat applies—if the causative gene is not at least two hops from the phenotypes, the model cannot possibly find it.\n\nThere is no code or data release mentioned, which makes independent verification harder, but the method description is detailed enough to reimplement.\n\nWho it's for: researchers working on phenotype-driven gene prioritization and decision-support pipelines for rare disease diagnosis. It deserves a serious referee, but with a request for paired evaluation, per-subset reporting, and ideally a release of the evaluation pipeline.","headline":"PhenoKG is a credible candidate-free gene prioritization method with strong simulated performance, but its real-data advantage over SHEPHERD is not statistically supported as reported.","tokens_in":13377,"tokens_out":2994,"would_cite":false,"duration_ms":32717,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"PhenoKG predicts a patient's causative gene from phenotypes alone, reporting higher MRR than the previous best method on both real and simulated rare-disease cohorts.","keywords":["gene prioritization","phenotype-driven diagnosis","knowledge graph","graph neural networks","rare disease","transformer","MyGene2","PrimeKG"],"falsifier":"Run PhenoKG and SHEPHERD on all 146 MyGene2 patients without the two-hop exclusion, or on an independent cohort where some verified causative genes are known to sit three or more hops from the patient's phenotype terms; if MRR drops toward or below SHEPHERD's 19.02% in that setting, the conditional advantage does not generalize.","tokens_in":12352,"feed_emoji":"🧬","tokens_out":8175,"duration_ms":80477,"temperature":0.7,"pith_summary":"PhenoKG is a method for rare Mendelian disease diagnosis that takes only a patient's phenotype terms and produces a ranked list of likely causative genes, with no requirement for a pre-filtered candidate list or sequencing data. It builds a patient-specific subgraph from the PrimeKG knowledge graph, encodes it with graph attention layers and a transformer, and ranks roughly 8,000 genes by cosine similarity to the patient embedding. On the real MyGene2 cohort the authors report an MRR of 24.64% and nDCG@100 of 33.64%, above SHEPHERD's 19.02% and 30.54%; on the simulated UDN-like test set, the best phenotype-only configuration reaches 91.08% MRR versus 79.35% for SHEPHERD. The paper argues that such a phenotype-only model can act as an upstream triage tool or pre-filter before variant analysis, and that patient embeddings capture biological structure beyond the single causative gene.","feed_headline":"PhenoKG ranks causative genes from phenotypes alone, beating SHEPHERD","feed_subtitle":"On MyGene2, phenotype-only ranking hits 24.64% MRR vs SHEPHERD's 19.02%, narrowing the search space.","key_machinery":"The load-bearing object is the patient-specific subgraph $G_p$: the nodes and edges on shortest paths, or in the $k$-hop neighborhood, connecting each patient's HPO phenotypes to candidate genes in PrimeKG. A GATv2 graph encoder refines node embeddings, a transformer encoder produces gene embeddings, and multi-head attention with learnable memory vectors pools the phenotype embeddings into a patient vector; the final relevance score is the cosine similarity between patient and gene vectors. Two losses carry the training: a semi-hard-negative triplet loss that pulls the true gene closer than the hardest plausible negative, and a patient-similarity loss, backed by a memory bank of past patients, that pulls together patients who share a causative gene. The subgraph simultaneously defines the candidate gene set and supplies the relational structure that the learned embeddings need.","core_discovery":"The central claim is that causative-gene prioritization can be solved from phenotypes alone by enriching HPO terms with a knowledge graph, and that the resulting model outperforms the leading baseline under identical evaluation conditions. PhenoKG reaches an MRR of 24.64% and nDCG@100 of 33.64% on the real MyGene2 dataset without a candidate gene list, compared with 19.02% and 30.54% for SHEPHERD, and its top-1 accuracy rises to 83.96% when an expert-curated candidate list of about 20 genes is available. The same architecture, trained with a combination of a gene-ranking triplet loss and a patient-similarity loss, is reported to reach 91.08% MRR on a simulated test set structured like the Undiagnosed Diseases Network. The authors frame the model as a pre-filter and prioritization step that narrows the search space for clinicians, not as a standalone diagnostic.","pith_inferences":["The evaluation on MyGene2 drops 25 of 146 patients whose true gene is not reachable within two hops, so the reported advantage is conditional; a full-cohort or multi-hop evaluation would likely lower the numbers, and that gap is the main thing prospective validation should measure.","The paper's observation that patient embeddings cluster beyond shared causative genes suggests a use it does not develop: phenotype-based patient matching or cohort discovery for undiagnosed patients.","A concrete next experiment would take genes ranked highly by PhenoKG that are absent from expert panels, subject them to variant testing, and measure how often an unsuspected gene is confirmed; the paper explicitly says this requires prospective validation."],"forward_implications":["On MyGene2, the no-candidate-list setting reaches 24.64% MRR and 33.64% nDCG@100, compared with SHEPHERD's 19.02% and 30.54%.","With a candidate list of roughly 20 genes, top-1 accuracy is reported at 83.96%, so the model serves as both a discovery layer and a re-ranker.","On the simulated UDN-like test set, the best phenotype-only configuration reaches 91.08% MRR, above SHEPHERD's 79.35%.","The combined gene and patient-similarity loss is the most reliable configuration on the real dataset, indicating that aligning patients with the same causative gene supports generalization.","Because the model starts from HPO terms alone and ranks around 8,000 genes, it can operate as a pre-filter before whole-genome variant analysis rather than after it."],"supporting_citations":[{"why":"Supplies the SHEPHERD baseline, the pretrained knowledge-graph encoder, and the MyGene2 preprocessing protocol that PhenoKG must beat.","marker":"[Alsentzer et al., 2022]"},{"why":"Supplies PrimeKG, the 105,220-node knowledge graph whose edges and node types define the patient subgraphs and candidate gene universe.","marker":"[Chandak et al., 2023]"},{"why":"Provides the simulated training and validation patients with candidate gene lists that PhenoKG is trained on.","marker":"[Alsentzer et al., 2023]"},{"why":"Provides the real-world MyGene2 cohort of 146 patients and 48 causal genes used for the no-candidate-list evaluation.","marker":"[University of Washington, Center for Mendelian Genomics]"},{"why":"Supplies the GATv2 graph attention operator used as the core GNN encoder over each patient subgraph.","marker":"[Brody et al., 2022]"},{"why":"Supplies the multi-head self-attention and transformer encoder used to build patient and gene representations.","marker":"[Vaswani et al., 2017]"},{"why":"Supplies the Human Phenotype Ontology, whose terms encode each patient's phenotype set and connect patients to the knowledge graph.","marker":"[Köhler et al., 2019]"}],"fun_headline_variants":["PhenoKG beats SHEPHERD for gene discovery from phenotypes alone","Phenotype-only gene ranking: PhenoKG outperforms SHEPHERD on MyGene2","Knowledge graph plus GNN finds genes from symptoms without candidate lists","From phenotypes to genes: PhenoKG surpasses SHEPHERD in MRR","PhenoKG narrows gene search from symptoms, topping SHEPHERD"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported MyGene2 numbers assume that a patient's true causative gene lies within two hops of the patient's phenotypes in PrimeKG, because the 25 of 146 patients whose genes were unreachable were excluded from the evaluation.","fun_headline_variants_meta":{"raw":{"variants":["PhenoKG beats SHEPHERD for gene discovery from phenotypes alone","Phenotype-only gene ranking: PhenoKG outperforms SHEPHERD on MyGene2","Knowledge graph plus GNN finds genes from symptoms without candidate lists","From phenotypes to genes: PhenoKG surpasses SHEPHERD in MRR","PhenoKG narrows gene search from symptoms, topping SHEPHERD"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000733,"raw_usage":{"total_tokens":3268,"prompt_tokens":924,"completion_tokens":2344,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":540,"completion_tokens_details":{"reasoning_tokens":2238}},"tokens_in":540,"tokens_out":2344,"duration_ms":18796,"temperature":1.0,"reasoning_tokens":2238,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:36:40.806948+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run PhenoKG and SHEPHERD on all 146 MyGene2 patients without the two-hop exclusion, or on an independent cohort where some verified causative genes are known to sit three or more hops from the patient's phenotype terms; if MRR drops toward or below SHEPHERD's 19.02% in that setting, the conditional advantage does not generalize.","supporting_citations":[{"cited_title":"Few shot learning for phenotype-driven diagnosis of patients with rare genetic diseases","cited_arxiv_id":null,"evidence_quote":"Supplies the SHEPHERD baseline, the pretrained knowledge-graph encoder, and the MyGene2 preprocessing protocol that PhenoKG must beat."},{"cited_title":"Building a knowledge graph to enable precision medicine","cited_arxiv_id":null,"evidence_quote":"Supplies PrimeKG, the 105,220-node knowledge graph whose edges and node types define the patient subgraphs and candidate gene universe."},{"cited_title":"Simulation of undiagnosed patients with novel genetic conditions","cited_arxiv_id":null,"evidence_quote":"Provides the simulated training and validation patients with candidate gene lists that PhenoKG is trained on."},{"cited_title":"How attentive are graph attention networks? In International Conference on Learning Representations, 2022","cited_arxiv_id":null,"evidence_quote":"Supplies the GATv2 graph attention operator used as the core GNN encoder over each patient subgraph."}],"review_version":1}