{"id":"dcc9befd-14ea-45f1-b3b1-e15f3e0e359b","arxiv_id":"1908.08135","paper_version":4,"verdict":"REJECT","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Using an aging-specific PPI subnetwork and graphlet-based features improves precision of aging gene prediction over whole-network baselines in some benchmarks, but not in all metrics or under an alternative ground truth.","lead":"This paper benchmarks supervised prediction of aging-related genes using an aging-specific protein interaction network against the full interaction network and existing feature sets. The aging-specific network improves precision in several tests, but the advantage is not consistent across evaluation metrics or ground truth definitions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The reported advantage is not established because the best model per feature is selected on the same test folds used for evaluation (Section 2.3.1), so AUPR, precision, F-score, and p-values are optimistically biased; label-derived features add a second leakage channel.","rationale":"The paper's central claim is that an aging-specific subnetwork improves supervised prediction of aging-related genes and that AGENT features outperform existing methods. For this claim to be true, the evaluation must give unbiased estimates of generalization and the comparison must be fair. Section 2.3.1 does neither: the same 5-fold test predictions are used both to select the best dimensionality/classifier combination and to report all accuracy measures. Consequently, the paired Wilcoxon tests in Section 2.3.3 are not valid, because the five fold-wise values come from models chosen on those very folds. This is not a minor implementation detail; it is the main quantitative support for hypotheses 1 and 2. The GTEx-DAG null result in Section 3.6 is also concerning, but the paper discusses it at length; the leakage is more directly fatal because it can create the appearance of signal where none exists. The label-derived cSGDV and UniNet features compound the problem, since GenAge labels of all nodes are used to build features before the train/test split, allowing test labels to influence training features. I credit the paper for being transparent about the AUPR result, the dimensionality-reduction debate, and the GTEx-DAG failure; these disclosures are honest, but they do not repair the selection leakage. A nested cross-validation re-run with fold-aware feature computation would settle whether the claimed advantage survives an unbiased protocol. Since this concern supports the existing rejection rather than changing it, I leave the reader's verdict unchanged.","tokens_in":22089,"tokens_out":7132,"duration_ms":75363,"concrete_test":"Re-run the entire evaluation with nested cross-validation: use an inner 5-fold CV on each training fold to select the dimensionality/classifier combination by AUPR, then evaluate the selected model once on the untouched outer test fold; recompute all label-dependent features, namely cSGDV colors and UniNet aging-neighbor dimensions, inside each outer training fold using only training-set GenAge labels before training, and apply the same mapping to the test fold. Compare the best aging-subnetwork feature against the best whole-network feature in AUPR, precision, and F-score. If the aging-subnetwork advantage in precision or F-score shrinks to non-significance or reverses, the headline claim is an artifact of the current protocol.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.3.1 states that, for each feature, the authors choose the dimensionality_choice-classifier combination that maximizes AUPR on the 5-fold cross-validation predictions. Those same folds are then used to report AUPR, precision, recall, and F-score and to run paired Wilcoxon tests. This is test-set model selection: the reported numbers are effectively the maximum over 72 configurations of a noisy estimate on the evaluation data, so every feature is optimistically biased and the reported p-values are invalid. The bias is not necessarily equal across features, and it directly affects the comparisons used to support hypotheses 1 and 2. For example, Section 3.2's claim that DGDV, SGDV, and cSGDV significantly outperform features on the entire network is based on models selected and evaluated on the same folds. A second, independent leakage channel is feature construction: cSGDV (Section 2.2.1) colors all nodes by GenAge status before the train/test split, and UniNet includes aging neighbor counts and ratios computed from GenAge labels of all nodes, so labels from test nodes enter training features. This can manufacture above-chance separation on its own. Because the central claim, that the aging-specific subnetwork yields more accurate predictions, is supported only by metrics and p-values from this leakage-prone protocol, the claim is not yet established.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes AGENT, a supervised machine-learning framework for predicting human aging-related genes from an aging-specific protein-protein interaction (PPI) subnetwork. The subnetwork is derived by integrating age-specific gene expression data with the HPRD static PPI network, and the framework uses eleven network-based node features (seven dynamic, two static, plus the existing UniNet and mBPIs features) combined with nine classifiers and eight dimensionality-reduction choices. The central empirical claims are that (i) using the aging-specific subnetwork yields more accurate predictions than using the entire static PPI network, and (ii) the newly used features outperform existing UniNet and mBPIs. Evaluation is performed primarily with 5-fold cross-validation on a GenAge-based label set, reporting AUPR, precision, recall, and F-score, with additional validation on cancer gene sets. The paper also reports a secondary analysis using human expression-derived GTEx-DAG labels, where all methods perform near chance. Overall, the paper is comprehensive and richly detailed, but the validity of its central claims depends critically on the soundness of its evaluation protocol.","tokens_in":22224,"tokens_out":4959,"duration_ms":49835,"significance":"If the empirical claims were valid, the paper would make a useful contribution by demonstrating that context-specific PPI subnetworks, built by integrating expression data with interaction data, carry predictive signal for supervised aging-gene classification that a static interactome does not. It is also one of the first systematic supervised frameworks for this task on such subnetworks, and it introduces several graphlet- and centrality-based features to the aging-prediction domain. The paper includes a valuable negative result (GTEx-DAG chance-level performance) and openly discusses limitations. However, the significance is conditional on the evaluation being unbiased; the central comparison is currently compromised by test-set leakage in model selection and by label-derived features, which means the reported advantages and p-values do not establish the paper's main hypotheses.","major_comments":[{"comment":"The model-selection protocol leaks test information. For each feature, the dimensionality_choice–classifier combination that maximizes AUPR is chosen using the same 5-fold cross-validation predictions that are subsequently reported for AUPR, precision, recall, and F-score, and that are used in the paired Wilcoxon tests (Section 2.3.3). This is effectively selecting the best of 72 configurations on the evaluation folds; the reported metrics are maxima over a noisy estimate on the test data, so they are optimistically biased, and the reported adjusted p-values are invalid. This bias directly affects the comparisons supporting hypotheses 1 and 2 in Sections 3.2 and 3.3. The authors should use a nested cross-validation or otherwise ensure that model selection is performed only on training folds, with the test folds reserved strictly for final evaluation.","section":"2.3.1"},{"comment":"Two features encode the aging ground truth directly and therefore leak test labels into training. cSGDV colors every node in the network (except the node currently being featurized) with one of three colors based on its GenAge status from Section 2.1.3, and these colors are computed on the full network before any train/test split. For a test node, its cSGDV feature therefore includes GenAge membership of other test nodes, which is label information from the test set. Similarly, UniNet's feature includes 'aging neighbor count' and 'aging neighbor ratio' computed from GenAge labels of all network nodes. Because these features are extracted before cross-validation, the labels of test nodes enter the feature vectors used for training, which can manufacture above-chance separation on its own. Features should be recomputed within each cross-validation fold using only training-node labels (and the node's own label should not be used at all).","section":"2.2.1"},{"comment":"The choice of the entire context-unspecific network is made after comparing performance on HPRD versus BioGRID, and HPRD is retained because it gives better results ('performance is better on the HPRD network than on the BioGRID network (results not shown)'). This is outcome-based data selection: the baseline 'entire network' is chosen because it is favorable to the paper's hypotheses. Since the aging-specific subnetwork is derived from HPRD, the comparison between subnetwork and entire network is not a fair test of the hypothesis that context specificity helps; a different entire network might change the results. The authors should report the BioGRID comparison in full, or justify the network choice on independent grounds, ideally by evaluating all hypotheses on both entire networks.","section":"2.1.1"},{"comment":"The central claim is contingent on the GenAge ground truth. When a human expression-derived definition (GTEx-DAG) is used instead, all features perform at chance and none is significantly better than random (adjusted p-values ≥ 0.054). The paper reports this negative result, but the abstract's general assertion that 'using an aging-specific subnetwork indeed yields more accurate aging-related gene predictions than using the entire network' is not supported under this alternative label set. The conclusions should be explicitly scoped to the GenAge-based definition, and the authors should discuss more thoroughly whether the method offers any advantage for predicting human expression-derived aging genes, given that the reported advantage disappears in that setting.","section":"3.6"}],"minor_comments":[{"comment":"There is a typo in the phrase 'we find hypotheis 3 to hold' where 'hypothesis' is misspelled.","section":"1.1"},{"comment":"The BioGRID comparison is mentioned as 'results not shown'; since this comparison is used to justify a load-bearing data choice, the results should be included at least in a supplement.","section":"2.1.1"},{"comment":"The text sometimes uses 'cGDV' instead of 'cSGDV' (for example, in the Figure 2 caption and in Section 3.1), which could confuse readers; please standardize the notation.","section":"2.2.1"},{"comment":"The explanation for why GDC uses up to 5-node graphlets (a limitation of the executable) is unusual; it would be helpful to state explicitly whether this choice could affect comparability with the other graphlet-based features.","section":"2.2.1"},{"comment":"In Table 2, the notation 'X + Y (Z)' is used, but for PCA the dimension Z varies across folds and is averaged; this should be stated directly in the table caption or in the surrounding text.","section":"3.1"}],"recommendation":"reject","confidential_remarks":"The manuscript has a serious methodological flaw in its evaluation protocol: model selection is performed on the test folds, and two of the features are constructed from the ground-truth labels of the full network before cross-validation. These issues invalidate the reported performance comparisons and p-values, meaning the paper's central claims are not currently established. While the framework and the negative GTEx-DAG result are interesting, a proper fix would require re-running all experiments in a leakage-free protocol (nested cross-validation and within-fold feature construction), which is a substantial re-analysis rather than a local revision. For this reason, I recommend rejection, although a future resubmission with unbiased evaluation could be considered."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth knowing: this is a legitimate first application of supervised node classification to an aging-specific PPI subnetwork, but the evaluation protocol leaks test information twice, so the headline comparison is not trustworthy. The authors are unusually transparent about some limitations—the GTEx-DAG results, the dimensionality-reduction variants—but they do not acknowledge the model-selection leakage.\n\nWhat is actually new: AGENT applies existing graphlet/centrality features and two existing methods (UniNet, 30BPIs) to a previously published dynamic aging-specific subnetwork. That specific combination is new. They run a broad benchmark: 14 feature-network combinations, 72 dimensionality/classifier configurations, multiple accuracy measures, plus cancer-related validation. That breadth is a plus. Their hypothesis 3 (dynamic beats static) failing in most tests is reported honestly, and Section 3.6's random-level results for GTEx-DAG are described candidly.\n\nThe problem: Section 2.3.1 chooses, per feature, the dimensionality-classifier combination with maximum AUPR on the same 5-fold test predictions that are later used for all reported comparisons and Wilcoxon p-values. That is test-set model selection; the reported AUPRs, precisions, F-scores, and p-values are optimistically biased, and the bias can differ across features. The stress-test note is correct on this. A second channel: cSGDV colors all other nodes by GenAge status before splitting, and UniNet includes aging-neighbor count/ratio computed from full-network labels. So test labels enter training features. The paper even tries to avoid circularity for cSGDV by coloring the target node neutral, but that does not fix leakage from other test nodes. Also, HPRD was chosen over BioGRID because HPRD gave better performance, with the comparison not shown; that is post-hoc selection, minor relative to the leakage.\n\nThe abstract says the aging-specific subnetwork yields more accurate predictions, but in the AUPR comparison the whole-network UniNet performs best, and the claim rests on precision/F-score at a threshold chosen from the same contaminated folds. The GTEx-DAG experiment, where all methods are at chance, adds a robustness concern. Still, I do not see fraud; the text is open about many limitations and negative results.\n\nFor whom: as a benchmark and as a source of candidate aging genes, it is not reliable until re-evaluated with nested cross-validation, fold-aware feature computation, and public code. As a teaching example of how easily test information leaks into model selection, it is useful. I would send it to a serious referee with a request for major revision; it is not acceptable in current form.","headline":"A legitimate first application of supervised prediction to an aging-specific PPI subnetwork, but test-set model selection and label leakage break the reported accuracy comparisons.","tokens_in":22922,"tokens_out":2053,"would_cite":false,"duration_ms":18842,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that supervised prediction of human aging-related genes improves when the classifier uses an aging-specific protein–protein interaction subnetwork rather than the entire static interactome.","keywords":["aging-related genes","protein-protein interaction network","context-specific subnetwork","supervised node classification","graphlet features","dynamic network","GenAge","gene expression integration"],"falsifier":"Rebuild the aging-specific subnetwork from a modern PPI interactome and RNA-seq expression data, rerun the same features against GTEx-DAG labels, and check whether any feature exceeds random AUPR; if none does, the context-specific advantage is confined to GenAge labels and the older network, while if some do, the chance-level result in Section 3.6 was an artifact of outdated data.","tokens_in":21723,"feed_emoji":"🧬","tokens_out":7448,"duration_ms":64181,"temperature":0.7,"pith_summary":"The paper proposes AGENT, a supervised framework that predicts human aging-related genes from a protein–protein interaction (PPI) subnetwork restricted to aging context, rather than from the full static interactome. It claims that this context-specific subnetwork generally yields more accurate predictions than the entire network, and that graphlet- and centrality-based node features used here for the first time in this task outperform existing UniNet and mBPIs features. The claims are tested with 5-fold cross-validation and by checking whether newly predicted genes appear among known cancer driver genes. The authors report that the subnetwork advantage holds in most evaluation tests, while the dynamic version of the subnetwork is not consistently better than its static version. However, when the ground truth is redefined using human expression data (GTEx-DAG), every method performs at chance, which the authors report as surprising and leave for future work.","feed_headline":"Aging-specific networks beat full interactome in gene prediction","feed_subtitle":"Context-specific subnetworks plus graphlet features beat whole-network baselines on precision in most tests.","key_machinery":"The load-bearing object is the aging-specific PPI subnetwork: a dynamic network of 37 age-specific snapshots, each an induced subgraph on genes significantly expressed at that age, plus a static version aggregating all snapshots. On these networks the paper computes node features that capture network position and its change over time—dynamic graphlet degree vectors, graphlet orbit transitions, centrality trajectories, static graphlet degree vectors, and colored graphlet degree vectors—and compares them against the UniNet centrality vector and the 30BPIs highest-degree-neighbor indicator under nine classifiers and eight dimensionality choices. The aging-specific subnetwork is the thing being tested: the argument runs by showing that features computed on it beat the same features, and the baselines, on the entire context-unspecific interactome from which it was derived.","core_discovery":"The central claim is that context matters: a supervised classifier for aging-related genes performs better, in most evaluation tests, when trained on a protein–protein interaction subnetwork restricted to aging-specific gene expression than when trained on the full static interactome. The authors evaluate fourteen feature–network combinations, pairing graphlet-based features (dynamic graphlet degree vectors, graphlet orbit transitions, static and colored graphlet degree vectors) and centrality-based features (eccentricity, k-core, degree, centrality mean and variation) with nine classifiers and eight dimensionality choices. Their best results come from features on aging-specific subnetworks: in terms of precision, dynamic DGDV on the dynamic subnetwork and SGDV and cSGDV on the static subnetwork outperform every feature on the entire network, and they also outperform the UniNet and 30BPIs baselines. The dynamic subnetwork does not consistently beat its static version, so the paper treats hypotheses 1 and 2—context-specific network and new features—as the contributions, not the dynamic aspect. When the ground truth is switched from GenAge to expression-derived GTEx-DAG labels, all features perform at chance, a result the authors report as surprising and leave for future work.","pith_inferences":["A testable extension the paper leaves implicit is to infer an aging-specific subnetwork via network propagation from a recent interactome and then check whether GTEx-DAG labels become predictable; the authors flag network propagation and newer data as future work but do not run this test.","The complementarity of predictions across features suggests that an ensemble or multi-view classifier would likely outperform every single feature; the paper mentions this as promising rather than testing it.","The GTEx-DAG result is consistent with a technology- or annotation-bias interpretation: the older HPRD network and microarray-derived subnetworks may embed sequence-based knowledge better than expression-based knowledge, which would explain why GenAge labels are learnable while GTEx-DAG labels are not."],"forward_implications":["If hypothesis 1 is correct, future computational prioritization of aging genes should use a context-specific subnetwork rather than the full interactome as the default network representation.","If hypothesis 2 is correct, graphlet-based node features—especially SGDV and cSGDV on the static aging-specific subnetwork—are better precision-oriented tools for this task than the UniNet and 30BPIs baselines.","The small overlaps between different features' newly predicted genes imply the features encode complementary topological information, so combining them, for example in an ensemble, should recover more true aging-related genes than any single feature.","The chance-level accuracy under GTEx-DAG labels means that with current networks, expression-derived human aging labels are not predictable from PPI topology, so the paper's reported advantage is specific to GenAge-defined aging genes.","The dynamic subnetwork's failure to consistently outperform the static one suggests that temporal aggregation alone does not improve supervised aging-gene classification, and better dynamic subnetwork inference would be needed to test the dynamic hypothesis."],"supporting_citations":[{"why":"Supplies the dynamic aging-specific subnetwork on which the framework's dynamic features are computed.","marker":"[4]"},{"why":"Provides the entire static PPI network used as the comparison baseline and as the source of the aging-specific subnetworks.","marker":"[24]"},{"why":"Defines the primary GenAge ground-truth label set for aging-related genes.","marker":"[32]"},{"why":"Baseline method whose 14-centrality feature is compared against the new features.","marker":"[8]"},{"why":"Baseline method whose highest-degree-neighbor feature (30BPIs) is compared against the new features.","marker":"[7]"},{"why":"Defines the dynamic graphlet degree vector feature used on the dynamic subnetwork.","marker":"[25]"},{"why":"Defines the static graphlet degree vector feature used on static networks.","marker":"[28]"},{"why":"Defines the colored graphlet degree vector feature that incorporates aging-related node labels.","marker":"[11]"},{"why":"Supplies the GTEx-DAG expression-derived labels used in the secondary ground-truth test.","marker":"[12]"},{"why":"Provides the aging-specific expression data from which the subnetworks' snapshots are built.","marker":"[14]"}],"fun_headline_variants":["Context-specific subnetwork boosts aging gene prediction","Aging-tuned networks beat full interactome in gene prediction","Subnetwork context sharpens aging gene prediction","Graphlet features on aging-specific networks outperform"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The results stand or fall with GenAge as the definition of aging-related genes: when that label set is replaced by human-expression-derived GTEx-DAG labels, every method drops to chance, so the reported predictive advantage is contingent on GenAge being the right ground truth.","fun_headline_variants_meta":{"raw":{"variants":["Context-specific subnetwork boosts aging gene prediction","Aging-tuned networks beat full interactome in gene prediction","Subnetwork context sharpens aging gene prediction","Graphlet features on aging-specific networks outperform"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1359,"prompt_tokens":1016,"completion_tokens":343,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":632,"completion_tokens_details":{"reasoning_tokens":284}},"tokens_in":632,"tokens_out":343,"duration_ms":3769,"temperature":1.0,"reasoning_tokens":284,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:48:41.814163+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Rebuild the aging-specific subnetwork from a modern PPI interactome and RNA-seq expression data, rerun the same features against GTEx-DAG labels, and check whether any feature exceeds random AUPR; if none does, the context-specific advantage is confined to GenAge labels and the older network, while if some do, the chance-level result in Section 3.6 was an artifact of outdated data.","supporting_citations":[{"cited_title":"Dynamic networks reveal key players in aging.Bioinformatics, 30(12):1721– 1729, 2014","cited_arxiv_id":null,"evidence_quote":"Supplies the dynamic aging-specific subnetwork on which the framework's dynamic features are computed."},{"cited_title":"Human protein reference database—2009 update","cited_arxiv_id":null,"evidence_quote":"Provides the entire static PPI network used as the comparison baseline and as the source of the aging-specific subnetworks."},{"cited_title":"Human ageing genomic resources: new and updated databases","cited_arxiv_id":null,"evidence_quote":"Defines the primary GenAge ground-truth label set for aging-related genes."},{"cited_title":"Prediction and characterization of human ageing-related proteins by using machine learning","cited_arxiv_id":null,"evidence_quote":"Baseline method whose 14-centrality feature is compared against the new features."},{"cited_title":"A data mining approach for classifying DNA repair genes into ageing-related or non-ageing-related","cited_arxiv_id":null,"evidence_quote":"Baseline method whose highest-degree-neighbor feature (30BPIs) is compared against the new features."},{"cited_title":"Exploring the structure and function of temporal networks with dynamic graphlets","cited_arxiv_id":null,"evidence_quote":"Defines the dynamic graphlet degree vector feature used on the dynamic subnetwork."},{"cited_title":"Uncovering biological network function via graphlet degree signatures","cited_arxiv_id":null,"evidence_quote":"Defines the static graphlet degree vector feature used on static networks."},{"cited_title":"From homogeneous to heterogeneous network alignment via colored graphlets","cited_arxiv_id":null,"evidence_quote":"Defines the colored graphlet degree vector feature that incorporates aging-related node labels."},{"cited_title":"An analysis of aging-related genes derived from the genotype-tissue expression project (GTEx)","cited_arxiv_id":null,"evidence_quote":"Supplies the GTEx-DAG expression-derived labels used in the secondary ground-truth test."},{"cited_title":"Gene expression changes in the course of normal brain aging are sexually dimorphic","cited_arxiv_id":null,"evidence_quote":"Provides the aging-specific expression data from which the subnetworks' snapshots are built."}],"review_version":1}