{"id":"507c1201-30e0-476b-9cd6-37f95e82b23e","arxiv_id":"2502.09686","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"Random Forest classified early vs late prostate cancer stage from TCGA RNA-seq with a best F1 of 83%, but the mean F1 over 100 runs was 70.2%.","lead":"This paper applies standard machine learning classifiers to RNA-seq data from prostate tumors to predict early versus late pathological T stage. The best test F1 score was about 83% for Random Forest, but that number comes from the best of 100 runs, not the average of 70.2%.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The headline RF F1 ≈83% is the maximum over 100 random splits, not the expected performance; the paper's own Table 5 mean RF F1 is 70.20%, tied with XGB, so the central claim rests on a selection artifact.","rationale":"The paper is a routine ML benchmark on public TCGA data, and the methodology is not inherently objectionable. The decisive problem is that the central claim—RF achieves approximately 83% test F1—is taken from the best of 100 random runs, while the paper's own Table 5 shows the mean RF F1 is 70.20%, identical to XGB's mean. This is a selection artifact that directly undermines the headline result. The reader's strongest_claim identified this issue, and I agree with the REJECT verdict. The reader's weakest_assumption (sample-count inconsistency, 486 vs 406) is also serious and should be corrected, but it is secondary: even with a consistent sample count, reporting the maximum over 100 splits as the model's expected F1 would still invalidate the claimed superiority. No code or exact preprocessing is provided, and the deep learning accuracies lack error bars, but those are reproducibility weaknesses rather than the core logical flaw. I would keep the reader's REJECT verdict unchanged, pending either a correction to report mean/CI performance or a predefined selection rule with a properly held-out final evaluation.","tokens_in":13672,"tokens_out":4869,"duration_ms":45806,"concrete_test":"Recompute Table 5 on a fixed set of 100 random 80/20 splits, storing F1 per split for each algorithm. Report mean, SD, 95% CI, and the percentage of runs where RF F1 ≥ 80%. Then run a paired permutation test comparing RF with XGB and LR on the same splits. If RF's mean F1 is not significantly higher than XGB/LR, or if fewer than 5% of runs reach 80%, the abstract's 'approximately 83%' and 'RF superior' claims should be replaced by distributional summaries of the 100-run mean.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing result is the claim that Random Forest classifies early vs late pathological T stage with test F1 ≈83%, superior to Logistic Regression (80%) and XGB/SVM (≈79%). Section 3.3 states results are based on 100 runs, and Table 5 reports both mean and best metrics: RF mean F1 = 70.20%, best F1 = 83.00%; LR mean = 69.30%, best = 80%; XGB mean = 70.20%, best = 79%. The abstract, contribution, and conclusion cite only the best run, and §3.3.1/§3.5 treat the 83% value as the model's performance. With a test set of only 82 samples (Table 2, regardless of the 486 vs 406 sample-count discrepancy), the maximum F1 over 100 random splits is highly sensitive to the split and to SMOTE/SelectFpr fitted on each training fold; it is not a stable estimate of expected generalization. Moreover, the superiority claim fails on mean F1, where RF and XGB are tied at 70.20%. Thus the central claim is unsupported unless mean or confidence-bounded performance beats the alternatives. The sample-count inconsistency is real but secondary; correcting it would not resolve the best-run selection problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript trains seven machine learning classifiers and several deep neural network configurations on TCGA prostate cancer RNA-seq data to classify tumors into early versus late pathological T stage, using SelectFpr feature selection, SMOTE, and stochastic feature augmentation. The principal reported result is that Random Forest achieves the highest test F1-score of approximately 83%, with deep learning reaching 71.23% accuracy under augmentation, alongside differential expression and GO enrichment analyses. The claims are undermined by reporting the best of 100 runs as the achieved performance and by internal inconsistencies in the reported sample size and resampling protocol.","tokens_in":13964,"tokens_out":5381,"duration_ms":42369,"significance":"The application is clinically relevant, and the use of a public dataset with repeated ML runs (100 per algorithm) is a strength; the tables present both mean and best metrics. However, the headline result is a selection artifact: the mean RF F1-score is 70.20%, tied with XGB, so the claimed superiority is not supported by the paper's own data. The internal sample-size contradiction further prevents reproducibility. If corrected and re-framed around mean performance with confidence bounds, the comparative evaluation could serve as a modest benchmark, but as written the central claim is not established.","major_comments":[{"comment":"The abstract, Section 1.4, Section 3.3.1, and the conclusion cite an RF F1-score of approximately 83% as the achieved result. Table 5, however, reports that 83% is the 'best' (maximum) F1-score across 100 runs, while the 'mean' RF F1-score is 70.20%, identical to XGB's mean of 70.20%. With a test set of only 82 samples, the maximum over 100 random splits is not an unbiased estimator of expected performance and is highly sensitive to the particular split and to the SMOTE and SelectFpr procedures fit on each training fold. The paper therefore does not supply evidence that RF outperforms LR (mean 69.30%) or XGB (mean 70.20%). The authors should base the central claim on mean performance with confidence intervals, or on a pre-specified model-selection rule, and should clearly distinguish 'best-of-runs' from expected performance.","section":"§3.3, Table 5"},{"comment":"Section 2.1 states that the dataset contains 486 primary prostate cancer tumors, and Section 3.1 repeats this. Table 2 reports class counts of 184 (early) and 302 (late), which sum to 486, yet the Total row is 406 and the train/test split sums to 324+82=406. These numbers cannot all be correct. Since every accuracy and F1-score in Tables 4–6 and 8 depends on the actual cohort, the discrepancy is load-bearing: the reader cannot know whether the results refer to 486 or 406 samples. The authors must reconcile the data matrix size, the class counts, and the train/test split, and state exactly which samples were used in each analysis.","section":"§2.1, Table 2"},{"comment":"In Section 3.4 and Table 8, all deep learning accuracies (baseline 67.12%, PCA 69.86%, ICA 60.27%, augmentation 71.23%, etc.) are reported as single-point estimates. Unlike the machine learning section, where 100 repeated runs are described (Section 3.3), the deep learning section reports no number of independent runs, no standard deviation, and no confidence interval. Single-point estimates from a train/test split of 82 test samples cannot support the comparative claims in Section 3.4.5 (e.g., that augmentation 'yields the best performance'). The authors should provide repeated-trial statistics for the neural network experiments.","section":"§3.4, Table 8"},{"comment":"The resampling protocol is described inconsistently. Section 2.7 states that k-fold cross-validation is used with the number of folds set to 5; Figure 2 shows 'Cross Validation 10-Fold'; and Section 3.3.2 and Table 6 report 10-fold cross-validation results. These are different protocols, and the metrics reported in Table 5 versus Table 6 (test-set performance versus cross-validation scores) are not clearly distinguished. The authors should specify the exact evaluation protocol used for each reported number and ensure Section 2.7, Figure 2, and the results sections agree.","section":"§2.7, §3.3.2, Fig. 2"}],"minor_comments":[{"comment":"The abstract contains '71. 23%' with a spurious space; it should read '71.23%'.","section":"Abstract"},{"comment":"The phrase 'ANOV A F-test' is a typo; it should be 'ANOVA F-test'.","section":"§2.3.1"},{"comment":"The augmentation noise scale is described only as 'very small'; Eqs. (8)–(10) do not give the concrete value used in the experiments, and Section 3.4.4 does not specify the noise standard deviation for the '10x' augmentation, so the approach is not reproducible.","section":"§2.5.2, §3.4.4"},{"comment":"The row labeled 'score func f classif' appears to be a formatting artifact; the table should state which feature-selection scoring function was used for each method, and the table header should be cleaned.","section":"Table 5"},{"comment":"Section 3.5 says the Random Forest model 'achieved an accuracy of 83%', but the tables report F1-score, precision, and recall; accuracy is a different metric, so the GO enrichment statement should use the correct term.","section":"§3.5"},{"comment":"The conclusion says 'maximum cross-validation score of 80.38%', while Table 6 shows the best RF with feature selection as 80.36%; these numbers should be reconciled.","section":"§4"},{"comment":"The staging labels list t1a, t1b, t1c, t2, t2a, t2b, t2c as early and t3a, t3b, t4 as late; the manuscript should clarify whether stage t2 without a letter is included and whether the same grouping was used for all classifiers.","section":"§2.1"},{"comment":"The name 'Na ¨ ıve Bayes' has a formatting issue with the dieresis; it should be 'Naïve Bayes'.","section":"§2.7.2"}],"recommendation":"major_revision","confidential_remarks":"The paper's placement in cs.LG is acceptable, but the novelty is limited given the standard nature of the methods. The main issue is the mismatch between the reported headline result and the paper's own tables, which must be corrected before any further consideration. I also note the absence of a code/data availability statement beyond the TCGA source; providing analysis code would improve reproducibility."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague, quick take on arXiv:2502.09686. It's a benchmark: seven off-the-shelf classifiers plus a dense network applied to TCGA prostate cancer RNA-seq for a binary early-versus-late pathologic T stage. The headline is that Random Forest reaches ~83% test F1. There is no new method here: feature selection is scikit-learn's SelectFpr, imbalance handling is SMOTE, and the 'stochastic feature augmentation' is Gaussian noise addition. That's not a sin, but it means the contribution is empirical, not methodological.\n\nWhat the paper does well: the classical ML part uses 100 repeated splits, reports mean and best metrics separately (Tables 4 and 5), and includes a 10-fold CV comparison with and without feature selection. That's more careful than much of the TCGA-ML literature. The differential expression and GO enrichment steps are standard and tie the features to plausible biology.\n\nThe load-bearing problem is that the abstract, contributions, and conclusion cite the best-run F1 of 83%, while the mean RF F1 reported in Table 5 is 70.20%, exactly tied with XGB. With an 82-sample test set, the maximum F1 over 100 random splits is not an estimate of expected performance; it is a selection statistic. So the claim that RF is superior is unsupported. A secondary but real issue is the sample-size inconsistency: Section 2.1 says 486 tumors, but Table 2 reports 406 total (the class counts sum to 486, while the Total row and 324/82 split sum to 406). This is never explained, and every accuracy depends on which cohort was used. The deep-learning numbers are single-point accuracies with no variance, and no code or exact preprocessing pipeline is supplied, so the benchmark is not reproducible.\n\nThese flaws are not fatal to the effort—the authors did run the experiments and reported both means and bests—but they sink the central claim. This is a routine benchmark with misleading headline reporting, not a result that merits referee time. A serious editor should desk-reject; the paper could become a minor benchmark contribution if the authors correct the data discrepancy, report mean metrics with confidence bounds, and stop cherry-picking the best run.","headline":"The 83% RF F1 is a best-run artifact; the mean is 70.2% and tied with XGB, so the paper's central claim does not hold.","tokens_in":14537,"tokens_out":3138,"would_cite":false,"duration_ms":26243,"reading_group":"no","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Random forest on gene-expression data separates early from late prostate cancer stage with an 83% best F1-score, the highest across seven machine learning methods and above deep learning.","keywords":["prostate cancer","pathological staging","RNA-seq","Random Forest","feature selection","SMOTE","deep learning","gene expression classification"],"falsifier":"Reconstruct the cohort from the paper's description, count the samples, and rerun the exact pipeline: SelectFpr at alpha=0.05 on the 60,660-gene matrix, SMOTE on the training split, grid-searched Random Forest with 5-fold cross-validation, and a 20% held-out test. If the total sample count is not 486, or if Random Forest's best F1 is not 83% and its mean F1 not 70.20%, the central claim as stated would not reproduce.","tokens_in":13471,"feed_emoji":"🧬","tokens_out":9678,"duration_ms":84722,"temperature":0.7,"pith_summary":"This paper tries to establish that a standard machine-learning pipeline can stage prostate cancer from gene-expression data, separating early-stage (T1/T2) tumors from late-stage (T3/T4) tumors. Using RNA-sequencing profiles of prostate tumors, the authors report that Random Forest, after false-positive-rate feature selection and SMOTE balancing, reaches a best test F1-score near 83%, the highest among seven classifiers, with a mean F1 of 70.20% over 100 runs. They also find that a deep neural network reaches 71.23% accuracy when trained with Gaussian-noise augmentation and 69.86% when trained on PCA-compressed features. If these results hold, transcriptomics could complement PSA, biopsy, and imaging for staging decisions.","feed_headline":"Random forest tops deep learning for staging prostate cancer from RNA","feed_subtitle":"Gene-expression classifier separates early from late tumor stage, showing transcriptomics can guide treatment intensity.","key_machinery":"The load-bearing mechanism is the pairing of SelectFpr feature selection with Random Forest. SelectFpr keeps genes whose univariate association with the early/late target passes a false-positive-rate threshold, reducing the 60,660-gene matrix to a smaller discriminative set; SMOTE generates synthetic minority-class samples to counter the imbalance between early and late cases; grid-search with 5-fold cross-validation selects hyperparameters. The paper attributes the 83% best F1 to this combination, contrasting it with deep-learning and PCA-based alternatives that reach lower accuracy.","core_discovery":"The central claim is that Random Forest extracts a usable stage signal from high-dimensional RNA-seq data for prostate cancer pathological T staging. The authors frame the finding as: after filtering 60,660 genes with SelectFpr at alpha=0.05, balancing classes with SMOTE, and tuning hyperparameters by 5-fold grid-search cross-validation, Random Forest achieved the best held-out precision, recall, and F1-score of 83%, ahead of Logistic Regression (80%), XGBoost and SVM (79%); its mean F1 over 100 runs was 70.20%. They interpret this as evidence that a relatively simple ensemble, not a deep network, is the most reliable transcriptomic classifier for this task, and that feature selection is the key to making the high-dimensional matrix usable.","pith_inferences":["Editorial inference: the paper's central number is the best of 100 runs (83% F1) rather than the typical run (70.20%), so a cautious reader should treat the 83% as an upper bound of reported performance, not the expected operating point.","Editorial inference: Section 2.1 says the cohort has 486 tumors, but Table 2 sums to 406; because every metric is computed on that cohort, the sample-size discrepancy is a load-bearing unknown that should be resolved before the numbers are used.","Editorial inference: a natural independent test is to run the same pipeline on other prostate cancer RNA-seq cohorts with pathological stage labels and check whether the mean F1 stays near 70% and which selected genes replicate.","Editorial inference: the result that augmentation helps the full-dimensional model but hurts PCA-compressed features suggests the order of noise addition and dimensionality reduction matters; testing interpolation between the two could clarify the mechanism."],"forward_implications":["If the claim holds, prostate cancer staging can be done from a tissue transcriptome with a small ensemble model, without deep learning or imaging-based grading, at least in a binary early-versus-late formulation.","Random Forest with feature selection would be the natural baseline for future transcriptomic staging studies, and deep learning would need to justify its added complexity with a clear accuracy gain.","The genes selected by the best Random Forest run point to cancer pathways, giving biologists a candidate list for stage-progression markers.","The 71.23% accuracy from noise-augmented deep learning suggests synthetic data can partly compensate for small sample sizes in high-dimensional gene-expression studies.","If staging predictions are reliable at the individual level, treatment intensity could be matched to aggressiveness, reducing both overtreatment of indolent tumors and undertreatment of advanced ones."],"supporting_citations":[{"why":"Supplies the SelectFpr feature-selection method used to filter the 60,660-gene matrix before classification.","marker":"[2]"},{"why":"Shows RNA-seq data can be combined with machine learning for cancer classification, the premise this study extends to staging.","marker":"[4]"},{"why":"Reports near-99% accuracy classifying prostate tumor location from gene expression, the closest prior gene-expression machine-learning result in prostate cancer.","marker":"[13]"},{"why":"Directly targets improved pathological staging of prostate cancer with classifiers, providing the task and baseline this study builds on.","marker":"[26]"},{"why":"Identifies differentially expressed transcripts in prostate cancer progression from RNA-seq with machine learning, motivating the stage-discrimination signal.","marker":"[28]"}],"fun_headline_variants":["RF hits 83% F1 for prostate cancer staging from RNA","Simple model beats deep learning for prostate cancer stage","Random Forest tops deep nets in RNA-based prostate cancer staging","RNA-seq staging: Random Forest hits 83% F1, beats deep learning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the dataset is exactly as described: the paper says 486 tumors, but its own tables sum to 406 samples, and every accuracy and F1 figure depends on which count is real.","fun_headline_variants_meta":{"raw":{"variants":["RF hits 83% F1 for prostate cancer staging from RNA","Simple model beats deep learning for prostate cancer stage","Random Forest tops deep nets in RNA-based prostate cancer staging","RNA-seq staging: Random Forest hits 83% F1, beats deep learning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2938,"prompt_tokens":964,"completion_tokens":1974,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":580,"completion_tokens_details":{"reasoning_tokens":1902}},"tokens_in":580,"tokens_out":1974,"duration_ms":13976,"temperature":1.0,"reasoning_tokens":1902,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T21:41:06.975489+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reconstruct the cohort from the paper's description, count the samples, and rerun the exact pipeline: SelectFpr at alpha=0.05 on the 60,660-gene matrix, SMOTE on the training split, grid-searched Random Forest with 5-fold cross-validation, and a 20% held-out test. If the total sample count is not 486, or if Random Forest's best F1 is not 83% and its mean F1 not 70.20%, the central claim as stated would not reproduce.","supporting_citations":[{"cited_title":"A study on comparative analysis of feature selec- tion algorithms for students grades prediction","cited_arxiv_id":null,"evidence_quote":"Supplies the SelectFpr feature-selection method used to filter the 60,660-gene matrix before classification."},{"cited_title":"Machine learning analysis of rna-seq data for diagnostic and prognostic prediction of colon cancer","cited_arxiv_id":null,"evidence_quote":"Shows RNA-seq data can be combined with machine learning for cancer classification, the premise this study extends to staging."},{"cited_title":"Prediction of tumor location in prostate cancer tissue using a machine learning system on gene expression data","cited_arxiv_id":null,"evidence_quote":"Reports near-99% accuracy classifying prostate tumor location from gene expression, the closest prior gene-expression machine-learning result in prostate cancer."},{"cited_title":"Machine learning for improved pathological staging of prostate cancer: a performance comparison on a range of classifiers","cited_arxiv_id":null,"evidence_quote":"Directly targets improved pathological staging of prostate cancer with classifiers, providing the task and baseline this study builds on."},{"cited_title":"Identifying differentially expressed transcripts associated with prostate cancer progression using rna-seq and machine learn- ing techniques","cited_arxiv_id":null,"evidence_quote":"Identifies differentially expressed transcripts in prostate cancer progression from RNA-seq with machine learning, motivating the stage-discrimination signal."}],"review_version":1}