{"id":"d2a774f3-6da8-40ec-b068-c3d65a0a9170","arxiv_id":"1908.06817","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"On COSMIC/TCGA RNA-seq data, Random Forest and Gradient Boosting classify 17 cancer types with 99.89% and 99.68% accuracy on a single 75/25 test split.","lead":"This paper compares five machine learning models for classifying 17 cancer types from RNA-seq gene expression data. Random forest and gradient boosting achieved around 99 to 100 percent per-class accuracy on a held-out test set, suggesting tumor type can be identified from gene activity.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Train/test independence is the load-bearing assumption: if COSMIC rows from the same patient or the same TCGA batch straddle the split, near-100% accuracy could reflect patient/batch identity rather than cancer biology.","rationale":"The central claim is plausible: with 16-18k expression features and multiple tissue-of-origin markers, random forest and boosting often achieve near-perfect pan-cancer classification on TCGA, so I would not object simply because the result is surprising. The weak point is not the algorithm but the independence of the evaluation. The manuscript says the models were 'assessed with a test set ... cancer types were withheld' but gives no detail about how rows were allocated, whether the same patient could appear in both partitions, or whether any normalization or feature-selection step saw the test data. This matters because RNA-seq z-scores from the same tumour sample are nearly deterministic; one duplicate in both partitions can make a class appear separable. The paper's dismissal of batch effects is unconvincing: ensuring all data came from one lab removes cross-lab variation but not within-lab batch structure, and TCGA expression data are known to have project-associated batch effects. Since the reported test set is only a single split, there is also no estimate of variability, so the accuracy point estimates may not be stable. These are fixable: patient-stratified split, train-only feature selection, repeated cross-validation, and ideally an external cohort. The reader's verdict (CONDITIONAL) is therefore fair. My pass does not move the verdict; it sharpens the reason. I also note the internal inconsistency in feature counts (18,019, 18,015, 16,718) as a supporting sign that the pipeline description is too imprecise, but I do not rely on it as the primary attack. The issue here is reporting completeness, not author intent.","tokens_in":7505,"tokens_out":7355,"duration_ms":72739,"concrete_test":"Using COSMIC v80 CosmicCompleteGeneExpression and CosmicSample, map each row to the TCGA patient barcode (participant portion of the sample ID) and to batch/plate identifiers if available. Re-run RF and GBM with a patient-stratified split (all aliquots of a patient in train or test, same 75/25 ratio) and compute the feature-importance ranking on training rows only. If patient-stratified accuracy remains 99.89%/99.68%, the concern is resolved; if it drops, or if the original split is found to contain same-patient rows on both sides, the near-100% headline is an artifact of non-independence. If a single split still yields >99% accuracy, repeat with 5x2 cross-validation to obtain a variance estimate; a wide or low interval would show the single-split estimate is not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The reported RF/GBM accuracies (Table 2, 99.89%/99.68%) can only support the paper's central claim if the 1,408 test samples were independent of the 4,221 training samples. The paper splits rows 75/25 (Section 3) but never states that samples from the same TCGA patient were kept in one partition. COSMIC's TCGA-derived expression tables can include multiple aliquots/portions from the same tumour or multiple tumour samples from one patient; row-wise splitting would place near-identical expression profiles in both train and test. The paper's rebuttal of batch bias ('all expression values were generated by a single source', Section 3) is not sufficient, since cancer types in TCGA were often processed in disease-specific batches/plates; z-score normalization does not remove batch identity. A second, related gap is that the reduced-gene experiments use importance scores from RF and GBM (Section 3) without stating that these rankings were computed on the training set only; if any variable selection used the full dataset, the 20-gene >95% result is also optimistically biased. The manuscript's attribute-count inconsistency (18,019 vs 18,015 vs 16,718) reinforces that the pipeline is not described precisely enough to rule out these leakages.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares five machine learning models (random forest, gradient boosting machine, random ferns, support vector machine, and k-nearest neighbors) for multi-class classification of 17 cancer types from RNA-seq expression data obtained from COSMIC/TCGA. Using a single 75/25 train-test split of 5,629 samples and the full gene expression matrix, the authors report near-perfect accuracy for the ensemble methods (RF 99.89%, GBM 99.68%) and substantially lower accuracy for SVM (47.18%) and KNN (75.43%). They also perform feature reduction based on ensemble variable importance and report that a 20-gene panel maintains ensemble accuracy above 95%. The paper concludes that ensemble algorithms, particularly RF and GBM, are well suited to this classification task.","tokens_in":7722,"tokens_out":7349,"duration_ms":66250,"significance":"If the reported accuracies are obtained from a genuinely independent test set, the paper provides a useful data point that RNA-seq expression profiles contain strong cancer-type-specific signals that standard ensemble classifiers can exploit, and it identifies a compact gene panel worthy of further investigation. The study is commendable for comparing five algorithms on a single large dataset with per-class breakdowns, and the feature-selection analysis is a constructive step toward clinical translation. However, the current manuscript does not demonstrate the independence of the test set, and the feature-selection protocol is underspecified, so the headline accuracies are not yet fully supported.","major_comments":[{"comment":"The manuscript does not describe how the 75/25 split was performed; if the random split was applied at the row level without checking for duplicate or multiple samples from the same TCGA patient, near-identical expression profiles can appear in both training and test sets, which would make the reported 99.89% and 99.68% accuracies reflect sample identity rather than biological signal. Please state the splitting procedure, whether any de-duplication by patient or sample identifier was performed, and if necessary, redo the evaluation with a patient-level split.","section":"Section 3, 'tested with 1,408 samples' and Table 2"},{"comment":"Variable importance for feature selection must come from the training set only, but the paper does not state this. If the importance scores were computed using the full 5,629 samples or the ensembles were refit on the whole dataset before selection, the reduced-feature results (including the 20-gene >95% accuracy) are optimistically biased. Please specify the exact timing of feature selection relative to the split, or repeat the procedure within each training fold.","section":"Section 3, 'features were reduced based on a combination of the important variables selected by both ensembles' and…"},{"comment":"The feature count is stated as 18,019 in Section 2, 18,015 in the abstract and Section 3, and 16,718 after filtering in Section 3, and the text says the models were tested 'with the full 18,015 genes' despite the earlier filtering to 16,718. This inconsistency prevents the reader from knowing what feature matrix was actually used for training and testing; please reconcile these numbers and state the exact input dimensions for each experiment.","section":"Sections 2-3 and the abstract"}],"minor_comments":[{"comment":"The abstract says '100% accuracy in the classification of 14 out of 17 types of cancer,' but the text lists 15 classes (including Lung Squamous cell Carcinoma) that were classified with 100% accuracy by one of the ensemble algorithms; please verify and correct the count.","section":"Table 2 and the abstract"},{"comment":"The conclusion reports KNN accuracy as 75.13% while Table 2 shows an average of 75.43%; please reconcile the discrepancy.","section":"Conclusion"},{"comment":"In the Lung Squamous cell Carcinoma row, the KNN accuracy value 76.03 appears twice; please remove the duplicate.","section":"Table 2"},{"comment":"Reference [11] is cited for the statement that TCGA expression data are z-score normalized, but reference [11] is a random-forest gene-selection paper; please cite the appropriate TCGA documentation instead.","section":"Section 2, reference [11]"},{"comment":"Figure 4 lacks a legend and has minimal axis labels; please label the model lines and provide clear axis titles so the reader can distinguish the five algorithms.","section":"Figure 4"},{"comment":"The sentence 'RF builds classification trees are using a bootstrap sample of the dataset' contains a grammatical error; please proofread the manuscript.","section":"Section 2, paragraph on RF"},{"comment":"The statement 'This ensemble is not prone to over-fitting since splitting points are randomly chosen' is an oversimplification; consider rephrasing to reflect that random feature selection reduces variance but does not eliminate overfitting risk.","section":"Section 2, paragraph on RF"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a conference-style paper whose central claims rest on a single train-test split and an underspecified feature-selection pipeline. The independence and leakage concerns are substantive but potentially addressable with additional analysis, so I recommend major revision rather than rejection. The authors should be asked to clarify or redo the split and feature selection, and to reconcile the inconsistent feature counts, before the accuracies can be taken as reliable."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a straightforward benchmark: five standard machine learning models (RF, GBM, random ferns, SVM, kNN) trained on RNA-seq expression data for 17 cancer types from COSMIC/TCGA, with a 75/25 split and per-class accuracy. The result that RF and GBM get 99.89% and 99.68% accuracy is internally consistent with the table, and it's not a surprise—pan-cancer expression signatures are strong. What's useful here is the scale: 5,629 samples and 17 classes, more than most earlier comparisons, plus a per-class breakdown and a feature-reduction experiment down to 20 genes with accuracy staying above 95% for the ensembles. The paper also benchmarks random ferns, which is rarely seen. That part is worth credit.\n\nThe soft spots are real. The biggest is that the paper never says the 25% test split is patient-disjoint. COSMIC's TCGA-derived tables can contain multiple samples from the same tumour or patient, and a row-wise 75/25 split can put near-identical profiles on both sides. The authors' rebuttal to batch bias—that all expression values came from one source and were z-scored—does not address the fact that TCGA cancer types were processed in disease-specific batches. The reduced-feature experiment (top genes from RF/GBM importance) also doesn't state whether the importance ranking was computed on training data only; if it used the full dataset, the 20-gene result is optimistically biased. There's no code, no hyperparameters, no confidence intervals or repeated cross-validation, and the attribute count changes between 18,019, 18,015, and 16,718 with no explanation. These are fixable, but the first two are load-bearing for the headline number.\n\nI would send this to a serious referee. The central comparative claim—ensembles beat SVM and kNN on this high-dimensional expression data—is almost certainly robust. But the exact 99.89% accuracy should be treated as an upper bound until the authors show patient-level splitting and train-only feature selection, and ideally add an external cohort. A revised version that fixes those points would be a useful reference for anyone building expression-based diagnostic classifiers.\n\nFor a reading group, it's a good case study in subtle leakage, but I'd put it lower on the pile.","headline":"A useful but methodologically loose benchmark of five classifiers on 17 cancer types; near-perfect accuracy is plausible, but the single split and unclear train/test separation keep me from trusting the headline number.","tokens_in":8304,"tokens_out":3116,"would_cite":false,"duration_ms":31659,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Standard ensemble classifiers reach about 99.9% accuracy in identifying 17 cancer types from RNA-seq expression profiles, and a 20-gene panel keeps accuracy above 95%.","keywords":["cancer classification","RNA-seq gene expression","multi-class classification","random forest","gradient boosting machine","random ferns","feature selection","ensemble learning"],"falsifier":"Re-run the pipeline with feature selection and z-score normalisation performed separately inside each cross-validation fold, and check whether any sample identifier appears in both training and test splits; also shuffle the batch labels and see if the top genes still separate the 17 classes. If accuracy collapses or duplicates are found, the near-100% figure is an artefact of leakage rather than biological signal.","tokens_in":7226,"feed_emoji":"🧬","tokens_out":9755,"duration_ms":92202,"temperature":0.7,"pith_summary":"This paper tries to establish that standard ensemble classifiers, trained on whole-transcriptome RNA-seq expression profiles, can identify the type of cancer a tumour sample came from almost perfectly. On a public pan-cancer expression dataset of 5,629 samples spanning 17 cancer classes and more than 18,000 measured genes, the random forest and gradient boosting machine reach 99.89% and 99.68% accuracy on a withheld quarter of the data, with 14 of 17 classes classified at 100%. The authors further claim that reducing the features to a 20-gene panel keeps ensemble accuracy above 95%, which suggests a small, inexpensive diagnostic test might be possible. The paper also shows that support vector machines and k-nearest neighbours do far worse on the full gene set, but improve after feature selection.","feed_headline":"Ensemble models identify 17 cancers with 99.9% accuracy","feed_subtitle":"Two tree-based ensembles beat SVM and KNN outright, and a 20-gene panel keeps accuracy above 95%.","key_machinery":"The engine of the result is the ensemble-of-decision-trees family: random forest grows many deep trees on bootstrap samples and averages their votes; gradient boosting adds weak learners sequentially to correct residual errors; random ferns uses a constrained tree ensemble where splits are fixed and features are randomly chosen per fern. These methods are suited to data with tens of thousands of features and comparatively few samples because they can pick out informative gene combinations while ignoring noise. The other load-bearing component is feature selection: after training, the important variables from random forest and gradient boosting are combined to form ranked gene lists of size 80 down to 10, and models are retrained on those subsets.","core_discovery":"The central claim is that tumour type can be read from gene-expression levels with near-perfect reliability using ensemble decision-tree methods. Trained on 4,221 samples (75%) and tested on 1,408 withheld samples (25%), the random forest classifies 17 cancer types with 99.89% accuracy and the gradient boosting machine with 99.68%; both ensembles classify 14 of the 17 classes with 100% accuracy, and those classes span most of the major tissue sites in the study. The paper attributes the gap between ensembles and simpler models to the high-dimensional noise of RNA-seq data, where the number of genes far exceeds the number of samples. A follow-up feature-reduction experiment, using the top genes identified jointly by random forest and gradient boosting, shows that 20 genes are enough to keep accuracy above 95% for the ensembles, with defensin, keratin-associated, interferon, actin, ribonuclease, and olfactory-receptor genes among the most influential.","pith_inferences":["If the result survives a stricter protocol where feature selection and normalisation are computed only inside training folds, a clinically usable minimal gene panel could be developed and tested on independent cohorts.","The appearance of olfactory-receptor genes in the top-20 list, whose link to cancer is disputed in the literature, suggests the feature ranking may capture tissue-specific expression patterns that are not all causally tied to carcinogenesis; that distinction matters for biomarker interpretation.","A direct extension would be to test the same ensembles on samples from external hospitals or different sequencing platforms; if accuracy drops, batch effects rather than biology are driving part of the signal.","Comparing the 20-gene panel against random 20-gene panels would quantify how much of the accuracy comes from the specific selected genes versus the mere reduction in dimensionality."],"forward_implications":["RNA-seq expression profiles contain enough tissue-specific signal that a decision-tree ensemble can separate 17 cancer types on held-out samples with near-perfect accuracy.","A roughly 20-gene expression panel preserves most of the classification performance, pointing toward cheaper diagnostic assays than whole-transcriptome sequencing.","On high-dimensional expression data, support vector machines and k-nearest neighbours need feature selection to be competitive; without it their accuracy is 47.18% and 75.43%.","Because the dataset has no healthy samples, the models can only distinguish cancer type from other cancer types, not detect cancer against normal tissue.","The near-100% class-level results imply that the genetic mutations driving these tumour types produce expression signatures consistent enough across samples to be learned by a single model."],"supporting_citations":[{"why":"Describes the pan-cancer gene-expression project whose data all experiments use.","marker":"[8]"},{"why":"Supplies the random-forest gene-selection method the study relies on and the z-score normalisation description.","marker":"[11]"},{"why":"Provides the earlier five-cancer benchmark where KNN beat SVM and an ensemble, setting up the comparison.","marker":"[9]"},{"why":"Provides the earlier three-cancer benchmark where KNN beat ensemble methods, another baseline extended here.","marker":"[10]"},{"why":"Supplies the boosting and randomised-tree machinery behind the gradient boosting machine.","marker":"[15, 17]"},{"why":"Introduces the random ferns ensemble used as the third model.","marker":"[18]"},{"why":"Underpins the bootstrap decision-tree construction from which the random forest is built.","marker":"[6, 14, 15]"},{"why":"Links defensin/interferon and keratin genes to cancer, supporting the biological reading of the top-20 gene list.","marker":"[25, 26]"}],"fun_headline_variants":["RNA-seq cancer typing: tree ensembles hit 99.9%","Ensemble ML reads cancer type from RNA with 99.9% accuracy","17 cancers, 20 genes: ensemble models keep >95%","Tree ensembles outclass SVM and KNN on RNA-seq cancer typing","Surpassing 99%: ensemble models classify 17 cancers from RNA"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claim of near-perfect accuracy depends on the 1,408 held-out test samples being truly independent of the 4,221 training samples, with no duplicate tumour samples, no technical batch that correlates with cancer type, and no gene-selection or normalisation step that used the test data.","fun_headline_variants_meta":{"raw":{"variants":["RNA-seq cancer typing: tree ensembles hit 99.9%","Ensemble ML reads cancer type from RNA with 99.9% accuracy","17 cancers, 20 genes: ensemble models keep >95%","Tree ensembles outclass SVM and KNN on RNA-seq cancer typing","Surpassing 99%: ensemble models classify 17 cancers from RNA"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000781,"raw_usage":{"total_tokens":3471,"prompt_tokens":988,"completion_tokens":2483,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":604,"completion_tokens_details":{"reasoning_tokens":2388}},"tokens_in":604,"tokens_out":2483,"duration_ms":15615,"temperature":1.0,"reasoning_tokens":2388,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T12:33:25.505383+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline with feature selection and z-score normalisation performed separately inside each cross-validation fold, and check whether any sample identifier appears in both training and test splits; also shuffle the batch labels and see if the top genes still separate the 17 classes. If accuracy collapses or duplicates are found, the near-100% figure is an artefact of leakage rather than biological signal.","supporting_citations":[{"cited_title":"Nature genetics 45, 1113 (2013)","cited_arxiv_id":null,"evidence_quote":"Describes the pan-cancer gene-expression project whose data all experiments use."},{"cited_title":"BMC bioinformatics 7, 3 (2006)","cited_arxiv_id":null,"evidence_quote":"Supplies the random-forest gene-selection method the study relies on and the z-score normalisation description."},{"cited_title":"Asian Pacific Journal of Cancer Prevention 17, 835-838 (2016)","cited_arxiv_id":null,"evidence_quote":"Provides the earlier five-cancer benchmark where KNN beat SVM and an ensemble, setting up the comparison."},{"cited_title":"Egyptian Informatics Journal 18, 151-159 (2017)","cited_arxiv_id":null,"evidence_quote":"Provides the earlier three-cancer benchmark where KNN beat ensemble methods, another baseline extended here."},{"cited_title":"Pattern Recognition 45, 3141-3153 (2012)","cited_arxiv_id":null,"evidence_quote":"Introduces the random ferns ensemble used as the third model."}],"review_version":1}