Pith. sign in

REVIEW 2 major objections 5 minor 4 references

The paper claims that splitting a ten-class breast cancer subtype prediction into ten binary LASSO classifiers yields lower cross-validated error and closer survival curves than the standard Nearest Shrunken Centroids classifier.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · deepseek-v4-flash

2026-08-01 06:35 UTC pith:4H6JMWK6

load-bearing objection A competent reanalysis whose central comparison is invalid: the OVR 0.0572 is a weighted average of binary errors, not a multiclass misclassification error. the 2 major comments →

arxiv 2607.21814 v1 pith:4H6JMWK6 submitted 2026-07-23 stat.AP

Another look at predicting molecular breast cancer subtypes from the METABRIC data

Isabella Lazarov , Robert Tibshirani This is my paper
classification stat.AP MSC 62H3062P10
keywords breast cancer subtypesone-versus-restLASSOnearest shrunken centroidshigh-dimensional classificationKaplan-Meier curveslog-rank testgenomic prediction
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

On the paper's own terms, the central claim is that a One-Versus-Rest LASSO strategy — ten separate binary classifiers, one per breast cancer cluster, each with its own penalty strength — predicts molecular subtype labels in a 1,980-patient genomic cohort more accurately than the Nearest Shrunken Centroids classifier used in the original study, while selecting far fewer genes. The paper reports a cross-validated error of 0.0572 for OVR versus 0.0822 for NSC, and a median absolute log-rank z-score of 0.380 versus 0.581, meaning the OVR-assigned patients reproduce the survival trajectories of the original discovery clusters more closely. The two pooled multinomial LASSO variants, at 0.138 and 0.106, do not beat NSC, which is why the finding is specific to splitting the multinomial problem into binomial pieces. The authors take this as evidence that decomposing a multinomial classification task into several regularized binary classifiers is a workable route for genomic patient clustering.

Core claim

The discovery is that the One-Versus-Rest LASSO model dominates the comparison on all three metrics the paper uses: lowest cross-validated error (0.0572), fewest selected genes overall (401 vs NSC's 754, with fewer than half as many per cluster), and closest validation-set survival curves (median absolute log-rank z = 0.380 vs 0.581). The default ungrouped multinomial LASSO and the grouped multinomial LASSO both show worse error than NSC, so the reported advantage comes specifically from the OVR construction: ten binomial logistic regressions, each regularized with its own lambda chosen by cross-validation, pooled by a weighted average of per-class error rates. The paper also notes that OVR'

What carries the argument

The key mechanism is the One-Versus-Rest decomposition: the ten-class subtype problem is split into ten binary LASSO logistic regressions, each predicting membership in one class against all others, with a separate sparsity penalty per class chosen by ten-fold cross-validation; the ten minimal error rates are then pooled into a single weighted average. This per-class regularization is the load-bearing idea — it lets rare or distinct clusters keep their own gene signatures instead of sharing one multinomial penalty. The comparison baseline, Nearest Shrunken Centroids, is a generative classifier that centers each class on a shrunken centroid and thresholds class-specific t-statistics; on this

Load-bearing premise

The load-bearing premise is that a weighted average of ten per-class binomial cross-validation errors is comparable to the multiclass misclassification error reported for the other models; every binary classifier sees a nine-tenths majority 'not this class' label, so the two kinds of error count different mistakes.

What would settle it

Recompute the OVR model's predictions as a single multiclass label for every held-out patient (e.g., the class whose binary classifier fires, breaking ties somehow), count the misclassifications directly, and compare that rate with the reported 0.0822 and 0.138. If the direct multiclass error rises above NSC's, the paper's central superiority claim fails regardless of survival-curve similarity.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • If the reported error rates hold, OVR LASSO is a sparser, more accurate substitute for NSC when assigning new patients to the ten molecular subtypes.
  • Because OVR's validation-set survival curves match the original discovery clusters' curves best under the log-rank metric, subtype assignments made this way preserve disease-specific survival information at least as well as the original classifier.
  • Per-class gene sets of a few dozen genes, instead of 754, could give researchers shorter candidate lists for the biology behind each subtype.
  • The winning recipe — split a multiclass problem into regularized binaries with independent penalties — generalizes to other high-dimensional, many-class genomics prediction tasks.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The paper does not show that its headline error comparison is apples-to-apples: the OVR number is a weighted average of ten binary error rates, while the NSC and multinomial numbers are multiclass misclassification rates; the same 0.0572 may not survive if OVR's ten binary predictions are collapsed into one multiclass label and scored directly.
  • If a direct multiclass evaluation still favors OVR, the work would support the broader idea that, in high-dimensional settings with heterogeneous classes, per-class regularization is worth more than a shared multinomial penalty.
  • The survival-curve similarity metric (absolute log-rank z) is a useful validation criterion precisely because this task has no ground-truth validation labels; a natural test is whether it selects the same model as a direct multiclass error rate on a dataset with known labels.
  • A straightforward extension is to apply the same OVR-vs-NSC comparison to other tumor-type datasets with independent subtype labels to check whether the reported advantage is specific to this cohort.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. This paper re-analyzes the METABRIC breast cancer dataset to compare four supervised classifiers for the 10 molecular subtypes defined by Curtis et al. (2012): Nearest Shrunken Centroids (NSC, via pamr), ungrouped multinomial LASSO, grouped multinomial LASSO, and one-versus-rest (OVR) binomial LASSO (via glmnet). Using the same discovery/validation split as the original study, the authors report cross-validated misclassification errors, number of selected features, and similarity of each model's validation-set Kaplan-Meier curves to the discovery clusters via absolute log-rank z-scores. The headline claim is that the OVR model achieves the lowest CV error (0.0572) and the lowest median log-rank statistic (0.380), implying that OVR is both more accurate and sparser than NSC.

Significance. If substantiated, the finding that a sparse OVR LASSO outperforms NSC on METABRIC would be a practically useful contribution to reproducible genomic subtyping, potentially reducing the feature set from 754 genes to 401 and improving the fidelity of predicted survival trajectories. The paper has strengths: it reproduces the original Curtis et al. NSC labels (pamr accuracy 1.0 on the validation set), uses standard public R packages, and states that code will be available on GitHub. However, the central accuracy claim is undermined by comparing a weighted average of per-class binomial CV errors with multiclass misclassification errors, and the survival comparison lacks uncertainty quantification. These issues are correctable but require reanalysis; the paper in its current form does not establish the headline conclusion.

major comments (2)
  1. [Section 2.2 / Section 3.1] The 'average minimum CV error' for OVR is a class-size-weighted average of ten binary cross-validated errors, not a multiclass misclassification error. No rule is given to combine the ten binary classifiers into a single predicted label, so the reported value 0.0572 cannot be compared to pamr's 0.0822 or multinomial glmnet's 0.138. In fact, a trivial majority-class predictor for each binary problem would achieve a weighted average error of approximately 0.10, so 0.0572 only indicates the binary classifiers beat that baseline. The Abstract's claim that OVR 'produces the lowest misclassification error' is therefore unsupported. The authors should either define an explicit combination rule and report the resulting multiclass CV error, or compare all models with a common metric (e.g., multiclass Brier score or log-loss).
  2. [Section 3.2] The survival comparison is based on per-cluster absolute log-rank z-scores, and the claim that OVR 'most successfully reproduces' the discovery KM curves (median z-score 0.380 vs. 0.581 for pamr) is used as a central piece of evidence. However, these z-scores are reported without confidence intervals or any measure of uncertainty; the median is computed over only 10 clusters. Moreover, the model with the lowest median is selected on the same validation data used to compute the statistic, so the comparison is a selected maximum. To support the survival claim, the authors should provide bootstrap or permutation intervals for the median z-score (or an alternative measure) and, if possible, pre-register the comparison or hold out a separate test set.
minor comments (5)
  1. [Manuscript text] The final line of the manuscript, 'test accuracy plot: start vertical axis at 0.5, add grouped and ovr, cluster labels under points', appears to be an editorial note left in the text. It should be removed or turned into a proper figure caption/description.
  2. [Section 2.1] The sentence 'pamr classification resulted in a test accuracy of 1.0' is potentially confusing. This is a reproduction check against the downloaded Curtis et al. validation-set labels, not an independent estimate of NSC's generalization error. Please rephrase to avoid implying that NSC achieves 100% test accuracy on unseen data.
  3. [Section 3.1] The statement 'OVR model selects 401 total features' needs clarification: since OVR is ten separate binary models, the number 401 could be the sum of features used by each model (with overlaps) or the number of unique features across all ten. This affects the sparsity comparison with NSC and should be made explicit.
  4. [Section 3.1] The sentence about comparing predicted classes to the 'true' classes of the discovery set contradicts the earlier statement that the validation set has no ground-truth labels. The validation-set labels from Curtis et al. are themselves NSC predictions; comparisons to them (e.g., Figure 4) should be described as agreement with NSC, not as accuracy.
  5. [Section 2.1] The imputation step before cross validation is mentioned but not described. Please add a sentence specifying the imputation method, as this can affect the ranking of models.

Circularity Check

1 steps flagged

No significant circularity in the central LASSO-vs-NSC comparison; one minor tautological NSC reproduction check.

specific steps
  1. self definitional [Section 2.1, 'Data Acquisition and Preprocessing'; Section 2.3, 'NSC Classification using pamr']
    "pamr classification resulted in a test accuracy of 1.0, meaning it predicted every sample's cluster correctly. This confirms that the discovery set, validation set, and the NSC classifier were successfully reproduced. ... Since the pamr model reproduces the NSC classifier, it yielded a score of 1.0 for all clusters."

    The validation-set cluster labels were downloaded from Curtis et al. and were themselves produced by the NSC classifier. Running pamr, which implements NSC, on the same data returns those same labels by construction, so a test accuracy of 1.0 is a self-consistency check rather than independent confirmation. This step is circular but it is not load-bearing for the paper's central claim about OVR versus NSC.

full rationale

The main derivation chain is not circular: LASSO and NSC models are trained on the discovery-set cluster labels and evaluated on held-out folds by CV and against external validation-set survival data via Kaplan-Meier curves and log-rank statistics. No fitted parameter is renamed as a prediction, and no load-bearing conclusion depends on a self-citation or on a uniqueness theorem from the authors' prior work. The only true circular step is the pamr 'test accuracy of 1.0' check, which is a tautology because the validation labels were produced by NSC and pamr implements NSC; the paper even acknowledges that pamr reproduces the NSC classifier. This does not affect the OVR comparison. Separately, the reported OVR 'misclassification error' of 0.0572 is a class-size-weighted average of ten binary CV errors, not a multiclass misclassification error, and comparing it directly to pamr's and multinomial glmnet's multiclass CV errors is a comparability/correctness concern rather than a circularity concern. Overall, the circularity score is low.

Axiom & Free-Parameter Ledger

4 free parameters · 5 axioms · 0 invented entities

The central comparison rests on Curtis et al.'s cluster labels as the target, the METABRIC survival data as the external benchmark, and a modeling assumption that the OVR weighted-average error can stand in for multiclass error. The first two are domain assumptions; the third is an unproven paper-specific assumption.

free parameters (4)
  • lambda.min for ungrouped multinomial LASSO = 0.00842
    Selected by 10-fold cross-validation to minimize misclassification error; the resulting CV error 0.138 is reported as that model's performance.
  • lambda.min for grouped multinomial LASSO = not reported
    Selected by 10-fold cross-validation; the resulting CV error 0.106 is reported.
  • per-class lambda for OVR LASSO (10 values) = not reported
    Each of the ten binomial classifiers has its own lambda minimizing CV error; the weighted average CV error 0.0572 is reported.
  • pamr delta threshold = 0.582
    First delta in the 30-value sequence at which CV error reaches its minimum (0.0822).
axioms (5)
  • domain assumption The Curtis et al. integrative-clustering discovery labels are a meaningful supervisory target for subtype classification.
    All models are trained on these labels as Y; if the clusters are not reproducible or meaningful the comparison is moot. Section 1, Section 2.1.
  • domain assumption The downloaded validation cluster labels (generated by NSC in Curtis et al.) are an appropriate reference for measuring model similarity.
    Figure 4 compares LASSO predictions to these labels, but they are not independent ground truth; they are the output of the same family of classifier being compared. Section 2.3.
  • domain assumption Disease-specific survival data are accurate and censoring is non-informative for log-rank comparisons.
    KM curves and log-rank z-scores treat survival as the external benchmark. Sections 2.1, 3.2.
  • standard math Standard glmnet/pamr modeling assumptions hold (samples approximately independent, features suitably standardized, class-conditional normal model for NSC).
    Inherited from the cited R packages and not independently checked by the authors. Sections 2.2-2.3.
  • ad hoc to paper A weighted average of ten per-class binomial cross-validation errors is comparable to the multiclass misclassification error of the other models.
    Introduced in Section 2.2 and used in Section 3.1 to claim OVR is the most accurate. No derivation, simulation, or citation establishes this comparability, and it is false as a plain reading of the metric.

pith-pipeline@v1.3.0-alltime-deepseek · 5008 in / 16823 out tokens · 163017 ms · 2026-08-01T06:35:49.434590+00:00 · methodology

0 comments
read the original abstract

Classifying patients into different clusters based on genomic data can offer valuable insights into their projected disease-specific survival trajectories over time. Here we apply two supervised learning methods---Nearest Shrunken Centroids and LASSO--- to the METABRIC Breast Cancer data {metabric} of 1980 patients and 754 genes to perform this task. The {pamr} R package implements the Nearest Shrunken Centroids classifier and the { glmnet} R package is used to fit an ungrouped multinomial model, a grouped multinomial model, and a One-Versus-Rest model. Splitting our data into discovery and validation sets, we evaluate all four models' classification performance and the survival implications of their class predictions using cross validation and Kaplan-Meier curves. We find that the One-Versus-Rest model produces the lowest misclassification error of 0.0572 and the lowest median log-rank test of 0.380 statistic measuring the similarity between its Kaplan-Meier curves and the discovery set's true Kaplan-Meier curves. We show that a multinomial classification task split into several LASSO binomial classifiers offers promising results for patient clustering.

Figures

Figures reproduced from arXiv: 2607.21814 by Isabella Lazarov, Robert Tibshirani.

Figure 1
Figure 1. Figure 1: Graphic visualizing integrative clustering assignments for discovery set and NSC clus￾ter classification of validation set. Integrative clustering analyzes the gene expression and copy number variation data of the discovery set and produces subgroups into which to divide its sam￾ples. The NSC classifier is trained on these “true” labels to predict the cluster subgroups of the validation set samples. cancer… view at source ↗
Figure 2
Figure 2. Figure 2: Ungrouped glmnet CV curve plot. From left to right, the two dotted lines and the bottom horizontal axis show the λ value at which CV error is at its minimum (lambda.min = 0.00842, CV error = 0.138) and the maximum λ value at which CV error is within one standard error from the minimum (lambda.1se = 0.0122, CV error = 0.145). The values at the top of each dotted line are the average numbers of features sele… view at source ↗
Figure 3
Figure 3. Figure 3: pamr CV plot. The dotted line on the top plot shows the misclassification rate at the best delta threshold (best delta threshold = 0.582, CV error = 0.0822). The values at the top of each plot are the number of features or genes selected depending on the delta threshold value (number of features for best delta threshold = 754). The bottom plot shows the CV curves for each cluster and the dotted line indica… view at source ↗
Figure 4
Figure 4. Figure 4: Similarity to NSC-predicted cluster labels plot. The similarity score is measured by the proportion of patients assigned to a cluster by both glmnet and pamr out of the patients assigned to it by pamr only. The pamr model scores 1.0 across all clusters since it reproduces the NSC classifier and thus predicts the same clusters for all patients as those in the downloaded cluster data. All other points show h… view at source ↗
Figure 5
Figure 5. Figure 5: Number of features selected for each cluster. 8 [PITH_FULL_IMAGE:figures/full_fig_p008_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: KM curves panel per cluster for clusters 1 to 6. The vertical axis plots the disease￾specific survival probability of breast cancer patients in each cluster from 0 to 180 months based on their aggregate survival data. 9 [PITH_FULL_IMAGE:figures/full_fig_p009_6.png] view at source ↗
Figure 7
Figure 7. Figure 7: KM curves panel per cluster for clusters 7 to 10. The vertical axis plots the disease￾specific survival probability of breast cancer patients in each cluster from 0 to 180 months based on their aggregate survival data. 10 [PITH_FULL_IMAGE:figures/full_fig_p010_7.png] view at source ↗
Figure 8
Figure 8. Figure 8: Log-rank test statistics per cluster for each model. For clusters 3, 4, 6, 7, and 8, all three glmnet KM curves are more similar to the true discovery KM curves than are the pamr model’s. OVR glmnet’s log-rank test are lower than pamr’s for clusters 2, 3, 4, 6, 7, 8, and 9. For the rest, pamr performs better than at least one glmnet model. The median z-scores for ungrouped, grouped, OVR, and pamr are 0.508… view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

4 extracted references

  1. [1]

    Accessed: 2026-07-16

    cBioPortal for Cancer Genomics (n.d.), ‘Breast cancer (metabric, nature 2012 & nat com- mun 2016)’,https://www.cbioportal.org/study/summary?id=brca_metabric. Accessed: 2026-07-16

    2012

  2. [2]

    D., Tavar´ e, S., Caldas, C

    Wishart, G., Pinder, S., Watson, P., Markowetz, F., Murphy, L., Ellis, I., Purushotham, A., Børresen-Dale, A.-L., Brenton, J. D., Tavar´ e, S., Caldas, C. & Aparicio, S. (2012), ‘The genomic and transcriptomic architecture of 2,000 breast tumours reveals novel subgroups’, Nature486(7403), 1–15

    2012

  3. [3]

    & Tibshirani, R

    Friedman, J., Hastie, T. & Tibshirani, R. (2013), glmnet: Lasso and Elastic Net Regularized Generalized Linear Models. R package version 1.9-3. URL:http://www.jstatsoft.org/v33/i01/

    2013

  4. [4]

    & Chu, G

    Tibshirani, R., Hastie, T., Narasimhan, B. & Chu, G. (2001), ‘Diagnosis of multiple cancer types by shrunken centroids of gene expression’,Proceedings of the National Academy of Sciences 99, 6567–6572. test accuracy plot: start vertical axis at 0.5, add grouped and ovr, cluster labels under points 12

    2001