Bayesian Receiver Operating Characteristic Metric for Linear Classifiers

Heikki Huttunen; Jari Niemi; Jussi Tohka; Syeda Sakira Hassan

REVIEW 2 major objections 5 minor 28 references

Bayesian Receiver Operating Characteristic Metric for Linear Classifiers

T0 review · 2 major / 5 minor · reviewed 2026-08-14 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper derives a closed-form expression for the posterior expected AUC of any linear binary classifier under Gaussian class-conditional densities with a shared covariance, so classifier quality can be assessed from one training pass…

desk verdict A genuine closed-form Bayesian AUC for Gaussian linear classifiers that is oversold in the abstract; their own MEG result shows the accuracy claim is conditional. read the letter →

arxiv 1908.08771 v1 pith:WE76B7QK submitted 2019-08-23 cs.LG stat.ML

Syeda Sakira Hassan , Heikki Huttunen , Jari Niemi , Jussi Tohka This is my paper · ORCID

classification cs.LGstat.ML MSC 62H3062F1568T10

keywords BayesianAUClinearclassifierROCcurveclosed-formexpressionnormal-inverse-Wishartpriorsmall-sampleestimationcross-validationGaussianclass-conditional

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to replace cross-validation as the default way to measure how good a linear classifier is. It derives a closed-form formula, CBAUC, for the posterior expected area under the ROC curve when the two classes are Gaussian with a common covariance and the prior is normal-inverse-Wishart. The formula uses only quantities available from a single training run, so no resampling or threshold sweeping is needed. In simulated small-sample settings and in a high-dimensional cancer proteomics dataset, the authors find CBAUC has lower mean absolute error and lower variance than 5-fold cross-validated AUC and than an empirical Bayesian AUC. The price is that the estimate inherits the Gaussian and shared-covariance assumption, and the authors report it is the worst of the three estimators when the training data is actually a mixture of two Gaussians.

What carries the argument

The load-bearing object is the normal-inverse-Wishart conjugate prior on $(\mu_1,\mu_2,\Sigma)$, which keeps the posterior in the same family and makes every integral tractable. The derivation chains three steps: the population AUC of a linear classifier under Gaussian classes is $\Phi(w^T(\mu_2-\mu_1)/\sqrt{2 w^T\Sigma w})$; averaging this Gaussian CDF over the posterior means gives another Gaussian CDF whose argument is a scaled posterior mean difference; and averaging over the inverse-Wishart posterior covariance reduces the remaining integral to a regularized incomplete $\beta$ function. The final identity has the same shape as the Bayesian error estimator the paper extends, but replaces the error-counting step with a $\beta$-function AUC expression.

What would settle it

A direct test: simulate training data from a two-component Gaussian mixture with overlapping components, fit a linear classifier, and compare CBAUC's error against 5-fold cross-validated AUC using a large held-out test set. The paper reports exactly this failure on the MEG data (CBAUC has the largest mean absolute error), so replicating that pattern in a fully synthetic mixture, or showing CBAUC's error exceeds cross-validation in the unequal-covariance Gaussian experiment of Figure 3, would settle that the superiority claim holds only under the model's distributional assumptions.

Watch

Extended reading notes

Core claim

The central claim is that the posterior expectation of the AUC of any linear classifier with weights $w$ has an exact closed form. Under Gaussian class-conditional densities $\mathcal{N}(\mu_1,\Sigma)$ and $\mathcal{N}(\mu_2,\Sigma)$ with a normal-inverse-Wishart prior, the posterior mean of the AUC is $$\mathrm{CBAUC}(w)=\frac12+\frac{\operatorname{sgn}(A^*)}{2}\, I\!\left(\frac{(A^*)^2}{(A^*)^2+w^T S^* w};\,\frac12,\,\frac{\kappa^*-P+1}{2}\right),$$ where $I(x;a,b)$ is the regularized incomplete $\beta$ function, $A^*$ is a scaled posterior mean difference along $w$, and $S^*$, $\kappa^*$ are posterior hyperparameters updated from sample means, sample covariances, and the prior. This makes the full Bayesian integral over means and covariance collapse to a one-line expression, so AUC estimation becomes a single pass over the training data. The paper also shows empirically that, when the distributional assumptions hold, this estimate is more accurate and less variable than 5-fold cross-validated AUC, especially with small training sets.

Load-bearing premise

The load-bearing assumption is that each class is Gaussian and the two classes share the same covariance matrix; the paper's own MEG experiment shows CBAUC becomes the worst estimator when the training data is a mixture of two Gaussians.

Editorial extensions

If this is right

AUC of a linear classifier can be estimated in one training run; no repeated training on data folds, so evaluation cost drops from multiple fits to a single fit plus a few matrix products.
In small-sample Gaussian settings the estimate has lower mean absolute error and lower variance than 5-fold cross-validated AUC and the empirical Bayesian alternative, making model selection less noisy.
Because only $w$, the posterior means, and $S^*$ enter the formula, the same expression applies to any linear classifier: linear discriminant, support vector machine, or logistic regression.
The closed form remains valid in $P\gg N$ regimes because the posterior scale $S^*$ is non-singular even when sample covariances are singular, so the estimator works on high-dimensional data such as the ovarian cancer set.
CBAUC stays comparatively stable as class imbalance increases, while the counting-based estimators deteriorate at extreme imbalance ratios.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

The paper does not pursue it, but the same integral structure should extend to partial AUC or to precision-recall area, since those are also one-dimensional integrals over a Gaussian CDF evaluated along a linear discriminant; a beta-function form may appear there too.
Because CBAUC is a closed-form and differentiable function of the classifier weights, it could be used directly as an AUC surrogate in hyperparameter search, replacing the costly inner cross-validation loop that currently scores each candidate setting.
The paper notes the estimator has higher bias but smaller variance than counting methods; an untested corollary is that CBAUC may rank models correctly even when its absolute AUC is overestimated, which would make it especially suited to model selection rather than to reporting absolute performance.
A natural stress test is to replace the common-covariance assumption with a prior that allows $\Sigma_1\neq\Sigma_2$; the linear classifier is then no longer Bayes-optimal, and it is unclear whether the closed form degrades gracefully or collapses as it does in the MEG mixture.

Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Colleague,

The real contribution here is Eq. (14): a closed-form posterior expectation of AUC for any linear classifier under Gaussian class conditionals with a common covariance and a normal-inverse-Wishart prior. As far as I can tell from the cited literature, that is new. [20] does Bayesian AUC by sampling; [18] does Bayesian error, not AUC. The derivation is sound, and I checked the algebra in Lemma 1; it reuses [18, Lemma E.1], so it is not fully independent, but it is a genuine extension.

The experiments give it real support where the model holds. On synthetic Gaussian data with equal covariance, CBAUC has lower mean absolute error and lower variance than CV-AUC and EBAUC in small samples. The ovarian cancer P>>N result is consistent, and the computational advantage of a single training pass is real.

The soft spots are largely about packaging. The abstract says CBAUC is "faster and more accurate" without qualifiers, but their own MEG experiment (Fig. 6a) shows CBAUC with the worst MAE. They explain this by the train set being a mixture of two Gaussians, which violates the model, and they also note in Section 3.1.1 that EBAUC handles violations better. That is honest, but it contradicts the abstract. The paper's actual value is a fast estimator under a specific model, not a universally better AUC. I would ask for an abstract rewrite. There is also no public code or data, which for a method paper is a reproducibility gap. Minor point: the conclusion says they tested non-Gaussian data, but the reported robustness experiment is Gaussian with unequal covariances; I do not see a non-Gaussian experiment in the text.

This is a subfield improvement for people doing model selection in small-sample, high-dimensional settings where Gaussian common covariance is reasonable. I would probably cite it if I worked on Bayesian classifier metrics, and it deserves a serious referee: the closed form is correct, the limitations are acknowledged inside the paper, and the main fix is reframing the claims.

Referee Report

2 major / 5 minor

Summary. The paper introduces CBAUC, a closed-form Bayesian estimator of the area under the ROC curve for linear binary classifiers. The estimator is defined as the posterior expectation of the population AUC, assuming Gaussian class-conditional densities with a common covariance matrix and a normal-inverse-Wishart prior. The main theoretical result, Eq. (14), expresses CBAUC as a regularized incomplete beta function of the posterior mean difference and the posterior scale matrix. Experiments compare CBAUC with 5-fold cross-validated AUC and with an empirical Bayesian AUC estimator on synthetic Gaussian data, an ovarian cancer proteomics dataset (P >> N), and the ICANN 2011 MEG dataset. The synthetic and ovarian experiments are favorable to CBAUC, while the MEG experiment shows CBAUC with the worst mean absolute error, which the authors attribute to a violation of the Gaussian common-covariance assumption.

Significance. The derivation is technically plausible: the algebra in Lemma 1 and the reduction to Eq. (14) via the inverse-Wishart expectation are internally consistent, and the estimator is a genuine single-pass closed-form quantity. The paper's strength is that it supplies an analytic expression for a Bayesian AUC estimate, avoiding sampling or cross-validation, with explicit distributional assumptions. If the result is accepted as a posterior expectation under the stated model, it is a useful contribution for small-sample Gaussian settings and for model selection where variance reduction matters. However, the advertised practical advantage is conditional: the paper's own experiments show that CBAUC is not more accurate when the Gaussian equal-covariance assumption fails, so the significance of the contribution depends on an honest statement of that scope.

major comments (2)

[Abstract; Section 3.1.3; Section 4] The unqualified claim in the Abstract that CBAUC is 'more accurate than conventional AUC estimators' is contradicted by the paper's own results in Section 3.1.3: in Fig. 6a, CBAUC has the largest mean absolute error of the three estimators for most training-set sizes on the ICANN 2011 MEG data. The authors' explanation (the training set is a mixture of two Gaussians, violating the model assumptions) is reasonable, but it demonstrates that the method's superiority holds only under the Gaussian equal-covariance assumption stated in Section 1. The Abstract and Section 4 should state this qualification explicitly, for example 'under Gaussian class-conditional densities with a common covariance,' rather than presenting CBAUC as unconditionally faster and more accurate.
[Section 3.1.1, Fig. 3] The unequal-covariance experiment (Sigma_1 != Sigma_2) in Fig. 3 is reported as showing that cross-validation is poorer than the Bayesian approach, but the text does not make clear that CBAUC itself is the most sensitive of the Bayesian estimators to this violation. The conclusion in Section 4 that CBAUC is superior 'provided that prior assumptions hold' should be tied to this specific failure mode. As written, the reader could take the robustness experiment as support for CBAUC when it in fact shows that the advantage is limited to the equal-covariance case.

minor comments (5)

[Section 2.2, Proposition 1 proof] The definition of A* in the proof has a typo in the denominator: it reads 'nu*_1 + nu*_1' instead of 'nu*_1 + nu*_2', and the reference to 'Eq. (8)' should be 'Eq. (13)'.
[Section 3.1.3, Fig. 6] Fig. 6b, computed with a hold-out test set within the training set, is not discussed in the text; a sentence interpreting this panel would help the reader understand the experimental protocol and the discrepancy between panels (a) and (b).
[Section 3.1.3] The assertion that the MEG training set is a mixture of two Gaussians is plausible but is not supported by a quantitative diagnostic; adding, for example, a normality test or a covariance-equality test would strengthen the explanation of why CBAUC fails in this experiment.
[Section 3.1.1] In the unequal-covariance paragraph, the sentence 'the cross-validation performance is poorer than that of the Bayesian approach' is ambiguous because Fig. 3 compares three estimators; please state explicitly which pairwise differences are being summarized.
[Abstract; Section 3] The claim that CBAUC is 'faster' than conventional estimators is not quantified anywhere; while the computational advantage is qualitatively clear because cross-validation is avoided, a small timing comparison would make the claim concrete.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: CBAUC is a direct posterior expectation under stated Gaussian/NIW assumptions; the paper's self-citations are not load-bearing.

full rationale

The derivation of Eq. (14) is a direct evaluation of the posterior expectation defined in Eq. (3), under the explicitly stated Gaussian class-conditional densities with common covariance and a normal-inverse-Wishart prior. No parameter of the estimator is fitted to the target AUC; the hyperparameters m1,2=0, S=I, nu1,2=0.5, kappa=P+2 are fixed and taken from external prior work [18]. The external lemmas imported from [18] are independent mathematical results, not author self-citations. The only self-citations ([19] and [23]) provide background on multiclass Bayesian error estimation and the MEG feature set; they are not load-bearing for the closed form. The paper's own MEG experiment (Section 3.1.3) reports 'the proposed Bayesian approach has the worst performance among the three estimators,' and the conclusions qualify the accuracy claim to hold only 'as long as the model assumptions are valid.' That is a robustness or correctness limitation, not circularity. Thus the derivation is self-contained relative to its stated assumptions.

Assumptions & free parameters 4 free parameters · 3 assumptions · 0 invented entities

The central derivation rests on two modeling choices (Gaussian common-covariance class conditionals, and a specific conjugate prior) and one cited mathematical lemma. The prior hyperparameters are listed as hand-chosen free parameters because they are constants set by the authors, though they are not fitted to optimize the reported results.

free parameters (4)

Prior mean hyperparameters m1, m2 = 0
Set to the zero vector following [18]; not fitted to the data, but chosen by hand and influencing the posterior means.
Prior scale matrix S = I
Identity matrix following [18]; hand-chosen, ensures S* positive definite.
Prior concentration ν1, ν2 = 0.5
Controls prior strength for the class means; set as in [18], not tuned to the experiments.
Prior degrees-of-freedom κ = P + 2
Hand-chosen following [18]; sets the shape of the inverse-Wishart prior on the covariance.

assumptions (3)

domain assumption Class-conditional densities are Gaussian with a common covariance matrix.
The entire derivation of the posterior and the closed-form AUC presumes this; Section 1 and Section 2.
domain assumption The prior is normal-inverse-Wishart with hyperparameters m=0, S=I, ν=0.5, κ=P+2.
Chosen following [18], not derived; the posterior formulas (Eq. 4-5) rely on conjugacy of this prior.
standard math The integral identity [18, Lemma E.1] for the expectation of Φ(A/√(w^TΣw)) under an inverse-Wishart.
This is the key step in Proposition 1's proof; the lemma is cited but not proved or stated in the paper.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Bayesian Receiver Operating Characteristic Metric for Linear Classifiers." pith.science (2026). https://pith.science/paper/WE76B7QK

@misc{pith2026190808771,
  author       = {Pith},
  title        = {Pith review of: Bayesian Receiver Operating Characteristic Metric for Linear Classifiers},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/WE76B7QK}},
  note         = {Machine review of arXiv:1908.08771}
}

read the original abstract

We propose a novel classifier accuracy metric: the Bayesian Area Under the Receiver Operating Characteristic Curve (CBAUC). The method estimates the area under the ROC curve and is related to the recently proposed Bayesian Error Estimator. The metric can assess the quality of a classifier using only the training dataset without the need for computationally expensive cross-validation. We derive a closed-form solution of the proposed accuracy metric for any linear binary classifier under the Gaussianity assumption, and study the accuracy of the proposed estimator using simulated and real-world data. These experiments confirm that the closed-form CBAUC is both faster and more accurate than conventional AUC estimators.

Figures

Figures reproduced from arXiv: 1908.08771 by the authors.

**Figure 2.** Accuracy assessment with respect to the target AUC. ( [PITH_FULL_IMAGE:figures/full_fig_p009_2.png] view at source ↗

**Figure 3.** Accuracy assessment using multivariate Gaussian da [PITH_FULL_IMAGE:figures/full_fig_p010_3.png] view at source ↗

Figures from the paper (3 more)

**Figure 4.** Figure 4: Accuracy assessment with imbalanced class in multiv [PITH_FULL_IMAGE:figures/full_fig_p011_4.png]

**Figure 5.** Figure 5: Accuracy assessment using ovarian cancer dataset. M [PITH_FULL_IMAGE:figures/full_fig_p012_5.png]

**Figure 6.** Figure 6: Accuracy assessment using ICANN 2011 MEG binary data [PITH_FULL_IMAGE:figures/full_fig_p013_6.png]

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

28 extracted references · 27 canonical work pages

[20]

L. A. Dalton, Optimal ROC-based classiﬁcation and perf ormance analysis under Bayesian uncertainty models, IEEE /ACM Transactions on Computational Biology and Bioinformatics 13 (4) (2016) 719–729. doi:10.1109/TCBB.2015.2465966

arXiv 2016
[18]

L. A. Dalton, E. R. Dougherty, Bayesian minimum mean-sq uare error estimation for classiﬁcation error—part II: The Bayesian MMSE error estimator for linear classiﬁcation of Gaussian distributions, IEEE Tran s. Signal Process 59 (1) (2011) 130–144

work page 2011
[1]

Alpaydin, Introduction to machine learning, MIT pres s, 2014

E. Alpaydin, Introduction to machine learning, MIT pres s, 2014

work page 2014
[2]

Hastie, R

T. Hastie, R. Tibshirani, J. Friedman, The elements of st atistical learning, Springer, 2009

work page 2009
[3]

R. O. Duda, P . E. Hart, G. E. Stork, Pattern Classiﬁcation , 2nd Edition, John Wiley & Sons, Inc., 2001

work page 2001
[4]

Stone, Cross-validatory choice and assessment of sta tistical predictions, Journal of the royal statistical soc iety

M. Stone, Cross-validatory choice and assessment of sta tistical predictions, Journal of the royal statistical soc iety. Series B (Methodological) (1974) 111–147

work page 1974
[5]

E. R. Dougherty, C. Sima, B. Hanczar, U. M. Braga-Neto, Pe rformance of error estimators for classiﬁcation, Current B ioinformatics 5 (1) (2010) 53

work page 2010
[6]

Efron, Bootstrap methods: another look at the jackkni fe, in: Breakthroughs in statistics, Springer, 1992, pp

B. Efron, Bootstrap methods: another look at the jackkni fe, in: Breakthroughs in statistics, Springer, 1992, pp. 56 9–593

work page 1992

Show all 28 references

[7]

Efron, Estimating the error rate of a prediction rule: improvement on cross-validation, Journal of the American s tatistical association 78 (382) (1983) 316–331

B. Efron, Estimating the error rate of a prediction rule: improvement on cross-validation, Journal of the American s tatistical association 78 (382) (1983) 316–331

1983
[8]

Efron, R

B. Efron, R. Tibshirani, Improvements on cross-validat ion: the .632 + bootstrap method, Journal of the American Statistical Asso ciation 92 (438) (1997) 548–560

1997
[9]

Braga-Neto, E

U. Braga-Neto, E. Dougherty, Bolstered error estimatio n, Pattern Recognition 37 (6) (2004) 1267–1281

2004
[10]

Glick, Additive estimators for probabilities of cor rect classiﬁcation, Pattern recognition 10 (3) (1978) 211– 222

N. Glick, Additive estimators for probabilities of cor rect classiﬁcation, Pattern recognition 10 (3) (1978) 211– 222

1978
[11]

Ferri, J

C. Ferri, J. Hern´ andez-Orallo, R. Modroiu, An experimental comparison of performance measures for classiﬁcatio n, Pattern Recognition Letters 30 (1) (2009) 27–38

2009
[12]

L. A. Jeni, J. F. Cohn, F. De La Torre, Facing imbalanced d ata–recommendations for the use of performance metrics, in : A ﬀective Computing and Intelligent Interaction (ACII), 2013 Humaine Association Conference o n, IEEE, 2013, pp. 245–251

2013
[13]

H. He, E. A. Garcia, Learning from imbalanced data, IEEE Transactions on Knowledge & Data Engineering 21 (9) (2008) 1 263–1284

2008
[14]

Cohen, M

G. Cohen, M. Hilario, H. Sax, S. Hugonnet, A. Geissbuhle r, Learning from imbalanced data in surveillance of nosocom ial infection, Artiﬁcial intelligence in medicine 37 (1) (2006) 7–18

2006
[15]

A. P . Bradley, The use of the area under the roc curve in th e evaluation of machine learning algorithms, Pattern recog nition 30 (7) (1997) 1145–1159

1997
[16]

Cortes, M

C. Cortes, M. Mohri, Auc optimization vs. error rate min imization, in: S. Thrun, L. K. Saul, B. Sch¨ olkopf (Eds.), Advances in Neural Information Processing Systems 16, MIT Press, 2004, pp. 313–320

2004
[17]

L. A. Dalton, E. R. Dougherty, Bayesian minimum mean-sq uare error estimation for classiﬁcation error—part I: Deﬁn ition and the Bayesian MMSE error estimator for discrete classiﬁcation, IEEE Trans. Signal P rocess 59 (1) (2011) 115–129

2011
[19]

Huttunen, J

H. Huttunen, J. Tohka, Model selection for linear class iﬁers using bayesian error estimation, Pattern Recognitio n 48 (11) (2015) 3739 – 3748. doi:https://doi.org/10.1016/j.patcog.2015.05.005

2015 doi
[21]

O. V . Demler, M. J. Pencina, R. B. D’Agostino, Equivalen ce of improvement in area under roc curve and linear discrimi nant analysis coe ﬃcient under assumption of normality, Statistics in medicine 30 (12) (20 11) 1410–1418
[22]

Gelman, H

A. Gelman, H. S. Stern, J. B. Carlin, D. B. Dunson, A. V eht ari, D. B. Rubin, Bayesian data analysis, Chapman and Hall /CRC, 2013

2013
[23]

Huttunen, T

H. Huttunen, T. Manninen, J.-P . Kauppi, J. Tohka, Mind r eading with regularized multinomial logistic regression, Machine Vision and Applications 24 (6) (2013) 1311–1325

2013
[24]

Raudys, V

S. Raudys, V . Pikelis, On dimensionality, sample size, classiﬁcation error, and complexity of classiﬁcation algo rithm in pattern recognition, IEEE Transactions on Pattern Analysis and Machine Intelligence (3) (1980) 242 –252

1980
[25]

R. Fan, K. Chang, C. Hsieh, X. Wang, C. Lin, Liblinear: A l ibrary for large linear classiﬁcation, The Journal of Machi ne Learning Research 9 (2008) 1871–1874

2008
[26]

Chakraborty, Observer Performance Methods for Diag nostic Imaging, Imaging in Medical Diagnosis and Therapy, C RC Press, 2017

D. Chakraborty, Observer Performance Methods for Diag nostic Imaging, Imaging in Medical Diagnosis and Therapy, C RC Press, 2017

2017
[27]

M. S. Pepe, The statistical evaluation of medical tests for classiﬁcation and prediction, Medicine, 2003

2003
[28]

J. Xue, P . Hall, Why does rebalancing class-unbalanced data improve auc for linear discriminant analysis?, IEEE tr ansactions on pattern analysis and machine intelligence 37 (5) (2015) 1109–1112

2015

Pith tools

Reviewed August 14, 2026 · model on record in the stance chip above.

[1] [20]

L. A. Dalton, Optimal ROC-based classiﬁcation and perf ormance analysis under Bayesian uncertainty models, IEEE /ACM Transactions on Computational Biology and Bioinformatics 13 (4) (2016) 719–729. doi:10.1109/TCBB.2015.2465966

arXiv 2016

[2] [18]

L. A. Dalton, E. R. Dougherty, Bayesian minimum mean-sq uare error estimation for classiﬁcation error—part II: The Bayesian MMSE error estimator for linear classiﬁcation of Gaussian distributions, IEEE Tran s. Signal Process 59 (1) (2011) 130–144

work page 2011

[3] [1]

Alpaydin, Introduction to machine learning, MIT pres s, 2014

E. Alpaydin, Introduction to machine learning, MIT pres s, 2014

work page 2014

[4] [2]

Hastie, R

T. Hastie, R. Tibshirani, J. Friedman, The elements of st atistical learning, Springer, 2009

work page 2009

[5] [3]

R. O. Duda, P . E. Hart, G. E. Stork, Pattern Classiﬁcation , 2nd Edition, John Wiley & Sons, Inc., 2001

work page 2001

[6] [4]

Stone, Cross-validatory choice and assessment of sta tistical predictions, Journal of the royal statistical soc iety

M. Stone, Cross-validatory choice and assessment of sta tistical predictions, Journal of the royal statistical soc iety. Series B (Methodological) (1974) 111–147

work page 1974

[7] [5]

E. R. Dougherty, C. Sima, B. Hanczar, U. M. Braga-Neto, Pe rformance of error estimators for classiﬁcation, Current B ioinformatics 5 (1) (2010) 53

work page 2010

[8] [6]

Efron, Bootstrap methods: another look at the jackkni fe, in: Breakthroughs in statistics, Springer, 1992, pp

B. Efron, Bootstrap methods: another look at the jackkni fe, in: Breakthroughs in statistics, Springer, 1992, pp. 56 9–593

work page 1992