REVIEW 4 major objections 4 minor 5 cited by
Good practices for evaluation of machine learning systems
T0 review · 4 major / 4 minor · reviewed 2026-08-11 · deepseek-v4-flash
Pith's one-line read This paper argues that the evaluation protocol, fixed before training, determines whether ML conclusions generalize.
desk verdict A useful, opinionated tutorial on ML evaluation; the advice is mostly sound, but it overrelies on the authors' own metrics and glosses over test-set representativeness in the bootstrap section. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The carrying mechanism is a three-part evaluation protocol. Data selection uses a held-out evaluation set that mirrors deployment, with a separate development set, and cross-validation or nested cross-validation when data is scarce. Metric selection centers on expected cost (EC), a cost-matrix-weighted generalization of error rate, and its normalized form (NEC) with a fixed reference of 1 for a naive system that ignores inputs; for posterior probabilities, proper scoring rules such as cross-entropy are recommended. Significance is handled by bootstrapping, which resamples the evaluation data (grouped by correlation-inducing conditions such as speaker identity) to produce confidence intervals for the metric, and can pool over random seeds and bootstrapped training sets when evaluating methods rather than frozen systems.
What would settle it
Take one evaluation set, compute bootstrap intervals for a metric, then measure the same system on several independent collections from the same deployment domain; if the measured values fall outside the intervals far more often than the declared confidence level, the assumption that the held-out set represents deployment is wrong.
Extended reading notes
Core claim
On its own terms, the paper's central claim is that the reliability of an ML experiment is set before training begins, by three decisions: selecting evaluation data that resembles deployment and is never used for development; choosing a performance metric derived from the application's actual costs, with the normalized expected cost and proper scoring rules as defaults; and quantifying variability with bootstrapped confidence intervals that account for the evaluation set, random seeds, and training data. If these decisions are made carefully, conclusions about systems and methods will transfer to unseen data; if not, reported results can be optimistically biased or simply irrelevant to the intended use.
Load-bearing premise
The guidance assumes that a single held-out evaluation set is a random sample of deployment conditions, so bootstrapping it gives intervals that cover what the system will do in practice.
Editorial extensions
If this is right
- If a held-out evaluation set mirrors deployment and is never touched during development, its reported metric should predict real-world performance.
- Choosing the metric from application costs means systems are optimized for what users lose from mistakes, not for generic accuracy.
- Confidence intervals from bootstrapping distinguish a real improvement from noise and prevent conclusions that flip with one seed or one test split.
- Reporting normalized metrics like NTE or NEC makes results comparable across datasets with different class priors.
- Slicing the evaluation set by demographic or acoustic subgroups turns an average number into a bias check.
Reading between the lines
- Editorial inference: if these practices became standard, single-number leaderboard rankings would be seen as incomplete; rankings would need cost matrices, data provenance, and confidence intervals to support claims.
- Editorial inference: the silence-based spurious-correlation diagnostic for speech suggests an analogous test for other modalities, such as predicting a class from image background patches, as a cheap check before trusting a benchmark.
- Editorial inference: the paper's downstream-task principle implies a concrete experiment: measure how often a proxy metric's system ranking differs from the downstream-task ranking across several applications; the disagreement rate would quantify the cost of informal metric choice.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper is a position/review article on evaluation protocol design for machine learning systems. It argues that the evaluation procedure—data selection, metric selection, and statistical treatment—should be designed and fixed before experimentation, and it gives practical guidance for each component: hold out the evaluation data and match it to deployment conditions; choose metrics that reflect application-specific costs, recommending expected cost (EC), normalized expected cost (NEC), normalized total error (NTE), and normalized cross-entropy (NCE); and use bootstrap confidence intervals to quantify variability, distinguishing evaluation of frozen systems from evaluation of methods. The paper illustrates these points with speech-processing examples and provides a list of common pitfalls in each area.
Significance. If taken as a checklist, the paper is genuinely useful: it collects practices that are often violated in applied ML, gives concrete speech-processing examples, and points to publicly available bootstrapping code. The distinctions between systems and methods and between categorical decisions and posterior probabilities are helpful clarifications. However, the paper's authority rests on several strong normative claims that are asserted rather than derived or independently validated in the manuscript, and the confidence-interval protocol assumes an exchangeability condition that is not verified. The core message is reasonable and defensible, but the load-bearing claims need qualification and support before the paper can be recommended for publication.
major comments (4)
- [§3.1] The paragraph beginning 'The most general and principled metric...' asserts that AUC, EER, and FPR@FNR are 'not appropriate' for categorical decision tasks and that F-beta and Matthews correlation coefficient should be avoided, but no derivation or empirical evidence is given. The expected-cost argument justifies EC when the decision stage is fixed, yet threshold-independent metrics can be legitimate development-time diagnostics when the operating point will be set later; the manuscript does not discuss this distinction. Please either provide the supporting derivation with explicit assumptions or weaken the blanket claims.
- [§4.1] The bootstrap confidence interval described in this section is valid for deployment performance only if the evaluation set is exchangeable with deployment data. Section 2.1 asks only that the evaluation data resemble the use-case data 'as closely as possible' and provides no procedure to verify representativeness, while Section 2.4 concedes that spurious correlations are hard to detect. If the evaluation set differs systematically from deployment, the bootstrap intervals will be too narrow. Add an explicit caveat and practical checks (e.g., report covariate overlap between evaluation and deployment, use multiple test sets, and test on temporally or geographically disjoint data).
- [§4.2] The paragraph beginning 'If we want to draw general conclusions...' suggests that the training data can be bootstrapped 'just like the test dataset in the section above.' This overstates the equivalence: bootstrapping the training set and retraining estimates sensitivity to resampling from the observed training data, not to drawing a new training sample from the underlying population or to a shifted deployment domain. In addition, correlated training samples (e.g., multiple recordings per speaker) require a clustered bootstrap, which the text does not mention. Please state these assumptions and limitations explicitly.
- [§3.5] NEC, NTE, and NCE are central to the paper's metric recommendations, yet they are introduced only by name with references to refs [4] and [5]; the defining formulas, the normalization denominators, and the claimed fixed-reference properties are not given in the manuscript. Since ref [4] is listed without a venue and ref [5] is an arXiv preprint, the reader cannot verify these claims without locating external documents. Provide at least the basic definitions and the normalization rationale in the text.
minor comments (4)
- [§4.1] In the sentence 'system A turns out to be better than B by 5% ... would system B still be better?', the final question appears to refer to system A; please correct the pronoun or rephrase.
- [§3.1] The name 'Matthew correlation coefficient' should be 'Matthews correlation coefficient'.
- [Footnote 1] The note 'v1.0: This document is a work in progress' is appropriate for an arXiv preprint but should be removed or replaced with a version statement in the published version.
- [§2.2] The statement that development data 'can, in principle, be determined in any way you like' is in tension with the following paragraph warning that development data should follow the same criteria as evaluation data; please rephrase to avoid the apparent contradiction.
Circularity Check
No circular derivation: the paper gives normative evaluation guidance; its metric definitions are explicit, its bootstrap method is externally referenced, and its self-citations are supportive rather than load-bearing.
full rationale
The paper is a methodological review, not a derivation. Its central claim—that the evaluation protocol should be designed before experimentation—is a normative recommendation, not an equation fitted to data. The metric section defines NEC, NTE, and NCE in the text: NEC is defined by normalizing expected cost by the naive-system cost, NTE is explicitly called 'a simple shift and scaling of the accuracy,' and NCE is defined by normalizing a proper scoring rule by the prior-only value. These are transparent definitions, so no fitted input is renamed as a prediction. The bootstrap confidence-interval advice is the standard method of Efron and Tibshirani [3], and the paper explicitly frames the question as resampling from the same domain; the exchangeability limitation raised in the skeptic summary is a validity condition, not a circular step. Self-citations [4] and [5] supply metric definitions and analyses, but the recommendations in Section 3 are argued in the text and are also supported by the external reference [2]; they are not invoked as a uniqueness theorem or used to forbid alternatives. The code link in Section 4 is an implementation, not evidence. Therefore no step in the paper reduces by construction to its inputs.
Assumptions & free parameters
assumptions (4)
- standard math Standard bootstrap methodology (Efron and Tibshirani) is valid for ML metrics.
- domain assumption The three-way split (train/dev/test) is the appropriate experimental design.
- ad hoc to paper The metrics NEC, NTE, NCE, developed in the authors' prior work, are appropriate replacements for accuracy, F1, and ECE.
- domain assumption The evaluation set must be representative of deployment data.
Cite this review
Pith. "Pith review of Good practices for evaluation of machine learning systems." pith.science (2026). https://pith.science/paper/NNNGUHXP
@misc{pith2026241203700,
author = {Pith},
title = {Pith review of: Good practices for evaluation of machine learning systems},
year = {2026},
howpublished = {\url{https://pith.science/paper/NNNGUHXP}},
note = {Machine review of arXiv:2412.03700}
}
read the original abstract
Many development decisions affect the results obtained from ML experiments: training data, features, model architecture, hyperparameters, test data, etc. Among these aspects, arguably the most important design decisions are those that involve the evaluation procedure. This procedure is what determines whether the conclusions drawn from the experiments will or will not generalize to unseen data and whether they will be relevant to the application of interest. If the data is incorrectly selected, the wrong metric is chosen for evaluation or the significance of the comparisons between models is overestimated, conclusions may be misleading or result in suboptimal development decisions. To avoid such problems, the evaluation protocol should be very carefully designed before experimentation starts. In this work we discuss the main aspects involved in the design of the evaluation protocol: data selection, metric selection, and statistical significance. This document is not meant to be an exhaustive tutorial on each of these aspects. Instead, the goal is to explain the main guidelines that should be followed in each case. We include examples taken from the speech processing field, and provide a list of common mistakes related to each aspect.
Figures
Figures from the paper (1 more)
Forward citations
Cited by 5 Pith papers
-
L2-Bench: An Evaluation Benchmark for Measuring LLM Capabilities in Second Language Education
L2-Bench provides a 1,000-task, expert-validated rubric benchmark showing that frontier LLMs score 50–86% on applied second-language learning-design competencies, with notable weakness on open-ended tasks.
-
Benchmarking Human and Automatic Speech Recognition of Diverse Speech: Initial Results
State-of-the-art speech recognizers match human word-error rates on Dutch child speech and outperform native listeners on older-adults and Flemish-teenager speech in a 120-utterance pilot.
-
A Semi-spontaneous Dutch Speech Dataset for Speech Enhancement and Speech Recognition
On real noisy Dutch semi-spontaneous speech, five of eight SOTA ASR models reach WER under 22%, yet five single-channel SE methods fail to improve and often degrade ASR.
-
Estimating the local star formation rate density from ASKAP RACS
A machine-learning-selected sample of 11,293 ASKAP radio galaxies yields a completeness-corrected local star formation rate density of (1.4 +/- 0.5) x 10^-2 solar masses per year per cubic megaparsec, consistent with ...
-
From Read Speech to Spoken Digits: A Task-Specific Evaluation of Speech Privacy With Informed Attackers
Task-specific digit-recognition attacks show that temporal smoothing, resampling, and shredding protect spoken digits less—and differently—than read-speech word-error rates suggest.
Reference graph
Works this paper leans on
-
[4]
L. Ferrer. Analysis and comparison of classification metrics. 09 2 022
-
[5]
L. Ferrer and D. Ramos. Evaluating posterior probabilities: Decis ion the- ory, proper scoring rules, and calibration. 08 2024
work page 2024
-
[1]
T. G. Dietterich. Approximate statistical tests for comparing s upervised classification learning algorithms. Neural Computation , 10(7), Oct. 1998
work page 1998
-
[2]
K. Dyrland, A. S. Lundervold, and P. Porta Mana. Does the evalu ation stand up to evaluation? a first-principle approach to the evaluation of classifiers. arXiv:2302.12006, May 2022
arXiv 2022
-
[3]
B. Efron and R. Tibshirani. An Introduction to the Bootstrap . Chapman and Hall/CRC, 1994
work page 1994
- [6]
-
[7]
T. Gneiting and A. E. Raftery. Strictly proper scoring rules, pre diction, and estimation. Journal of the American Statistical Association , 2007
work page 2007
- [8]
Show all 13 references
-
[9]
Ojala and G
M. Ojala and G. C. Garriga. Permutation tests for studying class ifier per- formance. Journal of Machine Learning Research , 11(62):1833–1863, 2010
2010
-
[10]
Peterson
M. Peterson. An Introduction to Decision Theory. Cambridge Introductions to Philosophy. Cambridge University Press, 2009
2009
-
[11]
Poh and S
N. Poh and S. Bengio. Estimating the confidence interval of exp ected performance curve in biometric authentication using joint bootstr ap. In Proc. ICASSP, Honolulu, Apr. 2007
2007
-
[12]
S. Raschka. Model evaluation, model selection, and algorithm s election in machine learning. arXiv preprint arXiv:1811.12808 , 2018
2018 arXiv
-
[13]
Y. Zhu, M. Imoussa ¨ ıne-A ¨ ıkous, C. Cˆ ot´ e-Lussier, and T. H. Falk. Investigat- ing biases in covid-19 diagnostic systems processed with automated speech anonymization algorithms. In 3rd Symposium on Security and Privacy in Speech Communication, 2023. 15
2023
Reviewed August 11, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.