{"id":"d4f873b0-f297-441a-a72d-596bbd512d59","arxiv_id":"2412.03700","paper_version":1,"verdict":"UNVERDICTED","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"low","formal_verification":"none","parameter_count":0,"one_line_summary":"A tutorial and position piece outlining guidelines for data splits, metric selection, and confidence intervals for ML systems, with speech processing examples.","lead":"This paper surveys good practices for evaluating machine learning systems, covering data selection, metric choice, and statistical significance via bootstrapping. It argues that evaluation protocols should be fixed before experiments start and that metrics should reflect end-user value.","discovery_kind":"review","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The protocol's promise that careful evaluation design determines generalization rests on unverified test/deployment exchangeability; §4.1 bootstrap CIs can overstate confidence when this fails.","rationale":"The reader's verdict (UNVERDICTED) is appropriate for a position paper, and this stress-test does not move it. The most load-bearing concern is the exchangeability assumption behind the central claim and the Section 4.1 bootstrap procedure. The abstract says the evaluation procedure 'determines whether conclusions generalize,' but this can only be true if the evaluator already knows the deployment distribution. The paper gives no tool to verify representativeness, only qualitative advice, and Section 2.4 shows that even careful selection can miss spurious correlations. A quantitative sensitivity check, calibrating the bootstrap CIs on a shifted deployment sample, would directly test whether the protocol delivers its promised confidence. Other recommendations, such as preferring expected cost over F-beta or MCC and using bootstrapping for method comparison, are opinions or standard practice rather than falsifiable claims, so they are not the primary vulnerability. The v1.0 footnote in the manuscript itself flags the document as a work in progress, which supports treating the guidance as provisional; the concern should therefore be read as a caveat for a later version, not as grounds for rejection.","tokens_in":9687,"tokens_out":10261,"duration_ms":110189,"concrete_test":"Run a distribution-shift calibration experiment on a multi-domain dataset (e.g., a speech corpus with different recording conditions or a benchmark with temporal versions). Following Sections 2 and 4.1, select a test set from one domain or time slice, compute the chosen metric for two frozen systems, and construct a 95% bootstrap CI (resampling at the condition level as described in Section 4.1). Then compute the same metric on data from a later or shifted deployment slice. Repeat over many random splits and systems, and measure the empirical coverage of the deployment values. If coverage is substantially below 95%, the exchangeability assumption fails and the concern lands; if coverage is near nominal, the protocol is self-correcting for this type of shift.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that careful pre-experiment design of the evaluation protocol determines whether conclusions generalize, as stated in the abstract. This claim depends on the evaluator's ability to select an evaluation set that is representative of deployment conditions. Section 2.1 only says the evaluation data should resemble the use-case data as closely as possible; it offers no procedure to test whether this resemblance actually holds. Section 4.1 then constructs bootstrap confidence intervals from the observed evaluation set, but these are valid intervals for the deployment metric only if the evaluation set is a random sample from the deployment distribution. If the test set differs systematically from deployment (e.g., different microphone, dialect, or time period), the bootstrap intervals will be too narrow, producing exactly the false confidence the paper warns against. Section 2.4 acknowledges that spurious correlations are hard to detect, so the failure mode is silent. This is a limitation of the protocol's warrant rather than an internal contradiction: the advice is reasonable, but the strong claim that the procedure determines whether conclusions generalize overstates what can be guaranteed without a validation step for representativeness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper is a position/review article on evaluation protocol design for machine learning systems. It argues that the evaluation procedure—data selection, metric selection, and statistical treatment—should be designed and fixed before experimentation, and it gives practical guidance for each component: hold out the evaluation data and match it to deployment conditions; choose metrics that reflect application-specific costs, recommending expected cost (EC), normalized expected cost (NEC), normalized total error (NTE), and normalized cross-entropy (NCE); and use bootstrap confidence intervals to quantify variability, distinguishing evaluation of frozen systems from evaluation of methods. The paper illustrates these points with speech-processing examples and provides a list of common pitfalls in each area.","tokens_in":9891,"tokens_out":3988,"duration_ms":39774,"significance":"If taken as a checklist, the paper is genuinely useful: it collects practices that are often violated in applied ML, gives concrete speech-processing examples, and points to publicly available bootstrapping code. The distinctions between systems and methods and between categorical decisions and posterior probabilities are helpful clarifications. However, the paper's authority rests on several strong normative claims that are asserted rather than derived or independently validated in the manuscript, and the confidence-interval protocol assumes an exchangeability condition that is not verified. The core message is reasonable and defensible, but the load-bearing claims need qualification and support before the paper can be recommended for publication.","major_comments":[{"comment":"The paragraph beginning 'The most general and principled metric...' asserts that AUC, EER, and FPR@FNR are 'not appropriate' for categorical decision tasks and that F-beta and Matthews correlation coefficient should be avoided, but no derivation or empirical evidence is given. The expected-cost argument justifies EC when the decision stage is fixed, yet threshold-independent metrics can be legitimate development-time diagnostics when the operating point will be set later; the manuscript does not discuss this distinction. Please either provide the supporting derivation with explicit assumptions or weaken the blanket claims.","section":"§3.1"},{"comment":"The bootstrap confidence interval described in this section is valid for deployment performance only if the evaluation set is exchangeable with deployment data. Section 2.1 asks only that the evaluation data resemble the use-case data 'as closely as possible' and provides no procedure to verify representativeness, while Section 2.4 concedes that spurious correlations are hard to detect. If the evaluation set differs systematically from deployment, the bootstrap intervals will be too narrow. Add an explicit caveat and practical checks (e.g., report covariate overlap between evaluation and deployment, use multiple test sets, and test on temporally or geographically disjoint data).","section":"§4.1"},{"comment":"The paragraph beginning 'If we want to draw general conclusions...' suggests that the training data can be bootstrapped 'just like the test dataset in the section above.' This overstates the equivalence: bootstrapping the training set and retraining estimates sensitivity to resampling from the observed training data, not to drawing a new training sample from the underlying population or to a shifted deployment domain. In addition, correlated training samples (e.g., multiple recordings per speaker) require a clustered bootstrap, which the text does not mention. Please state these assumptions and limitations explicitly.","section":"§4.2"},{"comment":"NEC, NTE, and NCE are central to the paper's metric recommendations, yet they are introduced only by name with references to refs [4] and [5]; the defining formulas, the normalization denominators, and the claimed fixed-reference properties are not given in the manuscript. Since ref [4] is listed without a venue and ref [5] is an arXiv preprint, the reader cannot verify these claims without locating external documents. Provide at least the basic definitions and the normalization rationale in the text.","section":"§3.5"}],"minor_comments":[{"comment":"In the sentence 'system A turns out to be better than B by 5% ... would system B still be better?', the final question appears to refer to system A; please correct the pronoun or rephrase.","section":"§4.1"},{"comment":"The name 'Matthew correlation coefficient' should be 'Matthews correlation coefficient'.","section":"§3.1"},{"comment":"The note 'v1.0: This document is a work in progress' is appropriate for an arXiv preprint but should be removed or replaced with a version statement in the published version.","section":"Footnote 1"},{"comment":"The statement that development data 'can, in principle, be determined in any way you like' is in tension with the following paragraph warning that development data should follow the same criteria as evaluation data; please rephrase to avoid the apparent contradiction.","section":"§2.2"}],"recommendation":"major_revision","confidential_remarks":"This is an opinionated tutorial rather than a research paper with new empirical results. Its strongest metric recommendations lean heavily on the authors' own prior work (refs [4] and [5]), one of which is unpublished and one an arXiv preprint; an editor may wish to have those sources independently reviewed or ask the authors to make the relevant definitions and proofs self-contained. The exchangeability caveat for bootstrap intervals should also be verified as addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague—\n\nQuick take: this is a tutorial/position paper, not a research result. Read it if you want a clear checklist for evaluation protocols; skip it if you're looking for new methodology. That said, it's a good checklist.\n\nWhat it does well: the system-vs-method distinction is genuinely useful and has direct consequences for how you compute confidence intervals. The pitfalls list (augmentation-before-split, ignoring speaker identity, accuracy without a majority-class reference) is concrete and practical. The argument for expected cost over AUC/EER for categorical decisions is sound as far as it goes; the same goes for proper scoring rules when the decision threshold is unknown. The bootstrap code link is a nice touch.\n\nWhere it's soft: (1) The paper pushes the authors' own NEC/NTE/NCE metrics, based on refs [4,5], without any external validation or a worked example showing they change a conclusion. They're sensible normalizations, but in a review you'd want them labeled as the authors' proposal, not as settled consensus. (2) The confidence-interval section assumes the evaluation set is representative of deployment. Section 2.1 says the data 'should resemble' deployment, but there is no procedure to actually test that resemblance. If the test set differs systematically (different mics, dialects, time period), bootstrap intervals will be too narrow—exactly the false confidence the paper warns against. The paper gestures at this when recommending out-of-domain reporting, but the caveat belongs in Section 4.1. This is a limitation of the warrant, not an internal contradiction.\n\nThe central advice holds up. The paper is honest that it's a work in progress, and the overall guidance matches standard statistical practice.\n\nWho it's for: practitioners, students, and reviewers who want shared vocabulary for evaluation design. It deserves a serious referee—not because the claims are novel, but because it will likely be cited widely and the unsupported metric absolutes should be checked before it's disseminated. I'd recommend peer review with a request for minor revision: add a caveat about representativeness in Section 4.1, and dial back the 'strongly recommend against F-beta/MCC' phrasing unless the citations really support it.","headline":"A useful, opinionated tutorial on ML evaluation; the advice is mostly sound, but it overrelies on the authors' own metrics and glosses over test-set representativeness in the bootstrap section.","tokens_in":10371,"tokens_out":3078,"would_cite":false,"duration_ms":31418,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper argues that the evaluation protocol, fixed before training, determines whether ML conclusions generalize.","keywords":["machine learning evaluation","held-out evaluation data","metric selection","expected cost","proper scoring rules","bootstrapping","confidence intervals","spurious correlations"],"falsifier":"Take one evaluation set, compute bootstrap intervals for a metric, then measure the same system on several independent collections from the same deployment domain; if the measured values fall outside the intervals far more often than the declared confidence level, the assumption that the held-out set represents deployment is wrong.","tokens_in":9491,"feed_emoji":"🎯","tokens_out":5173,"duration_ms":50003,"temperature":0.7,"pith_summary":"This paper argues that the evaluation protocol—what data is used for testing, which metric is reported, and how variability is measured—determines whether conclusions from machine learning experiments generalize to real use. It claims that the protocol must be designed and fixed before experimentation starts: the evaluation set should be held out and mirror the deployment conditions, the metric should reflect the cost a user actually pays, and uncertainty should be reported as confidence intervals rather than point values. The authors care because most reported failures of ML systems, from overfitting test sets to metric-driven shortcuts, trace back to a poorly chosen evaluation protocol. The paper is a practical guide, with examples from speech processing and lists of common mistakes.","feed_headline":"ML conclusions hold only if evaluation is designed first","feed_subtitle":"How a held-out test set, a cost-aware metric, and bootstrap intervals keep results honest.","key_machinery":"The carrying mechanism is a three-part evaluation protocol. Data selection uses a held-out evaluation set that mirrors deployment, with a separate development set, and cross-validation or nested cross-validation when data is scarce. Metric selection centers on expected cost (EC), a cost-matrix-weighted generalization of error rate, and its normalized form (NEC) with a fixed reference of 1 for a naive system that ignores inputs; for posterior probabilities, proper scoring rules such as cross-entropy are recommended. Significance is handled by bootstrapping, which resamples the evaluation data (grouped by correlation-inducing conditions such as speaker identity) to produce confidence intervals for the metric, and can pool over random seeds and bootstrapped training sets when evaluating methods rather than frozen systems.","core_discovery":"On its own terms, the paper's central claim is that the reliability of an ML experiment is set before training begins, by three decisions: selecting evaluation data that resembles deployment and is never used for development; choosing a performance metric derived from the application's actual costs, with the normalized expected cost and proper scoring rules as defaults; and quantifying variability with bootstrapped confidence intervals that account for the evaluation set, random seeds, and training data. If these decisions are made carefully, conclusions about systems and methods will transfer to unseen data; if not, reported results can be optimistically biased or simply irrelevant to the intended use.","pith_inferences":["Editorial inference: if these practices became standard, single-number leaderboard rankings would be seen as incomplete; rankings would need cost matrices, data provenance, and confidence intervals to support claims.","Editorial inference: the silence-based spurious-correlation diagnostic for speech suggests an analogous test for other modalities, such as predicting a class from image background patches, as a cheap check before trusting a benchmark.","Editorial inference: the paper's downstream-task principle implies a concrete experiment: measure how often a proxy metric's system ranking differs from the downstream-task ranking across several applications; the disagreement rate would quantify the cost of informal metric choice."],"forward_implications":["If a held-out evaluation set mirrors deployment and is never touched during development, its reported metric should predict real-world performance.","Choosing the metric from application costs means systems are optimized for what users lose from mistakes, not for generic accuracy.","Confidence intervals from bootstrapping distinguish a real improvement from noise and prevent conclusions that flip with one seed or one test split.","Reporting normalized metrics like NTE or NEC makes results comparable across datasets with different class priors.","Slicing the evaluation set by demographic or acoustic subgroups turns an average number into a bias check."],"supporting_citations":[{"why":"Supplies the general framework for model evaluation and algorithm selection that this paper distills into practical rules.","marker":"[12]"},{"why":"Introduces the distinction between evaluating systems and evaluating methods, which shapes the two experimental scenarios.","marker":"[1]"},{"why":"Provides the bootstrap method that underlies the confidence-interval guidance.","marker":"[3]"},{"why":"Extends bootstrapping to correlated samples, which the paper adapts for conditions such as speaker identity.","marker":"[11]"},{"why":"Supports the expected-cost metric and the critique of F-beta and Matthews correlation coefficient.","marker":"[4]"},{"why":"Supplies the theory of proper scoring rules used to recommend cross-entropy and Brier score for posterior probabilities.","marker":"[7]"},{"why":"Documents spurious correlations in a speech dataset, motivating the paper's diagnostics and data-selection warnings.","marker":"[6]"},{"why":"Supports evaluating posterior probabilities with normalized cross-entropy and proper scoring rules.","marker":"[5]"}],"fun_headline_variants":["ML results only as trustworthy as evaluation design","Plan evaluation first: ML conclusions depend on it","Three evaluation decisions that make or break ML findings","Reliable ML: design data, metric, intervals before training","Evaluation design determines whether ML conclusions stick"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guidance assumes that a single held-out evaluation set is a random sample of deployment conditions, so bootstrapping it gives intervals that cover what the system will do in practice.","fun_headline_variants_meta":{"raw":{"variants":["ML results only as trustworthy as evaluation design","Plan evaluation first: ML conclusions depend on it","Three evaluation decisions that make or break ML findings","Reliable ML: design data, metric, intervals before training","Evaluation design determines whether ML conclusions stick"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000182,"raw_usage":{"total_tokens":1258,"prompt_tokens":840,"completion_tokens":418,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":456,"completion_tokens_details":{"reasoning_tokens":347}},"tokens_in":456,"tokens_out":418,"duration_ms":4973,"temperature":1.0,"reasoning_tokens":347,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:09:34.484973+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take one evaluation set, compute bootstrap intervals for a metric, then measure the same system on several independent collections from the same deployment domain; if the measured values fall outside the intervals far more often than the declared confidence level, the assumption that the held-out set represents deployment is wrong.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the distinction between evaluating systems and evaluating methods, which shapes the two experimental scenarios."},{"cited_title":"Efron and R","cited_arxiv_id":null,"evidence_quote":"Provides the bootstrap method that underlies the confidence-interval guidance."},{"cited_title":"Poh and S","cited_arxiv_id":null,"evidence_quote":"Extends bootstrapping to correlated samples, which the paper adapts for conditions such as speaker identity."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supports the expected-cost metric and the critique of F-beta and Matthews correlation coefficient."},{"cited_title":"Gneiting and A","cited_arxiv_id":null,"evidence_quote":"Supplies the theory of proper scoring rules used to recommend cross-entropy and Brier score for posterior probabilities."},{"cited_title":"Gauder, P","cited_arxiv_id":null,"evidence_quote":"Documents spurious correlations in a speech dataset, motivating the paper's diagnostics and data-selection warnings."},{"cited_title":"Ferrer and D","cited_arxiv_id":null,"evidence_quote":"Supports evaluating posterior probabilities with normalized cross-entropy and proper scoring rules."}],"review_version":1}