{"id":"9c2a9a8c-f791-4426-99ac-5c82bd1134e6","arxiv_id":"2504.18385","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A multiple imputation framework produces approximately Gaussian predictive distributions for classifier evaluation metrics under missing labels, with finite-sample bounds and empirical support on six datasets.","lead":"This paper proposes PEMI and PEMI-Gauss, multiple imputation algorithms that return a full predictive distribution for classification metrics like precision, recall, and ROC-AUC when some test labels are missing. The practical value is for model monitoring and evaluation when ground-truth labels arrive late or are absent for some data points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Theorem 3's KS bound is not proven as written: the proof applies marginal Berry–Esseen bounds to events with correlated random thresholds, and Theorem 2 relies on an unproven Taylor dominance.","rationale":"The paper's central claim is a finite-sample Gaussian approximation with explicit KS bounds. The reader's weakest assumption was calibration bias, but the reader's rationale also flagged an unproven step in Theorem 2. My read agrees with the latter and locates the more precise gap in Theorem 3: the proof treats random thresholds as fixed when applying Lemma 1, and Theorem 2's proof itself has an unproven Taylor assertion. This is not an internal inconsistency in the stated assumptions, and the empirical direction is plausible, so the result may be repairable. However, as written the proof of the headline theoretical guarantee is incomplete. This does not change the reader's CONDITIONAL verdict, but it makes the condition more specific: the finite-sample bound should be re-derived and verified before the claim is accepted.","tokens_in":41230,"tokens_out":12651,"duration_ms":123120,"concrete_test":"Re-derive the proof without random thresholds: for fixed t, eZ − t fW = (α − tβ) + Σ(a_i − t b_i)Y_i is itself a sum of independent Bernoullis, so Lemma 1 gives a KS bound with coefficients c_i(t)=a_i − t b_i. Track the t-dependence and take sup over t; if the resulting bound contains an extra factor sup_t max_i|c_i(t)|/min_i|c_i(t)| absent from Eq. (39), the published bound is not established. As a numerical check, simulate n=20 Bernoulli sums with p_i ∈ {0.1, 0.9}, a_i = ψ_i ∈ {0,1}, b_i = 1+ψ_i, estimate the true KS distance of eZ/fW from N(µ, σ²) with 10^6 Monte Carlo draws, and compare to the RHS of Theorem 3; a substantial violation would falsify the stated bound.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The finite-sample Gaussian guarantee for recall and F1 (the main theoretical contribution) rests on Theorem 3. In its proof, the first term is bounded by writing |P(eZ − t fW ≤ 0) − P(Z − tW ≤ 0)| ≤ |P(eZ ≤ t fW) − P(Z ≤ t fW)| + |P(W ≤ Z/t) − P(fW ≤ Z/t)| and then invoking Lemma 1 for marginal KS distances. Lemma 1 bounds sup_z |F_eZ(z) − F_Z(z)|, but here the threshold t fW is random and dependent on eZ; conditioning on fW changes the conditional law of eZ. The same problem affects the second term, where Z/t is random. The proof therefore does not establish Eqs. (38)–(39). The second term of Theorem 3 uses Theorem 2, whose proof relies on the unstated assertion that the Taylor remainder of f(t) is dominated by quadratic terms (Eq. 22, “we expect”) and drops an integration-by-parts boundary term of indefinite sign. Consequently neither leg of the claimed KS bound is supported. This matters because the abstract's finite-sample convergence bounds are precisely this theorem; without a repair, the central claim is unproven even if the approximation itself is empirically plausible.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies how to evaluate a binary classifier when some test-time labels are missing, with the goal of producing not just a point estimate but a predictive distribution over the value of performance metrics. Each masked label is modeled as an independent Bernoulli variable with parameter p_i, interpreted as the conditional probability that the label is 1 given features and missingness; this induces a distribution over confusion-matrix-based estimators of precision, recall, accuracy, and F1, and over a ratio estimator for ROC-AUC. Two algorithms are proposed: PEMI, which samples the missing labels to build an empirical CDF, and PEMI-Gauss, which uses a closed-form Gaussian approximation. The theoretical contributions are Theorem 2 (a finite-sample Kolmogorov-Smirnov bound for the ratio of correlated Gaussians), Theorem 3 (a finite-sample KS bound showing that the ratio of sums of independent Bernoullis is approximately Gaussian, covering recall and F1), and Theorem 4 (the same conclusion when the Bernoulli parameters are themselves random with the correct means). Experiments on six datasets under MCAR and MNAR masking, evaluated through PIT-based diagnostics with confidence intervals, show that calibrated PEMI/PEMI-Gauss generally improve over the observed-label bootstrap, with documented exceptions for precision and recall at low MNAR positive fractions.","tokens_in":41501,"tokens_out":31739,"duration_ms":297916,"significance":"If the proofs were repaired, the main theoretical contributions — explicit finite-sample KS bounds for ratios of sums of Bernoullis (Theorem 3) built on a ratio-of-Gaussians bound (Theorem 2), and the robustness observation in Theorem 4 — would be a useful and, to my knowledge, novel contribution to metric estimation under missing labels. The problem is well motivated, and the paper engages appropriately with the missing-data, semi-supervised, and calibration literatures. Strengths I want to acknowledge explicitly: the experimental protocol is appropriate (six datasets, three missingness rates, MCAR and MNAR mechanisms, PIT-based evaluation of both location and shape, bootstrapped 90% confidence intervals); the MNAR failures for precision and recall are reported openly rather than hidden; and Theorem 4's Beta-Bernoulli marginalization argument is correct, showing that mean-preserving noise in the p_i does not change the marginal law of the metrics. The empirical message that calibrated multiple imputation improves over ignoring missing labels is supported by the tables.","major_comments":[{"comment":"The proof of Theorem 3 does not establish the claimed bound because it applies Lemma 1 to events with random, dependent thresholds. Eqs. (33)–(37) reduce the first term to |P(ẽZ ≤ tW̃) − P(Z ≤ tW̃)| + |P(W̃ ≤ Z/t) − P(W ≤ Z/t)|, and Lemma 1 is then invoked to bound these quantities. Lemma 1, however, bounds sup_z |F_{ẽZ}(z) − F_Z(z)| for a sum of independent Bernoullis; the threshold tW̃ is random and dependent on ẽZ through the shared Y_i, and conditional on W̃ the law of ẽZ is no longer that of an independent-Bernoulli sum, so the marginal KS bound does not transfer to the unconditional event. The same dependence problem affects the second term with the random threshold Z/t. Consequently Eqs. (38)–(39) do not follow, and the finite-sample convergence bound advertised in the abstract is not proven. The proof also invokes Theorem 2 without verifying its two '≫' conditions, so Theorem 3 should carry those hypotheses or quantify the regime; a repair would need a multivariate Berry-Esseen or equivalent argument for the pair (ẽZ, W̃).","section":"Appendix B.5.1, Eqs. (30)–(39)"},{"comment":"The proof of Theorem 2 contains two invalid steps. First, the integration-by-parts in Eq. (13) is a calculus error: with u = 1/z and dv = z exp(−z²/2) dz, the remainder is −∫ exp(−z²/2)/z² dz, not the displayed +∫ exp(−z²/2)/z dz, and in any event the inequality |A + B| ≤ |A| used to discard the remainder is false in general, so the step '≤ |boundary term|' is unjustified. Second, the bound on |f(t)| in Eqs. (23)–(24) rests on the assertion in Eq. (22) that the cubic-and-higher Taylor terms of f are dominated by the quadratic terms; the proof itself labels this as 'we expect', and no explicit neighbourhood of t = µ or coefficient bounds are given, so the uniform-in-t bound is not established. In addition, the proof replaces G(t) = P(Z/W ≤ t) by P(Z − tW ≤ 0) and never quantifies the discarded term P(W ≤ 0), which is bounded by a Gaussian tail under condition 1 but does not appear in the displayed bound. The '≫ 1' conditions are asymptotic in character and do not appear in the final bound, so the theorem does not identify a finite-sample regime in which the claimed inequality provably holds.","section":"Appendix B.4.1, Eqs. (13) and (22)"},{"comment":"The inequality (1 + a∗)/(2a∗) + (1 + b∗)/(2b∗) ≤ 1 + b∗/a∗ used to pass from Eqs. (38) to (39) fails for general positive a∗ ≤ b∗; for instance a∗ = b∗ = 0.5 gives 3 ≤ 2. The displayed first term in Theorem 3's bound is therefore not a consequence of the preceding line as stated for arbitrary real coefficients a_i, b_i. For the intended applications (precision, recall, F1) the coefficients are 0/1-valued and a∗ = b∗ = 1, for which the inequality holds, so the defect is in the generality of the theorem statement rather than in the specific use cases; the statement should be restricted accordingly or the inequality proved under the stated hypotheses.","section":"Theorem 3, Eq. (39)"},{"comment":"The abstract's claim that the predictive distribution's location and shape are 'generally correct, even in the MNAR regime' is stronger than the reported evidence. Section 6 states that for the Precision metric with η ∈ {0.1, 0.2}, the uncalibrated MaxEnt choices outperform the calibrated PEMI-Gauss, and that for Recall the bootstrap distribution beats PEMI-Gauss (Figure 4 and the surrounding text). The offered explanation — that calibrated probabilities are overestimated under MNAR — is plausible but is not directly tested, since no calibration error or predicted-versus-empirical positive-rate diagnostic is reported per missingness regime. The abstract and introduction should be qualified to reflect these documented exceptions, or the calibration-bias hypothesis should be verified with additional diagnostics.","section":"Section 6, Fig. 4; Abstract"}],"minor_comments":[{"comment":"The displayed formula for E[|Ti|³] uses a_i³ rather than |a_i|³; for negative a_i the right-hand side is negative, which is impossible for a third absolute moment. The subsequent inequalities still go through, so this is a presentation issue.","section":"Appendix B.3.1, Eq. (4)"},{"comment":"The statement does not say explicitly that the Pi are independent across i; this independence (together with conditional independence of Yi given Pi) is needed for the conclusion that the Yi are mutually independent Bernoulli(pi). The proof also contains a typo, 'Xi ∼ B(pi)', where Yi is meant.","section":"Theorem 4"},{"comment":"The introduction states that finite-sample convergence bounds are proven for 'precision, recall and ROC-AUC amongst others', but Section 5 ('Gaussian Approximation for ROC-AUC') explicitly excludes ROC-AUC from Theorem 3 and delegates it to future work; the introduction should state that the proven guarantees cover CM-based metrics only.","section":"Section 1 and Section 5"},{"comment":"For ROC-AUC with n > 120, the closed-form covariance computation is O(n⁴) and the paper instead uses PEMI with B = 10000 to estimate µ and σ²; this empirical fallback should be mentioned wherever PEMI-Gauss is described as a closed-form procedure (Algorithm 2, Section 6).","section":"Remark 14"},{"comment":"The same letter K is used for the set of known labels and for the set of masked labels ('K = {1, 2, ..., k} and the remaining indices as masked labels K = {k+1, ..., n}'); distinct notations (e.g., K and K̄) would avoid ambiguity.","section":"Section 2"},{"comment":"The caption says the widths are computed 'on the same experimental setup as in Section 6', but it is not specified how the widths are aggregated over the six datasets and 20 replications; stating the aggregation would make the figure interpretable.","section":"Figure 2"},{"comment":"The condition 0 < α ≤ β is not always satisfied in the applications (α = 0 whenever the known subset contributes zero to the numerator, e.g., zero known true positives), so the hypothesis should be 0 ≤ α ≤ β or the degenerate cases should be addressed.","section":"Theorem 3 hypotheses"},{"comment":"The right-hand sides of the bounds are frequently larger than 1 for the parameter values appearing in the experiments; a sentence indicating when the bounds are non-trivial (e.g., the scaling in Corollary 11) would help the reader judge the strength of the finite-sample claims.","section":"Theorems 2–3, bounds"}],"recommendation":"major_revision","confidential_remarks":"Both the accompanying reader's assessment and the skeptic's specific objections land on my reading of the manuscript. The proof of Theorem 3 misapplies a marginal Berry-Esseen bound to events with dependent random thresholds, and the proof of Theorem 2 contains a concrete integration-by-parts error as well as an openly flagged ('we expect') unproven Taylor dominance assertion; because the abstract's central advertised contribution is precisely these finite-sample bounds, these are load-bearing defects in the current version. I do not see a reason the results cannot be repaired within the scope of a revision — a multivariate Berry-Esseen route or appropriately weakened/restricted statements are plausible — and the empirical portion is solid and honest. The paper's priority claims ('first rigorous bounds', 'first to tackle missing labels and model evaluation') are strong; I could not verify them from the cited literature and suggest the editors have them checked. The introduction's implication that the convergence bounds cover ROC-AUC conflicts with the explicit exclusion in Section 5, and this should be resolved. No integrity concerns: the 'we expect' passage is openly labeled as such, and the robustness theorem's limited scope (correct-mean noise only) is stated honestly."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful practical idea—use multiple imputation to turn missing test labels into a predictive distribution over evaluation metrics—and the experiments give reasonable first evidence that it works. But the main advertised theory, the finite-sample Gaussian bounds, is not proven as written. The stress-test note is right: Theorem 3 applies marginal Kolmogorov–Smirnov bounds to events with random thresholds, and Theorem 2 leans on an unproven “we expect” term. These aren’t cosmetic problems; they are the difference between a guarantee and a heuristic.\n\nWhat is new: the missing-labels-at-evaluation framing is a real gap in the literature, and the paper is the first to attack it directly for classifier metrics. The PEMI and PEMI-Gauss algorithms are simple and sensible, the PIT-based evaluation is appropriate, and the experiments across six datasets and both MCAR and MNAR are thorough enough to show the method can work when the imputation probabilities are reasonable. Theorem 4’s robustness result is a nice touch.\n\nThe soft spots are proportionately serious. The Theorem 3 gap: the authors bound |P(eZ ≤ t fW) − P(Z ≤ t fW)| by the sup-norm KS distance between eZ and Z, but t fW is random and dependent on eZ. The sup-norm bound doesn’t control that difference; the same issue hits the second term. Theorem 2’s proof assumes without argument that quadratic terms dominate the Taylor remainder of f(t) and drops an integration-by-parts boundary term without discussion. So the finite-sample bound in the abstract is unsupported. The ROC-AUC extension is explicitly heuristic, but the abstract doesn’t make that clear. The reliance on well-calibrated p_i is real: their own MNAR low-η experiments show calibrated probabilities can be worse than MaxEnt for precision and recall, and Theorem 4 only covers unbiased calibration error. No code or data is provided, which makes rechecking harder.\n\nWho this paper is for: practitioners monitoring production classifiers and researchers working on evaluation under label scarcity will get value from the empirical method. The theory, as it stands, is not a reliable citeable result.\n\nRecommendation: send to serious peer review, but expect major revision—fix the proofs or clearly downgrade the claims to heuristics, release code and data, and make the abstract match what is actually proven.","headline":"Useful practical idea with honest experiments, but the advertised finite-sample guarantees are unproven as written; needs major revision, not desk rejection.","tokens_in":41970,"tokens_out":3159,"would_cite":false,"duration_ms":31206,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D10","62E17","68T05"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that classifier metrics like precision and recall remain reliably estimable when up to 30% of test labels are missing, via multiple imputation with calibrated Bernoulli probabilities.","keywords":["missing labels","model evaluation","multiple imputation","predictive distribution","Gaussian approximation","missing not at random","confusion matrix","classifier metrics"],"falsifier":"Reveal the true labels on an MNAR test set with an extreme positive fraction ($\\eta = 0.1$), build the PEMI-Gauss predictive distribution using both a well-calibrated and a deliberately mis-calibrated imputation model, and apply a probability integral transform test: if the central claim is right, the correctly calibrated PIT values should be approximately uniform and within the Theorem 3 KS bound, while the biased model should deviate systematically, as the paper's own Figure 4 already suggests for precision at small $\\eta$.","tokens_in":41035,"feed_emoji":"📊","tokens_out":7089,"duration_ms":70911,"temperature":0.7,"pith_summary":"Most work on missing data in machine learning has focused on missing features during training; this paper targets the quiet failure mode of missing ground-truth labels at evaluation time. Ignoring the unlabelled examples, the standard fix, is biased whenever the missingness is not completely random. The paper's proposal, Performance Estimation by Multiple Imputation (PEMI), treats each missing label as a Bernoulli draw with a calibrated probability, and its analytic sibling PEMI-Gauss reads off a predictive distribution for precision, recall, accuracy, F1-score, and ROC-AUC. The central theoretical claim is that, under conditions, this distribution is close to Gaussian in Kolmogorov-Smirnov distance with explicit finite-sample bounds, and that the approximation survives a realistic noise model on the imputation probabilities. This matters because model monitoring, fraud detection, and recommendation systems routinely evaluate models before all labels have arrived.","feed_headline":"Missing labels no longer block trustworthy model evaluation","feed_subtitle":"Imputation yields a predictive distribution for precision, recall, and AUC, with finite-sample Gaussian guarantees.","key_machinery":"The load-bearing object is the confusion matrix expressed as random variables: for example, $\\widehat{\\mathrm{TP}} = \\sum_{i\\in K} y_i \\psi_i + \\sum_{i\\in \\bar K} Y_i \\psi_i$ with $Y_i \\sim \\mathrm{Bern}(p_i)$ for each missing label, and similarly for FN, FP, and TN. Every confusion-matrix metric is then a weighted sum of Bernoullis (accuracy, precision) or a ratio of correlated sums (recall, F1). PEMI-Gauss approximates each metric by a Gaussian using Lemma 1, a Berry-Esseen-type bound for weighted Bernoulli sums, and Theorem 2, a KS bound for ratios of correlated Gaussians, while a scaling-binning calibrator supplies the $p_i$ and MaxEnt choices $p_i=1/2$ or $p_i=N_+/N$ serve as uncalibrated alternatives. The same structure produces the mean and variance used for ROC-AUC.","core_discovery":"On the paper's own terms, the central discovery is that evaluating a binary classifier with missing labels can be reduced to studying sums of independent Bernoulli random variables and ratios of such sums, and that these objects are provably near-Gaussian. Theorem 3 gives a finite-sample bound on the Kolmogorov-Smirnov distance between the distribution of a ratio of correlated sums of weighted Bernoullis and a matched normal distribution, covering recall, F1-score, precision, and accuracy. Theorem 4 extends the guarantee to the realistic case where each Bernoulli parameter is itself random with the correct mean, so that a well-calibrated imputation model yields predictive distributions of the right location and shape. The same machinery supplies means and variances for ROC-AUC, which the paper treats through its rank-based probabilistic query. Empirically, on six datasets with up to 30% of labels missing in both MCAR and MNAR settings, the calibrated PEMI and PEMI-Gauss distributions improve on the bootstrap baseline built only from the labelled examples.","pith_inferences":["(Editorial inference) The same ratio-of-Bernoulli machinery should extend to multi-class metrics via a multinoulli/Dirichlet construction, the direction the paper lists only as future work; a categorical analogue of Theorem 4 would likely follow from the same mixture argument.","(Editorial inference) A testable extension the paper does not pursue is using PEMI-Gauss as a stopping rule for label collection: keep querying labels until the predictive credible interval for precision or recall drops below a tolerance, with the PIT framework as the natural validation tool.","(Editorial inference) The MNAR results hint that at extreme class imbalance, a deliberately uninformative prior can beat a miscalibrated model; real deployments should therefore monitor calibration on the missing-not-at-random population rather than only on observed labels."],"forward_implications":["A decision-maker who must evaluate a deployed classifier before all labels arrive can quote a full predictive distribution instead of a possibly biased point estimate, with the distribution's width reflecting genuine ignorance about the missing labels.","The Gaussian approximation means the method costs almost nothing at inference time: no repeated sampling of bitstrings is needed, only the mean and variance of each metric.","The finite-sample bounds in Theorem 3 cover the common confusion-matrix metrics, so the guarantee is a concrete KS-distance statement at finite $n$, not just an asymptotic observation.","In the MNAR experiments, calibrated PEMI-Gauss outperforms the bootstrap baseline for most metrics, with recall at small $\\eta$ the notable exception, showing both the method's value and its limits.","Because Theorem 4 allows the Bernoulli parameters to be random with correct means, the theory applies to any well-calibrated imputation model, not only the specific calibrator used in the experiments."],"supporting_citations":[{"why":"Supplies the MCAR/MAR/MNAR taxonomy that defines the problem setting.","marker":"Rubin, 1976"},{"why":"Establishes that ignoring missing values yields biased estimates unless missingness is completely at random, the baseline the paper aims to beat.","marker":"Williams, 2015"},{"why":"Berry-Esseen theorem with the universal constant used in Lemma 1 and Theorem 3.","marker":"Esseen, 1956; Shevtsova, 2010"},{"why":"Shows ratios of normal variables are poorly behaved, motivating the KS bounds for ratios in Theorem 2.","marker":"Marsaglia, 1965"},{"why":"Scaling-binning calibrator that produces the Bernoulli parameter estimates used in the experiments.","marker":"Kumar et al., 2019"},{"why":"Maximum entropy principle behind the uncalibrated choices $p_i=1/2$ and $p_i=N_+/N$.","marker":"Jaynes, 1957"},{"why":"Probability integral transform methodology used to score the predictive distributions in the experiments.","marker":"Diebold et al., 1998"},{"why":"Defines ROC-AUC as a ranking probabilistic query, the basis for the ROC-AUC estimator.","marker":"Bradley, 1997"}],"fun_headline_variants":["Impute missing labels for reliable precision, recall, AUC","Missing labels? Robust metric evaluation via imputation","Gaussian bounds for classifier metrics with missing labels","Reliable model evaluation even when labels vanish","Predict metric distributions despite missing labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the imputation probabilities fed into the Bernoulli draws have the correct conditional means; a systematically biased calibrator shifts the whole predictive distribution, and the paper's robustness theorem does not cover that bias.","fun_headline_variants_meta":{"raw":{"variants":["Impute missing labels for reliable precision, recall, AUC","Missing labels? Robust metric evaluation via imputation","Gaussian bounds for classifier metrics with missing labels","Reliable model evaluation even when labels vanish","Predict metric distributions despite missing labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000173,"raw_usage":{"total_tokens":1241,"prompt_tokens":867,"completion_tokens":374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":483,"completion_tokens_details":{"reasoning_tokens":305}},"tokens_in":483,"tokens_out":374,"duration_ms":4101,"temperature":1.0,"reasoning_tokens":305,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:17:29.868892+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reveal the true labels on an MNAR test set with an extreme positive fraction ($\\eta = 0.1$), build the PEMI-Gauss predictive distribution using both a well-calibrated and a deliberately mis-calibrated imputation model, and apply a probability integral transform test: if the central claim is right, the correctly calibrated PIT values should be approximately uniform and within the Theorem 3 KS bound, while the biased model should deviate systematically, as the paper's own Figure 4 already suggests for precision at small $\\eta$.","supporting_citations":[],"review_version":1}