{"id":"bfbe6a93-58da-49e6-9ccb-accf662a91cd","arxiv_id":"2507.11297","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The energy-I-Score ranks imputation methods by comparing observed values with repeated imputations using the energy score, and is claimed to be proper under a new condition, CIMAR_j.","lead":"This statistics paper proposes the energy-I-Score, a method for choosing among imputation methods when the complete data are unavailable. It introduces a new missingness assumption and demonstrates the score in simulations and a 401(k) downstream analysis.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 3.1 is false as stated: Assumption 3.1 gives only one CIMAR_j, but the score averages over all j∈S, and the Appendix C proof implicitly requires CIMAR_j for every averaged component.","rationale":"The reader's primary concern is the right one and is load-bearing. The score averages S_j over every j∈S, but Assumption 3.1 guarantees CIMAR_j for only one j; Appendix C's proof needs the property for every averaged component. I constructed a concrete PMM-MAR setting satisfying Assumption 3.1 where a non-CIMAR component can be improved by an H≠P*, so the proposition is actually false as written. The fix is to strengthen Assumption 3.1 to all j∈S (or to redefine S as the set of CIMAR_j variables), in which case the energy-score argument goes through. I do not regard the Mj=0/Mj=1 discrepancy in (C.1) as a separate load-bearing problem; the intended conditioning is Mj=1, and the propriety inequality is correct with that substitution. Since the reader's verdict already conditions acceptance on this repair, I leave the verdict unchanged.","tokens_in":28102,"tokens_out":11968,"duration_ms":134675,"concrete_test":"Instantiate the three-pattern counterexample above (e.g., Gaussian copula for X1,X2, n large), compute the empirical energy-I-Score for H and P* with N=50 or larger. If bS_NA(H)>bS_NA(P*) reproducibly, the stated Proposition 3.1 is false; then re-run the proof with CIMAR_j required for all j∈S to see the corrected theorem. An analytic version: evaluate S_2^NA(H)−S_2^NA(P*) by the energy-score propriety identity; it is strictly positive whenever P*_{X2|X3,M2=1}≠P*_{X2|X3,M2=0}.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Proposition 3.1 asserts that, under Assumption 3.1 (one index j satisfies CIMAR_j) and PMM-MAR, the average S_NA over all j∈S is proper. Appendix C proves, at most, that the single CIMAR_j component is proper; the summation step needs CIMAR_j for every j∈S. This is not a minor gap. Take patterns m1=(0,0,0), m2=(0,1,0), m3=(1,0,0), with missingness probabilities P(M=m1|x)=x1/3, P(M=m2|x)=2/3−x1/3, P(M=m3)=1/3, and draw (X1,X2) dependent as in Appendix B.1. This satisfies PMM-MAR; CIMAR_1 holds because M1 is independent of X1 given X3, while CIMAR_2 fails. Let H∈H_P impute X1 in pattern m3 exactly as P* does, but impute X2 in pattern m2 from P*_{X2|X3,M2=0} instead of the true P*_{X2|X3,M2=1}. Compatibility with observed data is preserved because X2 is missing in m2. Then S_1^NA(H)=S_1^NA(P*), while S_2^NA(H)>S_2^NA(P*) by strict propriety of the energy score (H puts mass on the test-point distribution). Hence S_NA(H)>S_NA(P*), contradicting Proposition 3.1. The theorem becomes true if Assumption 3.1 is replaced by 'CIMAR_j holds for every j∈S', or if S is restricted to such indices; the manuscript states neither.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a new score, the energy-I-Score, for ranking imputation methods when the complete data are unavailable. The score is constructed variable by variable: for each variable that can be missing, it compares the candidate imputation distribution conditional on the always-observed variables (when the variable is missing) with observed test points of that variable, using the energy score. The authors identify a missingness condition, CIMAR_j, and state in Proposition 3.1 that under this condition for at least one j and under PMM-MAR the averaged score over all eligible variables is a proper I-Score. They provide an estimation algorithm using multiple imputation, compare the score with the DR-I-Score, and report simulations plus a downstream double-machine-learning application.","tokens_in":28484,"tokens_out":8278,"duration_ms":91472,"significance":"If the main theoretical claim were correct, this would be a practically valuable contribution: a simple, classifier-free, distribution-oriented tool for ranking imputations, directly addressing the known shortcomings of RMSE-based benchmarking and of the classifier-based DR-I-Score. The paper is clearly written, the simulations are informative, and the code is made available. The energy score component is standard and the estimation procedure is intuitive. However, the central theoretical result, Proposition 3.1, is false as stated; the gap is load-bearing for the paper's main claim. The simulations and downstream experiment appear consistent with a corrected version of the theorem, so the issue is fixable, but the manuscript in its current form overstates what is proved.","major_comments":[{"comment":"Proposition 3.1 is false as stated. Assumption 3.1 asserts only the existence of at least one j for which CIMAR_j holds, while S_NA averages over all j in S. The proof in Appendix C establishes, at most, propriety of the single component S_j^NA for a j satisfying CIMAR_j; the final averaging step then requires CIMAR_j for every j in S. In addition, the proof applies the energy-score propriety inequality to H_{Xj|XOj,Mj=0} in Eq. (C.1), whereas the score in Eq. (3.2) is defined with H_{Xj|XOj,Mj=1}. This is not a mere typo: for H in H_P, compatibility forces H_{Xj|XOj,Mj=0} = P*_{Xj|XOj,Mj=0}, so the inequality for the Mj=0 version is vacuous and says nothing about the imputation actually scored. The dependent uniform example in Section B.1 (with patterns m1=(0,0,0), m2=(0,1,0), m3=(1,0,0) and dependence between X1 and X2) satisfies PMM-MAR and Assumption 3.1 (since CIMAR_1 holds) while CIMAR_2 fails. For that example, an H that imputes X1 as P* does and imputes X2 in pattern m2 from P*_{X2|X3,M2=0} instead of P*_{X2|X3,M2=1} is compatible with the observed distribution, has S_1^NA(H)=S_1^NA(P*), and, by strict propriety of the energy score, has S_2^NA(H)>S_2^NA(P*), contradicting the proposition. The claim becomes true if Assumption 3.1 is replaced by the condition that CIMAR_j holds for every j in S, or if the set S is restricted to indices satisfying CIMAR_j; neither statement appears in the manuscript, and the practical scope of the method changes substantially under either fix.","section":"Section 3.1, Proposition 3.1; Appendix C"}],"minor_comments":[{"comment":"The text describing the generated samples contains a typographical error: it writes 'X~(l)_i,j, ..., X~(l)_N' where the last subscript should be 'i,j' rather than 'N'. This is a presentation issue but could confuse readers implementing the estimator.","section":"Section 3.2, Eq. (3.3)"},{"comment":"The estimation procedure is described as approximating H_{Xj|XOj,Mj=1} by imputing a concatenated dataset that includes the candidate method's own previously imputed values from patterns in Lc_j. This is a reasonable heuristic, but the paper does not discuss how the approximation error depends on the imputation method's internal iterative structure. A brief discussion or reference to the multiple-imputation literature would help.","section":"Section 3.2 and Algorithm 1"},{"comment":"The reference list contains two entries for Stekhoven and Buhlmann (2011), one written 'Stekhoven, D. J. and Buhlmann, P.' and another 'Stekhoven, D. J. and B\\\"uhlmann, P.'; these should be consolidated into a single entry with consistent accent formatting.","section":"References"},{"comment":"The figure caption and text refer to 'runifsq' as drawing from p*(x1|x2,x3,M=m1)=2x1, but the formula in the text uses an indicator notation that is somewhat terse; adding an explicit density expression would improve reproducibility.","section":"Section 5.1 and Figure 2"}],"recommendation":"major_revision","confidential_remarks":"The central theorem as stated is false, but the fix is straightforward: either strengthen the assumption to CIMAR_j for every scored variable or restrict the score to variables satisfying CIMAR_j. Notably, the paper's own Section B.1 contains the dependent uniform example that exposes the gap, so the authors were aware that Assumption 3.1 can fail for some variables in S; this makes the oversight in Proposition 3.1 particularly important to correct. The simulations and the DML application appear compatible with the corrected theorem, so a revised version could be suitable for publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: the energy-I-Score is a genuinely useful addition to the imputation-ranking toolbox, and the paper does the right thing by moving away from RMSE-style masked evaluation. But the central propriety claim, Proposition 3.1, is false as stated. The proof in Appendix C shows at most that a single CIMAR_j component is proper; the final averaging step over all j in S needs CIMAR_j to hold for every j in S. Assumption 3.1 only guarantees one. The stress-test counterexample (uniform marginals with dependence, patterns m1,m2,m3) checks out: CIMAR_1 holds, CIMAR_2 fails, and an H that imputes X1 correctly but imputes X2 in pattern m2 from the M2=0 conditional beats P* on the averaged score. So the theorem needs either a stronger assumption or a restriction of S. Also, the proof writes H_{Xj|XOj,Mj=0} where the score definition (3.2) uses H_{Xj|XOj,Mj=1}; that is another repairable slip.\n\nThat said, the paper earns credit elsewhere. The CIMAR_j condition is new and is a sensible weakening of RMAR/EMAR for variable-wise scoring. The demonstration that naive masking leads to MNAR and can misrank imputations is well argued, and the correction of the DR-I-Score's propriety to EMAR is a useful clarification. The simulations are honest: they include a case where Assumption 3.1 fails (dependent uniform) and show the score still works empirically, and they acknowledge that the score is not strictly proper and that rankings of suboptimal methods are metric-dependent. Code is available.\n\nThe practical estimation in Algorithm 1 wanders from the theory: when Oj is empty it substitutes a single best variable, and the final score uses weights wj that do not appear in the population definition. That may be fine in practice, but it means the implemented score is not exactly the object being proved proper.\n\nWho is this for? Applied statisticians who need to pick an imputation method for a dataset without complete data, and methodologists working on missing data scoring. The paper should go to peer review, but only after the authors tighten Assumption 3.1 (e.g., 'CIMAR_j for all j∈S'), fix the proof's conditioning, and either align Algorithm 1 with the theory or state the divergence as an approximation. With those repairs, it is a solid contribution.","headline":"Good idea, broken theorem as stated: the energy-I-Score needs CIMAR_j for every averaged component, not just one, and the proof swaps Mj=0/Mj=1.","tokens_in":28941,"tokens_out":4834,"would_cite":false,"duration_ms":52677,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H12","62G05"],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper proves that an energy-score-based imputation score ranks the true imputation distribution highest among all imputations compatible with the observed data, without needing the complete data, under its missingness assumption.","keywords":["energy score","proper scoring rules","I-Score","imputation ranking","missing at random","pattern-mixture models","multiple imputation","conditional independence"],"falsifier":"Take the dependent-uniform example of Appendix B.1, where $X_1$ and $X_2$ are dependent and missingness depends on $X_1$, so CIMAR$_2$ fails; compute the population energy-I-Score. If any imputation distribution other than $P^*$ achieves a strictly higher averaged score, the theorem as stated is falsified because the averaged score includes a variable whose CIMAR condition fails.","tokens_in":27914,"feed_emoji":"🎯","tokens_out":9920,"duration_ms":108617,"temperature":0.7,"pith_summary":"Choosing an imputation method usually means benchmark studies that artificially mask more data and compare imputed values to the true ones with RMSE; the paper argues this is generally invalid under MAR and can systematically misrank methods. Its aim is to rank imputations for a given incomplete data set without access to the complete data, by scoring whether each method reproduces the conditional distribution of a missing variable given the variables that are always observed. To that end it builds the energy-I-Score from the energy distance and proves it is a proper I-Score under pattern-mixture MAR (PMM-MAR) plus a new condition, CIMAR$_j$, which says that for the scored variable $j$, the distribution of $X_j$ given the always-observed variables is the same in observed and missing patterns. In simulations and a downstream 401(k) analysis the score's ordering tracks the ordering one would get with the full data, while the earlier DR-I-Score is shown to fail in the paper's leading MAR example.","feed_headline":"New energy score ranks imputations without complete data","feed_subtitle":"It masks each variable, imputes it many times, and scores how well the draws match observed values.","key_machinery":"The central object is the variable-wise energy score $S^j_{\\mathrm{NA}}(H,P)$, built from the energy score $\\mathrm{es}(P,y)=\\frac{1}{2}\\mathbb{E}_{X,X'\\sim P}\\lVert X-X'\\rVert-\\mathbb{E}_{X\\sim P}\\lVert X-y\\rVert$, a strictly proper scoring rule that favors the distribution actually generating the test points. The construction masks the observed values of $X_j$, imputes them $N$ times from the candidate distribution $H_{X_j\\mid X_{O_j},M_j=1}$, and compares these draws against the real observed values from $P^*_{X_j\\mid X_{O_j},M_j=0}$. Propriety is carried by Condition CIMAR$_j$, which equates these two conditional distributions, turning the missing-data ranking problem into a distributional prediction problem on always-observed variables. Estimation uses the empirical energy-score formula of Equation (3.3) after $N$ multiple-imputation runs, with the per-variable scores averaged over the variables that can be missing.","core_discovery":"The central claim is Proposition 3.1: under PMM-MAR and Assumption 3.1, the energy-I-Score $S_{\\mathrm{NA}}$ is a proper I-Score, meaning that the true imputation distribution $P^*$ receives the highest expected score among all imputations compatible with the observed data. For each partly observed variable $j$, the score takes observed units where $X_j$ is present, artificially masks $X_j$, imputes it $N$ times from the candidate method given the always-observed variables $X_{O_j}$, and compares those draws to the real observed values with the energy score; the per-variable scores are averaged over the variables that can be missing. The propriety argument runs through Condition CIMAR$_j$, which states that $P^*_{X_j \\mid X_{O_j}, M_j=0} = P^*_{X_j \\mid X_{O_j}}$, so a correct imputation in missing patterns is indistinguishable, for scoring purposes, from the distribution that produced the observed values. The paper also shows the earlier classifier-based DR-I-Score requires the stronger EMAR condition and fails in the paper's leading MAR example, whereas the new score is proper there.","pith_inferences":["The proof of Proposition 3.1 as written averages $S^j_{\\mathrm{NA}}$ over all variables in $S$, but Assumption 3.1 only guarantees CIMAR$_j$ for one variable; a correct theorem needs the condition for every scored variable, or the average can be pulled down by a variable whose component score is improper.","A natural extension is to replace the univariate energy score with a multivariate energy distance over blocks of variables; the paper's own counterexample shows the univariate score cannot detect whether the joint dependence between two missing variables is preserved, so a multivariate version would restore that sensitivity at the price of sample efficiency.","When no variable is always observed, the implementation's fallback rule (use the variable most often jointly observed) lies outside the propriety theorem; applying the score in such designs is best treated as heuristic unless the condition is checked.","A testable practical prediction: restricting the average to variables that satisfy CIMAR$_j$ should reproduce the full-data ordering more faithfully than the unrestricted average in data sets where the condition holds for some variables but not all."],"forward_implications":["Practitioners can select imputation methods for their own incomplete data by distributional fidelity, without needing a held-out complete sample or an auxiliary classifier.","The score deliberately ranks down methods that only impute conditional means, since near-identical draws across the $N$ replications score poorly on the energy score's spread term.","It extends proper I-Scores to settings like the paper's Example 1, where PMM-MAR holds but EMAR fails and the DR-I-Score is not proper.","Estimation is practical with roughly 20 to 50 imputation draws, and for high-dimensional data the score can be computed on a subset of variables with the most missingness.","In the downstream 401(k) study, the score's ordering agrees with which imputation brings double-machine-learning estimates and standard errors closest to the complete-data analysis."],"supporting_citations":[{"why":"Defines proper I-Scores and the classifier-based DR-I-Score that the new score extends and compares against.","marker":"Näf et al. (2023)"},{"why":"Supplies the strictly proper energy score that is the building block of the new score.","marker":"Gneiting and Raftery (2007)"},{"why":"Provides the PMM-MAR formulation, Proposition 2.4 about conditional distributions, and Example 4 used as the leading test case.","marker":"Näf et al. (2025)"},{"why":"Gives the sample-based energy score estimator used in the score's estimation algorithm.","marker":"Gneiting et al. (2008)"},{"why":"Introduces the pattern-mixture model factorization on which the paper's missingness definitions rest.","marker":"Little (1993)"},{"why":"Supplies the notation and discussion of MAR variants used to state the assumptions.","marker":"Seaman et al. (2013)"}],"fun_headline_variants":["Energy score ranks imputations without complete data","Rank imputations fairly with no complete data","Proper energy score for imputation ranking","Energy I-Score: rank imputations without full data","New score picks best imputation, no need for original"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The proof needs Condition CIMAR$_j$ to hold for every variable $j$ that contributes to the averaged score, whereas Assumption 3.1 only guarantees one such variable; if any scored variable violates it, that component can favor a wrong imputation and break propriety.","fun_headline_variants_meta":{"raw":{"variants":["Energy score ranks imputations without complete data","Rank imputations fairly with no complete data","Proper energy score for imputation ranking","Energy I-Score: rank imputations without full data","New score picks best imputation, no need for original"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000246,"raw_usage":{"total_tokens":1593,"prompt_tokens":1050,"completion_tokens":543,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":666,"completion_tokens_details":{"reasoning_tokens":469}},"tokens_in":666,"tokens_out":543,"duration_ms":6581,"temperature":1.0,"reasoning_tokens":469,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:11:45.327901+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take the dependent-uniform example of Appendix B.1, where $X_1$ and $X_2$ are dependent and missingness depends on $X_1$, so CIMAR$_2$ fails; compute the population energy-I-Score. If any imputation distribution other than $P^*$ achieves a strictly higher averaged score, the theorem as stated is falsified because the averaged score includes a variable whose CIMAR condition fails.","supporting_citations":[{"cited_title":"I., Grimit, E","cited_arxiv_id":null,"evidence_quote":"Gives the sample-based energy score estimator used in the score's estimation algorithm."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Introduces the pattern-mixture model factorization on which the paper's missingness definitions rest."},{"cited_title":"Missing at Random","cited_arxiv_id":null,"evidence_quote":"Supplies the notation and discussion of MAR variants used to state the assumptions."}],"review_version":1}