{"id":"f074147a-658e-4457-8458-775d0ee7d805","arxiv_id":"2607.21782","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Weighted versions of the KS, Cramér–von Mises, and Anderson–Darling tests are introduced using label permutation, and simulations suggest Anderson–Darling as a general-purpose balance check.","lead":"This paper adds case weights to three classic tests that compare entire distributions, so researchers can check whether weighted treatment and control groups have balanced covariates. It finds the Anderson–Darling test is a good all-purpose choice, while the Kolmogorov–Smirnov test still wins for a specific kind of mismatch.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Permutation test's 'no assumption about weights' claim is untested for IPTW weights; simulation uses exogenous weights.","rationale":"The reader's weakest assumption identifies exactly the concern I find most load-bearing: the label-permutation procedure's validity hinges on treating weights as fixed attributes, which is only a valid null mechanism if (X, W) are exchangeable with treatment labels. The paper's own simulation design (Section 2.6.1) assigns weights independently of y, g, and p, so it never exercises IPTW-type weights where weights encode treatment propensity. The claim in Section 2.5 that the procedure 'requires no assumption about how the weights were generated' is therefore not supported by evidence; it is an assertion about a regime the simulation does not cover. The applied example's p-values (Table 6) are exactly the kind of result that would be invalid if this concern lands. I considered the reader's other objections — the AD standardization formula is asserted without derivation, and Type I error inflates at re=0.2 — but those are secondary: even an unusual AD statistic would be calibrated by permutation under exchangeability, and the re=0.2 inflation is a boundary effect. The permutation-validity issue, by contrast, would invalidate every p-value reported for real estimated weights. A single simulation with IPTW weights under a correctly specified propensity model settles the matter; until then, the paper's conditional verdict is appropriate. Hence I keep the reader's CONDITIONAL verdict unchanged.","tokens_in":14488,"tokens_out":6664,"duration_ms":73751,"concrete_test":"Simulate the Type I error scenario with IPTW weights: for n=2000 (n1=800, n0=1200), draw X~N(0,1), assign G from a logistic propensity score e(X) with an intercept chosen so P(G=1)=0.4 and a nonzero coefficient (e.g., logit(e(X)) = -0.4 + 0.5X), and set w_i = 1/e(X_i) for G_i=1 and 1/(1-e(X_i)) for G_i=0 (optionally trim/normalize so Kish re ≈ 0.6, matching the paper's main simulation). Under this DGP the weighted distributions of X are equal in expectation, so the null of covariate balance holds. Compute the empirical rejection rate of the label-permutation test (R=1000, 2000 replications) at α=0.05 for KS, CVM, and AD. If rates exceed roughly 0.06, or are not within Monte Carlo error of nominal, the 'no assumption about how the weights were generated' claim fails for IPTW weights; if rates stay ~0.05, the concern is resolved and the method's generality is supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim (Section 2.5) is that the shared label-permutation procedure 'requires no assumption about how the weights were generated' because each observation's weight is treated as a fixed attribute. This is load-bearing: the permutation p-value is valid only if the null distribution of the weighted statistic is generated by reassigning labels among fixed (value, weight) pairs. That corresponds to the null under which (X, W) are jointly exchangeable across treatment labels. Under IPTW or entropy balancing, weights are functions of treatment and covariates: e.g., w_i = 1/e(X_i) for treated units and 1/(1-e(X_i)) for controls. Even when the weighted distributions are equal (the null being tested), (X, W) is not exchangeable with G, and permuting labels assigns a treated unit's weight (1/e(X)) to a control label, which is not the weight a control with that X would carry. The permutation distribution can therefore differ from the true null distribution. Section 2.6.1 explicitly assigns weights independently of y, g, and p, so Tables 2 and 5 only validate the permutation test under exogenous weights, not weights 'of any origin.' The applied example (Section 4) uses IPTW weights, so its p-values (Table 6) inherit this unvalidated assumption. If the permutation calibration fails under estimated weights, the headline claim that all three tests control Type I error for any weighting scheme — and the AD recommendation based on those p-values — collapses.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript extends the two-sample Kolmogorov-Smirnov, Cramér–von Mises, and Anderson-Darling tests to case-weighted data for covariate-balance assessment. It defines weighted ECDFs and weighted statistics, and calibrates all three using a single label-permutation procedure that treats each observation's weight as a fixed attribute. The simulation study uses exogenous lognormal weights at sample sizes 1,000–4,000 and reports Type I error near nominal and preserved comparative advantages (KS for central, AD for tail, AD/CVM for diffuse discrepancies). An IPTW application to BMI is presented, and Stata implementations are referenced. The paper recommends AD as a reasonable general-purpose default.","tokens_in":14849,"tokens_out":7707,"duration_ms":79681,"significance":"If the permutation calibration is valid for weights of the kind used in practice, the paper fills a practical gap: applied covariate-balance assessment after IPTW or entropy balancing currently relies mostly on moment-based summaries, and weighted distributional tests are not routinely available. The weighted KS and CVM forms follow naturally from weighted ECDFs, the shared permutation inference is transparent and reduces to the classical unweighted test, and the simulation includes paired power comparisons and a weight-variability sensitivity check. These are genuine strengths. However, the central generality claim — that the inference 'requires no assumption about how the weights were generated' — is not supported by the simulations, which assign weights exogenously and independently of treatment; the applied example uses IPTW weights. The AD_w standardization through Kish effective sample size is asserted rather than derived. These points affect the validity of the headline claims and require revision.","major_comments":[{"comment":"The claim that the label-permutation procedure 'requires no assumption about how the weights were generated' is broader than what is shown. Permuting labels among fixed (value, weight) pairs generates a null under which (X, W) are jointly exchangeable across treatment labels. For IPTW or entropy-balancing weights, W is a function of G and X; even when weighted covariate distributions are equal, (X, W) is typically not exchangeable with G. The simulation (Eq. 12) assigns weights independently of y, g, and p, so Tables 2 and 5 validate the procedure only under exogenous weights. The applied example (Sec. 4.1) uses IPTW weights, so the p-values in Table 6 inherit this unvalidated assumption. Please either prove/examine validity for estimated weights, add simulations with estimated propensity or entropy weights, or explicitly restrict the claim.","section":"Sec. 2.5 / Sec. 2.6.1"},{"comment":"The Anderson-Darling standardization is not derived. Eq. (7) contains an ambiguous factor 'N·2' (and Eq. (9) an analogous 'ne·2') whose origin is unclear; the usual null variance of the ECDF difference involves a term like D(1-D)(1/n1+1/n0), not an unexplained N×2 constant. Since permutation p-values are invariant to a common multiplicative constant, this ambiguity does not invalidate the reported p-values, but it affects the reported statistic and the interpretation of AD_w. More importantly, replacing N by the Kish effective sample size n_e (Eq. 8) is asserted as 'the standard adjustment' without derivation; this choice changes the relative weighting across x and should be justified or supported by simulation under estimated weights.","section":"Sec. 2.4, Eqs. (7)-(9)"},{"comment":"At the most severe weight variability examined (r_e = 0.2), all three tests show Type I error rates of .064-.067, a relative inflation of roughly 30% over nominal .05. The text acknowledges this but the abstract and Sec. 5.1 conclude that all three tests 'controlled Type I error close to nominal under substantial weight variability' without stating the boundary of validity. If r_e = 0.2 is within the intended scope, the claim should be qualified; if it is outside, the qualifying condition should be stated. As written, the conclusion is stronger than the table supports.","section":"Table 5 / Sec. 3.6"},{"comment":"The recommendation of AD as a general-purpose default is broader than the simulation evidence. The DGP is normal throughout, each discrepancy type is examined at one fixed effect size, and the scenarios are deliberately constructed to correspond to the known theoretical strengths of the three tests (Sec. 2.6). Such scenarios can show that weighting preserves known comparative advantages under exogenous weights, but they do not establish that AD is the best default under realistic imbalance structures, non-normal covariate distributions, or estimated weights. Please temper the applied recommendation or broaden the simulation accordingly.","section":"Secs. 2.6, 5.3"}],"minor_comments":[{"comment":"The formula is difficult to parse: the placement of the 'N·2' factor and the square root is unclear. Please rewrite with explicit numerators/denominators and define all symbols at first use.","section":"Eq. (7)"},{"comment":"The tie multiplier tau_k is retained as the raw unweighted count. For weighted data this is not obviously the right information measure; the caveat in Sec. 5.3 is useful, but the property is not examined in the simulations.","section":"Sec. 2.4.2 / Sec. 5.3"},{"comment":"The citation for Somers' D (Newson and Falcaro, 2023) points to a reference titled 'Robit regression in Stata', which appears unrelated. Please verify the reference.","section":"Introduction"},{"comment":"The definition of the permutation p-value (Eq. 10) uses '#' rather than an explicit count and could be made clearer by writing (1 + sum_j 1(T_j^* >= T_obs))/(R+1).","section":"Sec. 2.5"}],"recommendation":"major_revision","confidential_remarks":"To the editor: The manuscript addresses a practical need and the simulation work is substantial, but the central 'any origin' generality claim is not yet supported because the simulations use exogenous weights while the applied example uses IPTW. This is fixable in a revision: either add a proof or structural argument for exchangeability under estimated weights, or add simulations with estimated propensity/entropy weights, and adjust the abstract and conclusion accordingly. The AD_w standardization also needs derivation or further support. I do not see grounds for rejection, but the revision should be substantive."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, the weighted extensions are simple and the paper is clear: it replaces raw ECDFs with weighted ECDFs, uses a Kish effective sample size for AD, and runs a single permutation test. The simulations are competently done and the Type I error results under exogenous lognormal weights are fine, apart from a modest inflation at the most extreme weight variability. Second, the central claim that the permutation procedure 'requires no assumption about how the weights were generated' is not supported. The simulations assign weights independently of treatment and outcome, so they only validate exchangeability of (value, weight) pairs across groups. Under IPTW, weights are functions of treatment and covariates, and the pairs are not exchangeable. The applied IPTW example inherits this unvalidated assumption. The stress-test note is right.\n\nWhat is good? The paper addresses a real gap—routine balance assessment after weighting is usually mean-focused, and distributional tests are useful. The formulas reduce to known unweighted statistics when weights are equal. The Stata commands are a practical contribution. The simulation design, with paired comparisons and sensitivity analysis, is thoughtful.\n\nSoft spots, in order. The permutation validity issue is the main one. It is not a killer for the paper's applied recommendation if the permutation test happens to work for IPTW, but we have no evidence. The paper should either prove exchangeability-type conditions or simulate with estimated propensity score weights. Second, the AD standardization via Kish ESS is asserted without derivation, and Eq. (7) contains an ambiguous 'N·2' factor. The formula needs to be checked against the standard two-sample AD variance. Third, the claim that this is the first weighted extension is overstated; weighted KS and AD appear in survey statistics, and prior work should be cited. Minor: Type I error rises to about 0.065 at re=0.2, which the paper acknowledges.\n\nWho is this for? Applied causal inference researchers who want a distributional balance diagnostic. It deserves a serious referee. I would send it out, expecting a major revision that addresses the permutation assumption and the AD formula. The empirical comparison of KS/AD/CVM under weighting is worth keeping even if the general claim is narrowed.","headline":"Useful weighted ECDF tests for balance diagnostics, but the 'works for any weights' claim is only validated for exogenous weights—under IPTW the permutation null is not obviously correct.","tokens_in":15256,"tokens_out":5010,"would_cite":false,"duration_ms":50133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G10","62G30","62G09"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper extends the Kolmogorov-Smirnov, Cramer-von Mises, and Anderson-Darling tests to weighted samples and shows, through a single label-permutation calibration, that all three keep false-positive rates near nominal while preserving th","keywords":["covariate balance","weighted goodness-of-fit","Kolmogorov-Smirnov test","Anderson-Darling test","Cramer-von Mises test","permutation inference","case weights","propensity score weighting"],"falsifier":"Generate a dataset in which treatment depends on a covariate through a logistic model, construct inverse-propensity weights, and then apply the three permutation tests to a covariate that is truly balanced after weighting; if the tests reject more often than nominal across sample sizes, the claim that calibration is independent of how weights were generated is falsified.","tokens_in":14373,"feed_emoji":"⚖️","tokens_out":7397,"duration_ms":73897,"temperature":0.7,"pith_summary":"After matching, weighting, or any other adjustment, researchers need to check whether a covariate's whole distribution is balanced across groups, not just its mean or variance. This paper makes three classical distribution-comparison tests—Kolmogorov-Smirnov, Cramer-von Mises, and Anderson-Darling—usable on weighted samples, and calibrates all three with one label-permutation procedure that treats each case's weight as a fixed attribute. In simulations with heavily varying weights, all three tests rejected at rates close to the nominal 5% under a true null; KS remained the most powerful for a central discrepancy, AD was clearly best for a tail discrepancy, and AD and CVM performed comparably for a diffuse shift. The paper's practical conclusion is that AD is a reasonable general-purpose default for routine covariate-balance checks, with KS reserved for when a central imbalance is specifically suspected.","feed_headline":"Choose Anderson-Darling as the default weighted balance test","feed_subtitle":"Simulations show AD catches tail and diffuse imbalances; use KS only when the shift is central.","key_machinery":"The apparatus is (1) the weighted empirical CDF, where each observation contributes w_i/sum(w) instead of 1/n; (2) the Kish effective sample size inserted into AD's variance standardization, which prevents the weighted null variance from being understated when weights vary; and (3) the permutation p-value, computed as the proportion of relabeled datasets whose statistic equals or exceeds the observed one. These pieces make the weighted statistics reduce exactly to their classical forms when all weights are equal, and they make the inference independent of how the weights were generated.","core_discovery":"The paper's central claim is that the Kolmogorov-Smirnov, Cramer-von Mises, and Anderson-Darling statistics can be lifted to any positive case-weight setting by replacing ordinary empirical CDFs with weighted empirical CDFs, and in the Anderson-Darling statistic by replacing the sample size N with the Kish effective sample size inside the variance-standardization term. All three statistics are then calibrated with one label-permutation procedure that shuffles group labels over fixed (value, weight) pairs. The paper argues, and supports by simulation, that this yields tests whose false-positive rates stay near nominal under substantial weight variability, that the classical sensitivity orderi","pith_inferences":["If weights are estimated from treatment and covariates, as with inverse-propensity weighting, each unit's weight already encodes its treatment propensity; the fixed-weight permutation null may then not match the true null. A direct simulation with treatment-dependent weights would settle whether the 'no assumption about weight origin' claim holds beyond exogenously assigned weights.","The diffuse-scenario result suggests AD and CVM coincide near a uniform shift because AD's variance standardization has nothing to emphasize. Varying the discrepancy's functional form or the base distribution's shape could map the boundary between the tests' regimes.","Because every formula uses weights only in ratios, the tests are invariant to rescaling weights; this likely makes them applicable to survey weights or post-stratification weights without any extra adjustment.","The same weighted-ECDF-plus-permutation framework could be extended immediately to the Kuiper or Wasserstein statistics, which the paper itself names as future directions, adding tests sensitive to cyclical or cost-scaled discrepancies."],"forward_implications":["Applied researchers can now run full-distribution balance checks on weighted data, rather than relying only on standardized mean differences and variance ratios.","A single permutation calibration works across all three statistics and any positive weighting scheme, so no separate null-distribution formula is needed for each weighting method.","The known sensitivity profiles carry over: KS is the most powerful test for a centrally located imbalance, AD is dramatically the best for tail imbalance, and AD and CVM are comparably good (and both better than KS) for a diffuse shift.","The paper's recommendation is to use AD as the routine default for balance diagnostics, to prefer KS when a central imbalance is specifically suspected, and to consider reporting AD and KS together to cover both blind spots.","At very severe weight variability (effective sample size ratio 0.2), all three tests show a modest false-positive inflation—about 0.064 to 0.067 at nominal 0.05—so calibration is not perfectly invariant to extreme weights."],"fun_headline_variants":["Weighted KS, CVM, AD tests now handle any case weights","Anderson-Darling wins for weighted covariate balance checks","Extend KS, CVM, AD to weighted data with permutation test","AD test best default for weighted balance, KS for central shifts"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The load-bearing premise is that valid inference follows from permuting treatment labels across fixed (value, weight) pairs under the null; this exchangeability can break when weights are themselves estimated from treatment and covariates, because each weight already encodes information about the label being shuffled.","fun_headline_variants_meta":{"raw":{"variants":["Weighted KS, CVM, AD tests now handle any case weights","Anderson-Darling wins for weighted covariate balance checks","Extend KS, CVM, AD to weighted data with permutation test","AD test best default for weighted balance, KS for central shifts"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000635,"raw_usage":{"total_tokens":2781,"prompt_tokens":777,"completion_tokens":2004,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":521,"completion_tokens_details":{"reasoning_tokens":1930}},"tokens_in":521,"tokens_out":2004,"duration_ms":12234,"temperature":1.0,"reasoning_tokens":1930,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-01T06:42:01.253579+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Generate a dataset in which treatment depends on a covariate through a logistic model, construct inverse-propensity weights, and then apply the three permutation tests to a covariate that is truly balanced after weighting; if the tests reject more often than nominal across sample sizes, the claim that calibration is independent of how weights were generated is falsified.","supporting_citations":[],"review_version":1}