{"id":"f16ec03d-0a77-4de6-9cde-90d0826fd3bf","arxiv_id":"2608.01333","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new tree-based algorithm, utree, detects and quantifies subgroups with differential clinical risk model performance without pre-specifying the groups.","lead":"The paper proposes an unfairness tree (utree), a recursive partitioning method that automatically discovers patient subgroups where a clinical risk model performs differently. It tests the method in simulations and on six mortality models using the GUSTO-I trial data.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Permutation null in Sec 3.1 tests independence of (Y^0, s) from Z, not the calibration null; conditional-variance heterogeneity can make node-level type I error exceed nominal.","rationale":"The reader identifies the permutation null under recursive splitting as the weakest assumption; I agree that this is the right area, but I sharpen it to a node-level mismatch that is more fundamental than the already-acknowledged tree-level multiplicity gap. The central claim that the utree has empirical type I error near nominal rests on the permutation p-values being valid under the calibration null. The paper's own Appendix E shows that the null is a conditional-mean restriction, not an independence restriction, and the permutation procedure is exact for independence. Because Var(D|Z) varies with Z in the paper's own null simulation, the claimed 0.048 is evidence about one simulation configuration, not a general property. The real-data findings are presented as exploratory and the counterfactual assumptions are stated clearly, so the main risk is not those analyses but the method's inferential validity. This risk is addressable by a concrete simulation check, so a conditional acceptance is appropriate: the paper should either provide the check showing nominal type I error under a variance-heterogeneous null, or revise the inference procedure and re-report the operating characteristics. I do not see grounds for rejection: the framework is coherent, the simulations are extensive, and the limitation about tree-level guarantees is explicitly disclosed.","tokens_in":25643,"tokens_out":11749,"duration_ms":121880,"concrete_test":"Simulate the simplest possible null: n=1,000, Z~Uniform(0,1), true risk p(Z)=logit^{-1}(-3Z), Y^0~Bernoulli(p(Z)), and RAI score \\hat s=p(Z) exactly (perfect calibration). With only this single covariate, compute the Section 3.1 KS statistic and its permutation p-value (500 permutations) at alpha=0.05; repeat 10,000 times and record the rejection rate. Under a valid node-level test the rate should be 0.05 plus or minus about 0.005. If it exceeds roughly 0.07, the permutation null is miscalibrated and the utree's type I error claim does not generalize. If it is near 0.05, repeat with p(Z)=logit^{-1}(-3(1-Z)) (variance concentrated at low Z) and with the full utree plus 10 noise covariates to check the adaptive-splitting contribution separately.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The load-bearing assumption is that the permutation procedure in Section 3.1 gives a valid reference distribution for the KS statistic under the stated null H0: F_j(z; w_{\\hat Y^0}) = F_j(z; w_{\\hat s}) for all z. Let D_i = \\hat Y^0_i / E[\\hat Y^0] - \\hat s_i / E[\\hat s]. Appendix E shows H0 is equivalent, for continuous Z_j, to E[D_i | Z_i = z] = 0 almost surely: a conditional-mean restriction. The permutation test, however, generates its null by randomly reassigning the pairs (\\hat Y^0_i, \\hat s_i) to Z values, so under the permutation distribution D is independent of Z. These two nulls coincide only if the conditional law of D given Z is free of Z. When E[D|Z] = 0 but Var(D|Z) varies with Z, the true null distribution of the sup-statistic depends on the integrated conditional variance \\int_{-\\infty}^z Var(D|Z=u) dF(u), whereas the permutation distribution uses E[D^2] F(z). These can differ substantially; if Var(D|Z) is large in regions where F(z) is small, the observed statistic has heavier tails than the permutation statistic and p-values are anti-conservative. This is not a pathological corner case: in the paper's own null simulation, \\hat s_i equals the true risk s^0_i and Y^0_i is Bernoulli(s^0_i), so Var(D|Z) = s^0_i(1-s^0_i)/E[s^0]^2 varies with Z. The reported 0.048 null split probability therefore reflects one conditional-variance profile rather than a validity guarantee. The Section 6 limitation about tree-level guarantees is honest but does not cover this node-level null mismatch; if the node-level test is miscalibrated, the entire subgroup discovery step inherits the error. A focused simulation is needed before the central type I error claim can be accepted as general.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the unfairness tree (utree), a recursive partitioning method for data-driven discovery of subgroups in which a clinical risk prediction model exhibits differential performance. The method uses conditional-inference trees, with variable and split-point selection guided by a Kolmogorov–Smirnov-type statistic comparing covariate distributions weighted by counterfactual outcome estimates and predicted risks. A doubly robust estimator is used to estimate counterfactual outcomes under a reference treatment. The authors present simulations varying sample size, correlation, and unfairness-generating mechanisms, reporting type I error near nominal level, reasonable power to detect higher-order interactions, and small bias in leaf-level discrepancy estimates. They then apply the method to six mortality risk models in the GUSTO-I trial, identifying subgroups defined by combinations of age, sex, blood pressure, Killip class, and other variables. The paper acknowledges several limitations, including the lack of formal tree-level inference and the exploratory nature of the subgroup estimates.","tokens_in":26071,"tokens_out":5871,"duration_ms":50681,"significance":"If the approach is valid, it addresses a real gap in fairness evaluation: existing methods typically require pre-specified groups, whereas the utree automatically discovers intersecting subgroups with heterogeneous model performance. The paper provides a useful algorithmic framework that extends to multiple performance metrics through weighting functions, builds on the conditional-inference tree literature to avoid variable-selection bias, and is evaluated in both simulations and a substantial clinical dataset. The authors are also commendably explicit about the exploratory nature of the estimates and the open theoretical questions. However, the central statistical guarantee—nominal type I error under the stated null—is not established by the current permutation procedure, which is a load-bearing issue for the paper's main claim.","major_comments":[{"comment":"The permutation null in Section 3.1 is not the same as the null hypothesis stated for the test statistic. The statistic T_j in Eq. (5) is said to test H0: F_j(z; w_{Y^0}) = F_j(z; w_s) for all z, which Appendix E shows is equivalent to E[Y^0 | Z=z]/E[Y^0] = E[s | Z=z]/E[s] almost surely, i.e., a conditional-mean restriction. The permutation procedure jointly permutes the pairs (Y^0_i, s_i) across observations while holding Z fixed, which instead generates a reference distribution under the stronger condition that (Y^0, s) is independent of Z. When the conditional-mean null holds but Var(Y^0 | Z) varies with Z, the true null distribution of the supremum statistic depends on the conditional variance function, whereas the permutation distribution uses the unconditional variance. In the paper's own null simulation (Section 4.1), s_i equals the true risk s^0_i and Y^0_i is Bernoulli(s^0_i), so Var(Y^0_i | Z_i) = s^0_i(1 - s^0_i) varies with Z; the reported split probability of 0.048 (Table A3) therefore reflects one conditional-variance profile and does not establish general type I error control. The Section 6 limitation about tree-level guarantees does not address this node-level validity gap. I recommend either proving that the permutation procedure yields a valid reference distribution for the KS-type statistic under the conditional-mean null, or replacing it with a method that is valid under conditional heteroskedasticity (e.g., a wild bootstrap), and re-running the type I error simulations under several conditional-variance profiles.","section":"§3.1 / Appendix E"},{"comment":"The split-point selection step is not given a valid inference justification. After selecting the covariate Z_j, candidate split points are evaluated with the same statistic T, and the split point that maximizes T is chosen. The p-value associated with the selected split point is then used in subsequent steps, but no multiplicity adjustment for the search over candidate split points is described. Because the split point is selected to maximize the test statistic, the reported p-value for the resulting split is stochastically smaller than nominal under the null. The simulation result of a root-level split probability near 0.05 is encouraging but does not replace a formal treatment, and the recursive use of these p-values for further splitting is even less well understood. The authors should clarify how split-point multiplicity is handled, or state explicitly that the p-values are heuristic guides rather than exact inferential quantities.","section":"§3.1 / Split-point selection"},{"comment":"The GUSTO-I analysis lacks uncertainty quantification for leaf-level discrepancies, and the doubly robust estimator is not cross-fitted. Section 5.1 states \"Cross-fitting was not employed,\" and Figures 3 and A5 report only point estimates. Because the tree structure is learned adaptively on the evaluation set, the leaf discrepancies are affected by selection; Section 6 appropriately cautions that they are exploratory summaries, but the absence of any confidence intervals or standard errors makes it difficult to judge which of the reported terminal-node discrepancies (e.g., relative discrepancies of +250% or +500% in Figure 3) are compatible with sampling noise. Cross-fitting the nuisance models in the doubly robust estimator would reduce bias due to overfitting the outcome and propensity models in the evaluation sample. I request that the authors add standard errors or confidence intervals for node-level discrepancies, even if labeled exploratory, and consider cross-fitting for the DR nuisance models.","section":"§5.1 / §3.3"}],"minor_comments":[{"comment":"The phrase \"a novel data-driven method that that partitions\" contains a duplicated \"that\" that should be corrected.","section":"§3.1"},{"comment":"The notation ⊕ [ Z3 < −2 ] and similar expressions appear to be rendering errors; they should use standard indicator notation such as 𝔹[Z3 < −2].","section":"§3.1"},{"comment":"The Hothorn reference is typeset incorrectly: \"Hornik , Kurt, , and Achim Zeileis\" should be \"Kurt Hornik and Achim Zeileis.\"","section":"References"},{"comment":"The caption contains missing spaces in \"increasingly smallterminalleavesareexcluded,\" which should read \"increasingly small terminal leaves are excluded.\"","section":"Figure A6 caption"},{"comment":"The table uses the label \"KillipII\" for Killip class II, while the figures use the abbreviation \"K2\"; please make the notation consistent.","section":"Table A5 / Figures"}],"recommendation":"major_revision","confidential_remarks":"The paper is within the journal's scope and the authors are transparent about limitations. The central concern is the validity of the permutation null for the stated calibration hypothesis; this is fixable with a revised inference procedure and additional simulations, so I do not recommend rejection. The real-data uncertainty quantification issues are secondary but should be addressed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe utree is a useful tool for a real gap: finding unknown intersecting subgroups where a clinical risk model is miscalibrated. The paper is honestly written and the simulations are well designed, but the type I error claim is not as general as it appears, because the permutation test used for splitting tests a stronger null than the calibration null.\n\nWhat's new: the combination of conditional inference trees with the weighted-KS discrepancy statistics from Wastvedt et al., plus doubly robust counterfactual outcomes. That is a legitimate, practical step forward. The simulations cover a nice range of sample sizes, correlations, and unfairness mechanisms, and they report variable and interaction recovery and bias, not just power. The GUSTO-I analysis is thorough: six models, three metrics, out-of-sample validation, and sensitivity to leaf size. The limitations section is candid about tree-level inference and post-selection issues.\n\nSoft spots:\n\n1. The permutation test in Section 3.1 is the main issue. The null hypothesis of interest is equality of the two weighted CDFs, which Appendix E shows is equivalent to E[D|Z]=0 for D = Y^0/E[Y^0] - s/E[s]. But the permutation procedure imposes independence of (Y^0, s) from Z. These coincide only if the conditional distribution of D given Z is free of Z. Under the calibration null with variance heteroscedasticity, the permutation distribution can be miscalibrated, and the reported size of 0.048 is one variance profile, not a guarantee. The paper's own Section 6 limitation doesn't cover this node-level mismatch; it only discusses tree-level guarantees. This deserves a focused simulation or a theoretical justification.\n\n2. The real-data analysis uses no cross-fitting for the doubly robust outcomes, and the leaf-level discrepancies have no confidence intervals. The out-of-sample correlations are a good start, but I'd like uncertainty bands on the reported discrepancies. Minor, addressable.\n\n3. No code or data is provided, which limits reproducibility. Also minor for this stage.\n\nThe paper is worth a serious referee. The central idea is sound, the writing is clear, and the limitations are mostly acknowledged. I would send it out, with the permutation-test issue as a required revision. I'd probably cite it as the data-driven alternative to pre-specified subgroup fairness testing.","headline":"Useful, honest method for discovering unknown subgroup disparities, but its permutation null is not the calibration null, so the type I error guarantee is narrower than claimed.","tokens_in":26534,"tokens_out":3884,"would_cite":true,"duration_ms":37133,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62H30","62G10","62P10"],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper proposes the unfairness tree (utree), a recursive partitioning method that discovers covariate-defined subgroups where a clinical risk model is miscalibrated, without requiring the user to pre-specify groups, and reports that…","keywords":["Statistical machine learning","Model evaluation","Risk prediction","Counterfactual inference","Recursive partitioning","Clinical prediction models","Algorithmic fairness","Unfairness tree"],"falsifier":"A large-scale null simulation is the direct check: generate data with no differential performance, run the utree on tens of thousands of replicates at several sample sizes, and compare the proportion of trees making at least one split with the nominal 0.05. If that proportion rises systematically with sample size rather than staying flat near 0.05, the permutation null is not valid after adaptive splitting, and the method's central operating-characteristic claim fails. A within-tree check would compare permutation p-values at a node reached after several splits with p-values computed from an independent sample using the same node definition.","tokens_in":25467,"feed_emoji":"🌳","tokens_out":10143,"duration_ms":84653,"temperature":0.7,"pith_summary":"Clinical risk models can be well calibrated overall yet badly wrong in subgroups defined by several intersecting characteristics, and standard fairness audits check only groups the analyst names in advance. This paper proposes the unfairness tree (utree), a recursive partitioning method that lets the data reveal subgroups where predictions and observed outcomes diverge. The paper's central claim is that the utree detects local miscalibration without pre-specified groups, keeps the probability of a spurious split near the nominal level (about 0.05), and grows better at finding true disparities as sample size and signal strength increase. A sympathetic reader would care because this turns subgroup fairness from a checklist question into a discovery problem: which patients, defined by which combinations of characteristics, are being systematically over- or under-predicted.","feed_headline":"Tree finds hidden subgroups where risk models misfire","feed_subtitle":"No pre-specified groups needed; split selection stays near nominal false-alarm rate and grows sharper with more data.","key_machinery":"The engine is the utree algorithm, built on conditional-inference trees. For each candidate splitting covariate $Z_j$, it computes the statistic $T_j = \\sup_z \\left| \\hat{F}_j(z; w_{\\hat{Y}^0}) - \\hat{F}_j(z; w_{\\hat{s}}) \\right|$, where $\\hat{F}_j(z; w)$ is the weighted empirical cumulative distribution function of $Z_j$ under weights given by the doubly robust counterfactual outcome estimates and by predicted risk. This is a Lorenz-curve comparison: under calibration the two weighted distributions coincide, and their maximum gap is the test signal. Variable and split-point selection are separated, and significance is assessed by permuting the $(\\hat{Y}^0, \\hat{s})$ pairs jointly while holding $Z$ fixed. The same recursion can target true-positive rate or precision by changing the weight functions, so the tree machinery is a general search over subgroups for whichever performance discrepancy the user defines.","core_discovery":"The central discovery is that a conditional-inference tree whose splits are guided by a Kolmogorov–Smirnov-type statistic can localize differential model performance. At each node the method compares the covariate distribution weighted by estimated counterfactual outcomes $\\hat{Y}^0$ with the same distribution weighted by predicted risks $\\hat{s}$; a large gap means observed and predicted risk are distributed differently along that covariate, so the covariate is informative about where calibration fails. In simulations the method shows split probabilities near nominal level under the null (0.048 at $N=10^3$), increasing power under miscalibration and imbalance scenarios, and leaf-wise estimates of subgroup discrepancy with small bias. In six mortality models fitted to the GUSTO-I trial, age, sex, systolic blood pressure, and Killip class were repeatedly selected, and the terminal subgroups it found often involve interactions of three or more variables rather than single-axis partitions.","pith_inferences":["Editorial extension: because Appendix E shows the normalized statistic is blind to a constant multiplicative bias, a real-world audit should pair utree with a global calibration check; a model that is uniformly overconfident could pass the tree screen.","Editorial extension: the same recursive template transfers to any setting with a binary outcome, a risk score, and a defensible counterfactual reference, provided positivity, consistency, and no-unmeasured-confounding hold.","Editorial extension: a natural next simulation study, named but not carried out here, is to run the same battery with TPR and precision weight functions to check whether type I error and recovery guarantees extend to threshold-based metrics.","Editorial extension: until selective inference for adaptive splits exists, users should read terminal-leaf discrepancies as ranked hypotheses to validate, not as confidence intervals."],"forward_implications":["Fairness audits can operate without a pre-specified group list; intersectional subgroups are the output of the tree, not the input.","Evaluation covariates need not be the model's inputs, so utree can ask whether a risk model performs differently across characteristics like socioeconomic status that the model never saw.","The same partitioning logic with different weight functions targets calibration, true-positive rate, or precision, and the GUSTO-I analysis shows the chosen metric changes which subgroups are flagged.","Leaf-wise discrepancy estimates give each patient a predicted calibration error for their subgroup, and out-of-sample checks in the paper indicate these estimates are stable enough to use as exploratory summaries.","Because the statistic ignores uniform multiplicative miscalibration shared by all groups, utree is a test for heterogeneous relative miscalibration, not a substitute for checking overall calibration."],"supporting_citations":[{"why":"Supplies the conditional-inference tree framework whose unbiased variable and split-point selection the utree adopts.","marker":"[Hothorn et al., 2006]"},{"why":"Provides the discrepancy statistics and counterfactual intersectional fairness setup that the utree adapts, along with the simulation's outcome-generating settings.","marker":"[Wastvedt et al., 2023]"},{"why":"Supplies the doubly robust estimators of potential outcomes and the threshold-based metric definitions used to build counterfactual calibration.","marker":"[Coston et al., 2020]"},{"why":"Establishes the Lorenz-curve ordering idea that motivates comparing outcome-weighted and prediction-weighted cumulative distributions.","marker":"Yang [2024]"},{"why":"Supplies the acute myocardial infarction trial data on which the six mortality risk models are fit.","marker":"[GUSTO, 1993]"},{"why":"Defines the mortality prediction models whose subgroup performance the utree evaluates.","marker":"[Lee et al., 1995]"},{"why":"Defines the CART baseline whose variable-selection bias motivates the conditional-inference approach utree uses.","marker":"[Breiman, 1984]"}],"fun_headline_variants":["Tree finds groups where risk models misfire","UTREE: automatic discovery of risk model disparities","No pre-specified groups: tree finds unfair calibration","Data-driven tree reveals hidden risk model biases","Statistical tree localizes differential model performance"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the permutation test—which jointly shuffles outcome estimates and predicted risks while holding covariates fixed—still gives a valid reference distribution after the tree has chosen splits greedily and adaptively from the data. The paper states that formal guarantees for tree-level inference and subgroup stability are open, so if those p-values drift, some reported subgroups could be artifacts of the search rather than real disparities.","fun_headline_variants_meta":{"raw":{"variants":["Tree finds groups where risk models misfire","UTREE: automatic discovery of risk model disparities","No pre-specified groups: tree finds unfair calibration","Data-driven tree reveals hidden risk model biases","Statistical tree localizes differential model performance"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000301,"raw_usage":{"total_tokens":1727,"prompt_tokens":931,"completion_tokens":796,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":547,"completion_tokens_details":{"reasoning_tokens":728}},"tokens_in":547,"tokens_out":796,"duration_ms":6972,"temperature":1.0,"reasoning_tokens":728,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T15:07:11.822617+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A large-scale null simulation is the direct check: generate data with no differential performance, run the utree on tens of thousands of replicates at several sample sizes, and compare the proportion of trees making at least one split with the nominal 0.05. If that proportion rises systematically with sample size rather than staying flat near 0.05, the permutation null is not valid after adaptive splitting, and the method's central operating-characteristic claim fails. A within-tree check would compare permutation p-values at a node reached after several splits with p-values computed from an independent sample using the same node definition.","supporting_citations":[],"review_version":2}