{"id":"05343bef-c8b3-4f33-a764-1b98955a1784","arxiv_id":"1908.03652","paper_version":2,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A matched-pair instrumental variables method using CART on absolute adjusted outcome differences and closed testing discovers and tests heterogeneous complier treatment effects with familywise error control.","lead":"This paper proposes a way to find subgroups of people whose response to a treatment differs when the treatment is studied through a random lottery or similar instrument. It combines matching, a decision tree on absolute outcome differences, and a multiple-testing correction, and it applies the method to Medicaid in Oregon.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"FWER proof for sparse subgroup statistics relies on an imported asymptotic normality result that is not established for the small, low-compliance intersections produced by CART.","rationale":"The reader identified exactly the same load-bearing concern: the size of subset-specific statistics is imported from prior work without reproof for CART-generated intersections, and the depth tuning in Section 4.3 is not shown to be a function of (X, |Y|) only. My stress-test pass confirms both points from the manuscript text. Proposition 1's proof is conditional on G, but for the guarantee to apply to the actual algorithm, G must be a function of (F, Z) only through |Y|; Section 4.3 admits that depth was chosen by pruning back to an interpretable tree, which is a subjective, potentially Z-dependent decision. The simulation evidence in the paper does not test the sparse-subgroup regime: Section 3 uses 2000 pairs with compliance rates from 0.25 upward, and Web Appendix A reports one FWER simulation with 2000 draws but does not isolate small intersections or the OHIE compliance level of 0.29. These are not internal inconsistencies in Proposition 1's logic, but they are unverified conditions on which the central claim depends. The OHIE findings themselves are plausible and the method is a coherent extension of Hsu et al. (2015), so I would not reject the paper. I would keep the CONDITIONAL verdict: the paper should either prove or cite a uniform asymptotic normality result for data-dependent subsets, add a calibration simulation at the OHIE sample size and compliance rate, and either fix the tree depth a priori or demonstrate that the pruning choices are functions of X and |Y| only. No accusation of dishonesty is intended; these are standard robustness gaps in a new procedure.","tokens_in":23060,"tokens_out":2106,"duration_ms":20577,"concrete_test":"Recompute the OHIE tree and the full closed testing procedure under an alternative depth-selection rule that is explicitly a function of (X, |Y|) only, e.g., fixing depth 4 a priori instead of pruning for interpretability after inspecting results. If the set of rejected nodes changes, the reported discoveries are not robust to the data-dependent tuning and the Proposition 1 guarantee cannot be invoked for the published tree. Additionally, run a calibration simulation matching the OHIE pair count, compliance rate (about 0.29), and tree configuration: generate data under the sharp null lambda0 = 0 with the OHIE covariates and pair structure, fit the same rpart settings, and compute the empirical FWER of Algorithm 1 over 10,000 repetitions, reporting the largest per-intersection rejection rate among all visited intersections.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim is Proposition 1: conditional on (F,Z,G), Algorithm 1 makes at least one false rejection with probability at most alpha under the sharp nulls. The proof in Web Appendix B hinges on every intersection hypothesis H0L being testable at level alpha by the statistic |T_s(lambda0)/S_s(lambda0)|. That size claim is imported from Baiocchi et al. (2010) and Kang et al. (2016), where it is derived for a fixed full matched sample with I growing and a fixed pair-matched design, not for the variable, data-dependent intersections s = union of CART leaves that Algorithm 1 visits. In the OHIE analysis, leaves can contain small numbers of pairs (Figure 4 shows nodes with I_s in the hundreds, and the tree search can form any intersection including unions of small leaves), and the compliance rate is about 0.29, so the effective sample of compliers contributing signal to T_s(lambda0)/S_s(lambda0) can be small. Under the sharp null with lambda0 = 0, the adjusted responses R_ij - lambda0 D_ij are just the null outcomes and the test statistic is asymptotically Normal under mild moments, but no proof is given that the asymptotic approximation is uniform over the arbitrary subsets generated by CART, nor that S_s(lambda0) is bounded away from zero for those subsets. The paper's own Web Appendix A only checks FWER by simulation for one configuration; it does not verify size for sparse or low-compliance intersections. A second premise, the independence of G from Z, is also threatened by the Section 4.3 tuning of tree depth by pruning 'until a more interpretable tree was obtained' after seeing results; if depth selection uses the outcome R or Z-dependent information, G can depend on Z and the conditional randomization argument fails.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This paper proposes a method for discovering and testing heterogeneity in the complier average causal effect (H-CACE) in instrumental variables studies with paired matching. The method first pair-matches units on covariates, then grows a CART tree on the absolute value of the adjusted pairwise outcome difference |Y_i|, which is a function of the potential outcomes only under the sharp null, and finally applies closed testing to all unions of tree leaves using the IV test statistic of Baiocchi et al. (2010) and Kang et al. (2016). Proposition 1 claims strong familywise error control at level α conditional on (F, Z, G). The authors present simulations comparing the method with BCF-IV and apply it to the Oregon Health Insurance Experiment, reporting Medicaid effects among older English-preferring non-Asian men and younger English-preferring individuals with at most a high school diploma or GED.","tokens_in":23394,"tokens_out":12384,"duration_ms":128559,"significance":"The proposal addresses a real gap: existing tree-based heterogeneity methods for IV settings either rely on sample splitting or do not provide strong familywise error control, whereas the use of |Y| to decouple discovery from the instrument assignment is elegant and, if valid, permits honest inference without sample splitting. The H-CACE estimand is defined before estimation, so the analysis is not circular. The closed-testing structure is sound conditional on having level-α tests for every intersection hypothesis, and the OHIE application is a useful demonstration. However, the strength of the contribution depends on closing the gap between the imported asymptotic test and the sparse, data-dependent subsets used by CART, and on clarifying the tuning and matching assumptions.","major_comments":[{"comment":"The proof of Proposition 1 in Web Appendix B reduces familywise error control to the claim that each intersection hypothesis H0L is tested at level α by |T_s(λ0)/S_s(λ0)|, and it states the equality P(|T/S| ≥ z_{1−α/2} | F,Z,G) = α/2. This equality is not exact in finite samples: the normal approximation is imported from Baiocchi et al. (2010) and Kang et al. (2016), where it is derived for a fixed matched sample with I growing, not for the variable, data-dependent unions of CART leaves that Algorithm 1 visits. In the OHIE analysis the nodes in Figure 4 have I_s in the hundreds and compliance near 0.29, so the subset statistics can have a much smaller effective sample of compliers; no uniform asymptotic or bounded-variance condition is provided for S_s(λ0). Please either prove the size property under explicit conditions on the class of possible groupings, or replace the normal approximation with an exact randomization test conditional on (F,Z,G), so that Proposition 1 holds exactly as stated.","section":"Web Appendix B; Eqs. (2)-(3); Algorithm 1"},{"comment":"Proposition 1 relies on G being a function of X and |Y| only. Section 4.3 reports that the OHIE tree depth was chosen by forming trees of larger depth and pruning back \"until a more interpretable tree was obtained.\" If this search uses estimated subgroup effects, confidence intervals, or p-values, then G depends on Z and the conditional randomization argument fails. Please specify the exact tuning procedure for the complexity parameter, maximum depth, and pruning, and verify that it is a deterministic function of (X, |Y|) alone.","section":"Section 4.3; Algorithm 1, Step 3"},{"comment":"The definition of H-CACE at covariate value x is rewritten as a pair-subset estimand λ_s under the assumption that units within a matched pair have identical covariate values. The OHIE matching in Section 4.1 uses an optimal caliper and a robust rank-based Mahalanobis distance on age and several categorical variables, which does not generally yield exact covariate agreement within pairs. The paper does not define the pair-level covariate Xi used by CART or explain how approximate matching changes the interpretation of λ_s as a covariate-defined H-CACE. Please state the exact-matching assumption explicitly or redefine the estimand and the discovered subgroups at the level of matched pairs.","section":"Section 2.3, Eqs. (4)-(5); Section 4.1"},{"comment":"The simulation evidence for finite-sample FWER is limited to one configuration with a strong signal (λ00=2, others 0), and the text itself notes that the trees from Y and |Y| are nearly identical at the upper levels in this configuration, so the simulation is not informative about the level of the subset-specific tests in the problematic small or low-compliance intersections. Please add simulations that report the size of |T_s(λ0)/S_s(λ0)| for sparse and low-compliance intersection hypotheses, or otherwise justify the asymptotic approximation in those settings.","section":"Web Appendix A, Table 1"}],"minor_comments":[{"comment":"The simulation plots show averages over 1000 replications without Monte Carlo error bars or confidence bands; adding them would help interpret claims such as a false positive rate of zero.","section":"Figures 1-2, 7-10"},{"comment":"The legend labels the comparison method as \"BCV-IV\"; this should read \"BCF-IV.\"","section":"Figure 9"},{"comment":"The statement that the honesty property is preserved for \"any supervised machine learning algorithm\" and \"subsequent visual heuristics\" is broader than what is proved; visual heuristics are not formal deterministic algorithms, and their independence from Z should be justified or the sentence qualified.","section":"Section 2.4"},{"comment":"The proof writes \"H0K : λs = λ0\" for the subset null, but the algorithm's nulls are sharp unit-level nulls; aligning the notation would make the closed-testing argument easier to follow.","section":"Web Appendix B"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable fit for the journal and the core idea is novel and worth publishing once the proof gap around the subset-specific test statistics is addressed. I do not see grounds for rejection, but the current form of Proposition 1 overstates the guarantee that is actually established."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my read of 1908.03652. The genuinely new piece is the H-CACE estimand plus Algorithm 1, which wraps the absolute-difference trick from Hsu et al. (2015) around the matching-IV test statistic from Baiocchi and Kang. That combination is not in the cited literature, and the paper is upfront about borrowing the two ingredients. The FWER proof in Web Appendix B is a textbook closed-testing argument and it is correct, conditional on each intersection null being testable at level alpha. The simulations are more thorough than most: varying compliance, compliance heterogeneity, equal-but-opposite effects, and an OHIE-like setting. The OHIE application is plainly described, and the authors themselves flag the exclusion restriction as the assumption most likely to fail. The paper also candidly lists the method's limits, including the sign-collapse problem from using absolute differences.\n\nThe soft spots are real but not fatal. First, the size of the subset-specific test statistic is imported from Baiocchi et al. (2010) and Kang et al. (2016), which prove asymptotic normality for a fixed full matched sample. Nothing in the paper shows the approximation is uniform over the data-dependent, possibly small intersections that CART can produce; the OHIE leaves have pairs in the hundreds with 29% compliance, so the effective signal per leaf is limited. Web Appendix A checks FWER for one simulation configuration, not for sparse or low-compliance subsets. Second, the tree-depth selection in Section 4.3 is described as pruning back until a more interpretable tree was obtained; if that pruning uses the estimated H-CACEs rather than only |Y| and X, then G becomes Z-dependent and the conditional randomization argument breaks. The authors should say exactly what the pruning looked at.\n\nThe central argument holds up. Proposition 1 is sound given the imported level condition; the paper does not fit parameters to the data for inference, and the citation pattern is fair. This is a paper for anyone working on IV with noncompliance who wants to search for effect modification while keeping FWER control. I'd send it to a serious referee: the two concerns above are assessable in revision, and the method is useful enough that an editor should not desk reject it. I'd probably not cite it unless I were working directly on IV effect heterogeneity, but it earns a place in that conversation.","headline":"A useful synthesis of matching-IV and the absolute-difference honesty trick; structurally sound FWER control, but subset-level size is imported rather than proved.","tokens_in":23982,"tokens_out":3906,"would_cite":false,"duration_ms":39238,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Tree-found subgroups can be tested on the same data in IV studies","keywords":["heterogeneous complier average causal effect","instrumental variables","matching","classification and regression trees","closed testing","familywise error rate","Oregon Health Insurance Experiment","effect modification"],"falsifier":"Simulate data with no true effect but with a rare covariate combination that forces the tree to create a leaf with very few pairs, run Algorithm 1 repeatedly, and compare the empirical familywise error rate to $\\alpha$. If the small-leaf statistic is not approximately standard normal, the rejection rate will exceed $\\alpha$, which would falsify Proposition 1 as an unconditional guarantee.","tokens_in":22876,"feed_emoji":"🌳","tokens_out":8402,"duration_ms":84772,"temperature":0.7,"pith_summary":"The paper proposes a way to find and test subgroups with different causal effects in instrumental-variable studies, targeting what it calls the heterogeneous complier average causal effect (H-CACE), the usual complier effect restricted to a covariate-defined subgroup. The method matches units on pre-instrument covariates, uses a regression tree on the absolute value of the adjusted within-pair outcome differences to propose a grouping, and then runs closed testing on every union of the tree's leaves. Its central claim is that this pipeline can use the same data for discovery and inference: conditional on the data and the tree, the chance of at least one false rejection among the tested hypotheses is at most the nominal level. On the Oregon Health Insurance Experiment, the procedure finds that Medicaid's effect on days when health did not impede usual activities is concentrated in older, English-preferring, non-Asian men and younger, English-preferring people with at most a high school diploma or GED.","feed_headline":"Tree-found subgroups can be tested on the same data in IV studies","feed_subtitle":"Matching plus an absolute-difference tree plus closed testing keeps false discoveries at the nominal rate.","key_machinery":"The load-bearing device is the absolute-difference construction: the tree is trained on $|Y_i|$, the absolute value of the adjusted outcome difference within each matched pair, rather than on the signed difference, so the instrument assignment $Z$ is hidden from the tree and the randomized assignment distribution inside matched pairs remains intact. This is combined with the matched-pair IV test statistic $T_s(\\lambda_0)/S_s(\\lambda_0)$ from the matching-IV literature and closed testing over all unions of the tree's leaves; the absolute difference makes every test level $\\alpha$ under the sharp null, and closed testing converts level-$\\alpha$ tests into strong familywise error control. The entity being tested is the H-CACE $\\lambda_s = \\frac{\\sum_{i\\in s}\\sum_j (r^{(1)}_{1sij}-r^{(0)}_{0sij})}{\\sum_{i\\in s}\\sum_j (d_{1sij}-d_{0sij})}$, which extends the usual complier average causal effect to a covariate-defined subgroup.","core_discovery":"The paper's central result is Proposition 1: under the sharp null hypotheses used in Algorithm 1, the conditional probability, given the potential outcomes and covariates $F$, the instrument assignment $Z$, and the tree grouping $G$, that the algorithm makes at least one false rejection is at most $\\alpha$. The mechanism is to feed the regression tree the absolute pairwise difference $|Y_i| = |(Z_{i1}-Z_{i2})(R_{i1}-\\lambda_0 D_{i1} - (R_{i2}-\\lambda_0 D_{i2}))|$ instead of the signed difference, so that the grouping depends only on $F$, not on $Z$; the instrument assignment therefore remains a fair coin within each matched pair, and the randomization-based null distribution used by the matching IV test statistics is preserved. The paper then runs closed testing over all unions of the tree's leaves, rejecting a subgroup hypothesis only when every superset hypothesis is rejected, which gives strong familywise error control. On the OHIE data the procedure reports significant H-CACEs in two subgroups, and simulations show high true discovery rates and low false-positive rates for effect modifiers, especially at low compliance rates.","pith_inferences":["Editorial inference: replacing the single tree with a forest would require each leaf's test to remain level $\\alpha$, so an ensemble version would need either leaf-level calibration or a different multiplicity correction before the Proposition 1 guarantee could be claimed.","Editorial inference: the closed-testing machinery is invertible in $\\lambda_0$, so a natural extension would construct simultaneous confidence sets for all H-CACEs rather than testing a single fixed null value.","Editorial inference: the OHIE tree depth was chosen by pruning back for interpretability, a judgment call that is not demonstrated to be independent of outcome-dependent information; an automatic, outcome-free tuning rule would make the procedure fully reproducible.","Editorial inference: if the cited sharp-null-to-weak-null validity result holds generally, the procedure's error control would extend from sharp hypotheses to the weaker hypotheses $\\lambda_s = \\lambda_0$, which is what many users would care about in practice."],"forward_implications":["Data-driven subgroup discovery and inference can be done on the full sample, so the method avoids the power loss of sample splitting while still controlling the familywise error rate.","Any supervised grouping rule that uses only $X$ and $|Y|$ inherits the honesty property, so the same wrapper can be applied beyond the specific tree algorithm used here.","In the paper's simulations, the method achieves high true discovery rates and near-zero false positive rates for effect modifiers even when compliance is below 40 percent, a regime where IV inference is usually weak.","Subgroups with equal effect magnitudes but opposite signs are invisible to the method, because the absolute-value transformation collapses the direction of the effect.","The OHIE analysis locates Medicaid's effect in older, English-preferring, non-Asian men and younger, English-preferring adults with at most a high school diploma or GED."],"supporting_citations":[{"why":"Supplies the matched-pair IV test statistic $T(\\lambda_0)/S(\\lambda_0)$ whose asymptotic null distribution underlies every subset test in Algorithm 1.","marker":"Baiocchi et al. (2010)"},{"why":"Extends matching-based IV inference and provides the estimator and confidence-interval machinery that the subset tests reuse.","marker":"Kang et al. (2016)"},{"why":"Provides the absolute-difference tree construction and the closed-testing argument showing strong familywise error control for data-discovered subgroups.","marker":"Hsu et al. (2015)"},{"why":"Supplies the closed testing procedure that Algorithm 1 runs over all intersection hypotheses.","marker":"Marcus et al. (1976)"},{"why":"Defines the complier average causal effect and the four compliance types that H-CACE extends.","marker":"Angrist et al. (1996)"},{"why":"Provides the Oregon Health Insurance Experiment data and the lottery-based instrument used in the application.","marker":"Finkelstein et al. (2012)"}],"fun_headline_variants":["Tree that ignores instrument yields valid IV heterogeneity tests","Closed testing on tree leaves for IV effect heterogeneity","New method: find heterogeneous IV effects with interpretable trees","Absolute-difference tree keeps IV tests valid for subgroup discovery","IV heterogeneity via tree-based subgroups with strong error control"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The guarantee rests on each subgroup test statistic $T_s(\\lambda_0)/S_s(\\lambda_0)$ being at least asymptotically level $\\alpha$ under every sharp null visited by closed testing, even for the small, low-compliance leaves a regression tree can create, and on the tree being built from $X$ and $|Y|$ only so that the instrument assignment remains randomized within pairs.","fun_headline_variants_meta":{"raw":{"variants":["Tree that ignores instrument yields valid IV heterogeneity tests","Closed testing on tree leaves for IV effect heterogeneity","New method: find heterogeneous IV effects with interpretable trees","Absolute-difference tree keeps IV tests valid for subgroup discovery","IV heterogeneity via tree-based subgroups with strong error control"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000212,"raw_usage":{"total_tokens":1412,"prompt_tokens":934,"completion_tokens":478,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":550,"completion_tokens_details":{"reasoning_tokens":402}},"tokens_in":550,"tokens_out":478,"duration_ms":5768,"temperature":1.0,"reasoning_tokens":402,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T14:07:52.321839+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Simulate data with no true effect but with a rare covariate combination that forces the tree to create a leaf with very few pairs, run Algorithm 1 repeatedly, and compare the empirical familywise error rate to $\\alpha$. If the small-leaf statistic is not approximately standard normal, the rejection rate will exceed $\\alpha$, which would falsify Proposition 1 as an unconditional guarantee.","supporting_citations":[{"cited_title":"S., et al","cited_arxiv_id":null,"evidence_quote":"Extends matching-based IV inference and provides the estimator and confidence-interval machinery that the subset tests reuse."},{"cited_title":"Y., Zubizarreta, J","cited_arxiv_id":null,"evidence_quote":"Provides the absolute-difference tree construction and the closed-testing argument showing strong familywise error control for data-discovered subgroups."},{"cited_title":"P., Allen, H., Baicker, K., and Group, O","cited_arxiv_id":null,"evidence_quote":"Provides the Oregon Health Insurance Experiment data and the lottery-based instrument used in the application."}],"review_version":1}