{"id":"f0c7492c-d2e1-41ca-8fa1-b321448085ad","arxiv_id":"2506.13107","paper_version":5,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Honest estimation in causal forests can reduce CATE accuracy and require up to 27% more data, especially when effect heterogeneity is strong and detectable.","lead":"A large benchmark study of causal forests finds that the standard 'honest' splitting default, which holds out separate data for estimating treatment effects, often performs worse than using all data adaptively, sometimes requiring 27% more data to match accuracy. The paper interprets honesty as regularization and suggests it should be tuned rather than assumed.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Tuning protocol may be unfair to honest forests: the min-leaf grid and transformed-outcome CV are never validated against oracle tuning, so the AE advantage and the 27% data-cost figure could be a tuning artifact.","rationale":"The paper is a serious empirical study with a clear, falsifiable message, and it has real strengths: 7,500 benchmark datasets, a regret-based selection analysis, an independent large-scale learning-curve check on MegaFon, and an explicit bias-variance decomposition with subcomponents. The qualitative finding that honesty can hurt when heterogeneity is strong and detectable is credible and consistent with prior work cited in Appendix G. However, the central quantitative claim rests on a comparison of two tuned estimators, and the tuning protocol is the least secure link. The paper itself flags the complexity mismatch in Section 4.4 and attempts to address it by leaf matching, but the matching is based on the same unverified m-versus-2m heuristic used in the tuning grids. The one-standard-error rule is biased toward HE, which makes the empirical selector conservative; it does not protect against a tuning bias that could disadvantage HE. The data-cost calculation (27%) is also under-specified and depends on the same tuned models, so it cannot be independently checked from the text. A targeted oracle-tuning validation on a subset of datasets would settle whether the headline gap is intrinsic or an artifact of model selection. Since the concern is concrete and addressable rather than demonstrably fatal, the reader's CONDITIONAL verdict remains appropriate; my read does not change that verdict, but it sharpens the condition that should be imposed.","tokens_in":26108,"tokens_out":5766,"duration_ms":69467,"concrete_test":"On a random subset of 300 ACIC datasets, compute the min_samples_leaf selected by the paper's Section 4.2 protocol for AE and HE. Using the known true CATEs, also compute the oracle-tuned leaf size for each method by minimizing true CATE MSE on a held-out validation split across the same grids. Compare the average absolute log-ratio of selected to oracle leaf size for AE versus HE; then rerun the full S2, regret, and data-cost analysis with oracle-tuned hyperparameters. If the AE advantage and the 27% figure shrink materially (e.g., AE wins in fewer than half of the SNR deciles), the headline claim is a tuning artifact rather than an intrinsic property of honesty.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.2 tunes AE and HE separately by 5-fold cross-validation on transformed-outcome MSE, with HE's min_samples_leaf grid fixed at exactly half the AE grid values (10...320 vs 5...160). The stated rationale is that HE uses half the subsample for splitting, so half the leaf size yields comparable leaf counts. This equivalence is a heuristic; it is not verified by measuring the number of leaves actually produced, and it can fail when the splitting criterion interacts with the data or when CV noise in the transformed-outcome MSE is large. If the CV-selected complexity for HE is systematically farther from the true-CATE-optimal complexity than is the case for AE, then HE is evaluated at a disadvantage that has nothing to do with honesty as such. The Section 4.4 leaf-matched regimes rely on the same 2m/m equivalence, so they do not independently resolve the issue. The empirical selector's one-standard-error rule defaults to HE and therefore biases against AE, so the AE advantage in Table 1 is not explained by the selection rule; the tuning stage itself is the unvalidated link. The 27% data-cost estimate in Section 4.3 is computed from these same tuned models and inherits the concern. Because the paper never validates the transformed-outcome proxy against oracle tuning on the known ground-truth CATEs, the central empirical claim rests on an untested assumption about tuning fairness.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper studies whether the standard practice of honest estimation in causal forests—separating the sample used to build leaves from the sample used to estimate leaf-level effects—improves point prediction of conditional average treatment effects. It contrasts honest estimation (HE) with adaptive estimation (AE), argues that honesty acts as a form of regularization in a bias-variance trade-off, and supports this with a stylized analytical example, a covariance decomposition, and a large empirical study on the ACIC benchmark (7,500 datasets) plus a MegaFon learning-curve analysis. The main empirical claims are that AE is often more accurate, that an empirical selector combining the two dominates both defaults, and that defaulting to honesty can cost up to 27% more data in high-signal regimes. The paper concludes that honesty should be treated as a tunable design choice rather than a universal default.","tokens_in":26449,"tokens_out":9926,"duration_ms":123649,"significance":"If the empirical results are validated, the paper makes a practically important point: the default use of honesty in software such as grf and EconML may be counterproductive for individual-level CATE prediction in high-signal settings. The paper's strengths include a large public semi-synthetic benchmark with ground-truth CATEs, a clean stylized model that isolates selection bias versus approximation bias, a transparent decomposition of forest variance into target coupling, noise overlap, and spillover, and a quantitative falsifiable claim (the 27% data-cost figure). The analytical section is self-contained, and the empirical workflow is reproducible in principle from public data and libraries. The main risk is that the headline numbers rest on an unvalidated tuning protocol; this is addressable and does not undermine the value of the paper if the authors can provide the requested sensitivity analysis.","major_comments":[{"comment":"The tuning protocol is not validated against oracle tuning, and this is load-bearing for the central empirical claims. The HE min_samples_leaf grid is fixed at exactly half the AE grid (10...320 for AE versus 5...160 for HE), and both methods are selected by 5-fold cross-validation on transformed-outcome MSE, Eq. (35). The paper never verifies that the selected leaf counts for HE and AE actually match, nor that the transformed-outcome proxy selects the same model complexity as would be selected on the true CATEs—even though true CATEs are available in the ACIC benchmark. The Section 4.4 leaf-matched regimes use the same assumed 2m/m equivalence, so they do not independently resolve the issue. If the proxy or the grid matching systematically disadvantages HE, the AE advantage in Table 1, the 27% data-cost estimate in Figure 5, and the bias-variance comparisons in Figure 6 could be tuning artifacts rather than intrinsic properties of honesty. The one-standard-error rule defaults to HE and therefore cannot explain AE's advantage, which makes the tuning stage the unvalidated link in the argument. I ask the authors to report the actual leaf-count distributions, validate the transformed-outcome proxy against oracle tuning on the known CATEs, and recompute the headline quantities under both tuning rules.","section":"Section 4.2, Eq. (35); Figures 4–6"},{"comment":"The proposition that the splitting criterion is an unbiased estimator of the MSE reduction is not correct in finite samples. For independent child samples, E[(\\hat{\\tau}_1 - \\hat{\\tau}_2)^2] = (\\tau_1 - \\tau_2)^2 + Var(\\hat{\\tau}_1) + Var(\\hat{\\tau}_2), so the squared difference contains additive variance terms. The random weight n_{\\ell_1} n_{\\ell_2}/(n_{\\ell_1}+n_{\\ell_2})^2 does not remove these terms and introduces additional sampling randomness. Thus Eq. (49) is at best a consistent or heuristic proxy for the MSE gain, not an unbiased estimate as stated. The authors should either correct the proposition, add the additional assumptions under which unbiasedness holds, or reframe the claim as an asymptotic or ranking-equivalent criterion.","section":"Appendix A, Eq. (9)/(49)"}],"minor_comments":[{"comment":"The sentence 'Theory predicts it helps when SNR is low and hurts when it is high' is stronger than the derivations support: Appendix B derives a bias ordering and Appendix C derives component-wise variance effects, but no complete MSE dominance theorem is proved. I suggest softening this to 'the analytical decomposition suggests' or adding a formal statement if intended.","section":"Section 3.4"},{"comment":"The main text says that empirically HE produces lower target coupling than AE even at high SNR, while Figure 2's caption and the surrounding discussion say HE increases target coupling in high-SNR settings. Please clarify that Figure 2 is the theoretical prediction and Figure 7 is the empirical result, and reconcile the wording so the contrast is explicit.","section":"Section 3.3.1 and Figure 2"},{"comment":"The analytical example assumes the split rule \\hat{\\ell} = \\arg\\max_j |\\Delta_j|, which is not the EconML splitting criterion of Eq. (9). The equivalence holds for balanced binary splits, but for the general algorithm it is only heuristic; please state this explicitly when interpreting the empirical results through the theory.","section":"Appendix B, Eq. (64)"},{"comment":"The calculation of the 'extra data honesty needs' is described only verbally. Please specify how the required training size is interpolated or selected when, at a given sample size, neither method significantly outperforms the other at the 5% level, and state whether the 27% figure is a decile mean or a maximum.","section":"Section 4.3, Figure 5"},{"comment":"There are minor typographical issues, including 'ashonest estimation' in Section 1 and the spaced author name 'Y anfang Hou' in the header; these should be corrected in the final version.","section":"Throughout"},{"comment":"The text cites two recent Bachelor's theses (Havelka 2022; Prodan 2025) that already report context-dependent advantages of adaptive versus honest estimation. The novelty statement in the introduction should more clearly position the present paper relative to these works, emphasizing the scale, the 27% quantitative estimate, and the variance decomposition rather than claiming the trade-off itself is new.","section":"Introduction and Appendix H"}],"recommendation":"major_revision","confidential_remarks":"The central empirical claim is credible and the paper is well organized, but the tuning-fairness issue is the key gate: if the authors can show that the transformed-outcome proxy and the half-grid leaf-size equivalence do not systematically disadvantage HE, the paper should be publishable. The Appendix A unbiasedness error is a correctable but real mathematical flaw. I would not reject on novelty grounds, but the authors should more clearly acknowledge the prior empirical evidence in Havelka (2022) and Prodan (2025)."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a careful, readable empirical study that makes a practical case against honesty as an automatic default in causal forests. Second, the centerpiece number — the 27% data cost — is softer than it looks, because the tuning protocol may not treat honest and adaptive forests fairly.\n\nWhat the paper adds: a 7,500-dataset ACIC benchmark with SNR stratification, a bias-variance decomposition (approximation vs. estimation bias, target coupling, noise overlap, spillover), a selection rule with one-standard-error default to honesty, and a MegaFon learning-curve check at 480K observations. The authors also deserve credit for Appendix G, which plainly credits Havelka (2022), Prodan (2025), and Kallus and Mao (2023) with the core qualitative finding. So the paper's incremental value is scale, decomposition, and a specific data-cost number, not the discovery that adaptive can beat honest.\n\nThe soft spots are real but addressable. The tuning grid for HE is exactly half the AE grid (5...160 vs 10...320), justified by a 'comparable leaf counts' heuristic that is never checked against actual leaf counts or oracle tuning. If HE is systematically over-regularized at the selected complexity, the AE advantage and the 27% figure could be partly a tuning artifact. The one-standard-error rule that defaults to HE mitigates some selection bias, but the tuning stage itself is the unvalidated link. The 27% data-cost calculation is also described too briefly in the main text; I had to go hunting for how the 'smallest training size' was chosen. The theory in Appendix B is clean but uses a simplified split rule (max |Delta_j|) that does not match the EconML implementation used in experiments; that is fine for intuition but not a proof about the actual estimator. And there is no code released, which slows verification.\n\nNone of this is fatal. The qualitative pattern (HE better at low SNR, AE better at high SNR) shows up in the leaf-matched regimes and in the MegaFon curves, and it is consistent with prior work. I would not be surprised if the quantitative 27% shrinks once the tuning is pinned down, but the direction seems robust.\n\nWho is this for? Applied researchers using causal forests for targeting and anyone who sets defaults in grf or EconML. It deserves a serious referee. I'd send it out, but I'd ask for code/analysis scripts, a validation of the tuning protocol against oracle-tuned models on a subset of ACIC, and a clearer write-up of the data-cost procedure.","headline":"Useful large-scale confirmation of an existing finding, with a clean bias-variance story and a 27% data-cost number that deserves scrutiny before you repeat it.","tokens_in":26879,"tokens_out":2798,"would_cite":true,"duration_ms":29345,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62D20","62G05","62G08"],"pacs":[],"model":"deepseek-v4-flash","headline":"Honest estimation—the default in causal-forest software—can reduce the accuracy of individual treatment effect estimates by limiting the data used to detect heterogeneity, and the cost can reach 27% more training data.","keywords":["honest estimation","causal forests","conditional average treatment effect","bias-variance trade-off","effect heterogeneity","adaptive estimation","regularization","transformed outcomes"],"falsifier":"Re-run the ACIC comparison with both forests tuned on the benchmark's ground-truth CATEs instead of the transformed-outcome proxy, holding everything else fixed: if honest forests then match or beat adaptive forests, the claimed cost of honesty is an artifact of the tuning protocol, and if the gap persists, the central claim survives its most direct challenge.","tokens_in":25901,"feed_emoji":"🌳","tokens_out":13835,"duration_ms":120123,"temperature":0.7,"pith_summary":"The paper challenges the near-universal default of honest estimation in causal forests—tree-ensemble models that estimate how treatment effects vary across individuals. Honest estimation splits the data so that one sample defines subgroups and a separate, disjoint sample estimates effects within them, and the paper argues this acts as a form of regularization: it lowers overfitting-driven variance but raises bias by halving the data available to detect real treatment-effect heterogeneity. In large, strongly heterogeneous datasets the bias costs more than the variance saves, and across 7,500 ACIC benchmark datasets the authors find that defaulting to honesty can require up to 27% more training data to match the individual-treatment-effect accuracy of adaptive forests that use all data for both tasks. If the paper is right, a safety practice baked into widely used software is better understood as a tunable design choice, with adaptive estimation the more defensible default for point prediction and honesty reserved for tasks that depend on its theoretical guarantees, such as confidence intervals and hypothesis tests.","feed_headline":"Defaulting to 'honest' causal forests can cost 27% more data","feed_subtitle":"On 7,500 benchmarks, the standard honest data-split underfits strong treatment effects, so adaptive forests often win.","key_machinery":"The load-bearing machinery is a bias–variance decomposition of causal forest predictions. Each tree's error splits into approximation error—the gap between the leaf-level subpopulation average treatment effect and the individual's true CATE—and estimation error, which under adaptive estimation carries a selection bias because the same sampling error that makes a split look attractive also inflates its leaf-level estimates. Honesty eliminates estimation bias by construction, because separating the splitting and estimation samples makes leaf assignment and effect estimation independent, while increasing approximation bias because smaller splitting samples detect heterogeneity less well. At the forest level, variance decomposes into target coupling, noise overlap, and target–estimation spillover, and honesty reduces variance by removing the spillover and shrinking noise overlap at the price of higher bias. A stylized single-split example with exactly one informative binary feature shows adaptive estimation selecting the informative feature more often and, despite a positive selection bias, ending with smaller overall bias. The empirical analysis is organized around the signal-to-noise ratio $\\text{SNR} = \\operatorname{Var}(\\beta(X))/\\operatorname{Var}(\\xi)$, the $S^2$ metric $1 - \\text{MSE}_{\\text{te}}/\\operatorname{Var}_{\\text{te}}(\\beta)$, and tuning on transformed outcomes with a one-standard-error selection rule that defaults to honesty.","core_discovery":"The central claim is that honest estimation degrades rather than protects the accuracy of individual treatment effect estimates in causal forests. Because honesty splits the data, trees have less information to detect where effects differ, and the resulting approximation bias—summarizing heterogeneous individuals into poorly aligned leaves—typically outweighs the estimation bias and selection effects that honesty removes, so overall bias is lower under adaptive estimation. Empirically, across 7,004 benchmark datasets with detectable heterogeneity, adaptive forests capture a larger share of explainable effect variance ($S^2$) than honest forests in most signal-to-noise deciles, and the cost of defaulting to honesty grows with signal strength, reaching the equivalent of 27% more data in the strongest-heterogeneity settings. The paper reads honesty as a regularization choice that trades bias for variance: honest forests have lower variance in every regime, but the variance gap closes faster than the bias gap as the signal strengthens, which is why adaptive estimation wins where heterogeneity is detectable. The authors do not declare adaptive estimation universally better: when the signal is weak, honesty can match adaptive performance with roughly 7% less data, but in that regime the signal is often too weak for either forest to model heterogeneity reliably.","pith_inferences":["The paper's own regularization account implies a middle ground it does not explore: the fraction of data reserved for estimation could be tuned continuously, and if a small split captures most of honesty's variance reduction at a fraction of its approximation cost, a partially honest forest would dominate both endpoints on the same benchmarks.","The headline metric is point-prediction accuracy ($S^2$); downstream uses such as budget-constrained targeting or policy learning optimize a decision value rather than an error, and the honest-versus-adaptive ranking could differ under that loss even where $S^2$ favors adaptivity.","The observed crossover between methods is sample-size dependent, so in deployed systems the honest/adaptive decision should be revisited as data accumulates rather than fixed once; the paper's learning-curve analysis illustrates the crossover but leaves this operational implication implicit."],"forward_implications":["Honesty should be treated as a hyperparameter: practitioners should tune over it rather than accept the software default, because its value depends on signal strength, sample size, and the goal of the analysis.","Adaptive estimation is the more defensible default for point prediction of individual treatment effects: it outperforms honesty in most settings, and its advantage grows as effect heterogeneity becomes stronger and easier to detect.","Defaulting to honesty carries a concrete data cost: in the strongest-heterogeneity settings an honest forest needs up to 27% more training data to match an adaptive forest's accuracy, a number that belongs in experimental design conversations.","Honesty keeps a distinct role for uncertainty quantification: its theoretical guarantees support confidence intervals and hypothesis tests, and the paper argues that point estimation and inference need not use the same model—an adaptive forest for estimates and an honest forest for testing can coexist.","The bias–variance reading carries beyond forests: the same tension between separating selection from estimation and preserving flexibility appears in variable selection and meta-learners, and the paper reproduces the pattern with an adaptive-versus-honest Lasso on the same benchmarks."],"supporting_citations":[{"why":"Establishes causal forests as the object of study and supplies the honest-estimation framework whose default use the paper interrogates.","marker":"Wager and Athey (2018)"},{"why":"Introduces honest estimation and the causal-tree splitting criterion; the practice the paper argues against as a default.","marker":"Athey and Imbens (2016)"},{"why":"Provides the 7,500-dataset ACIC benchmark with ground-truth CATEs that anchor the empirical comparison.","marker":"Dorie et al. (2019)"},{"why":"The generalized random forests method behind the widely used package whose honest default is the status quo being challenged.","marker":"Athey et al. (2019)"},{"why":"The implementation used to fit and tune the forests in the experiments.","marker":"Battocchi et al. (2019)"},{"why":"Defines the transformed-outcome proxy on which cross-validation tuning of both methods depends.","marker":"Athey and Imbens (2015)"},{"why":"The heterogeneity test used to filter datasets and to decide whether honesty or adaptivity is selected.","marker":"Imai and Li (2025)"},{"why":"Supplies the one-standard-error model-selection heuristic that defaults the empirical selector toward honesty.","marker":"Hastie et al. (2009)"},{"why":"The randomization-based perspective used to estimate the bias and variance decompositions.","marker":"Athey and Imbens (2017)"}],"fun_headline_variants":["Honesty in causal forests has a 27% data tax","When honesty hurts: causal forests underfit strong effects","Adaptive beats honest causal forests on strong effects","Honesty is a trade-off, not a default for causal forests"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes that tuning both methods on transformed outcomes with a one-standard-error rule that defaults to honesty gives the two approaches fairly matched model complexity; if that tuning proxy systematically disadvantages honest forests, the headline performance gaps and the 27% data-cost estimate could be tuning artifacts rather than properties of honesty itself.","fun_headline_variants_meta":{"raw":{"variants":["Honesty in causal forests has a 27% data tax","When honesty hurts: causal forests underfit strong effects","Adaptive beats honest causal forests on strong effects","Honesty is a trade-off, not a default for causal forests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001001,"raw_usage":{"total_tokens":4255,"prompt_tokens":982,"completion_tokens":3273,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":598,"completion_tokens_details":{"reasoning_tokens":3206}},"tokens_in":598,"tokens_out":3273,"duration_ms":22583,"temperature":1.0,"reasoning_tokens":3206,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:37:11.719932+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the ACIC comparison with both forests tuned on the benchmark's ground-truth CATEs instead of the transformed-outcome proxy, holding everything else fixed: if honest forests then match or beat adaptive forests, the claimed cost of honesty is an artifact of the tuning protocol, and if the gap persists, the central claim survives its most direct challenge.","supporting_citations":[],"review_version":1}