{"id":"fdb65130-421a-4ef7-8bc0-ab8a2708665d","arxiv_id":"1908.09967","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A random forest that reweights training data by an estimated likelihood ratio between training and test covariate distributions improves prediction under covariate shift, with modest and uneven real-data gains.","lead":"The paper trains random forests to emphasize training examples that resemble the data they will predict on, by weighting splits and predictions with an estimated density ratio. Tests on simulations and six hurricanes show this often improves forecasts under distribution shift, though gains are modest and some storms fare worse.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Proposition 1's weighted-OOB consistency guarantee does not cover the implemented ESS-regularized weights, and its proof omits a boundedness condition on the likelihood ratio; the tuning rationale is unsupported as stated.","rationale":"The reader's conditional verdict already flags the fragility of the weight-quality assumption. My stress-test sharpens that concern in two ways: the implemented effective-sample-size regularization directly violates Proposition 1's weight-proportionality hypothesis, and the proof of Proposition 1 contains a missing boundedness condition plus an apparent limit error. These are internal-consistency issues rather than mere disagreements with prior work. The paper does provide honest simulations, code, and a clearly stated method, and the method may still be useful as a heuristic; the conditional verdict is appropriate because the formal guarantee is not currently supported. I therefore do not change the verdict.","tokens_in":22799,"tokens_out":10435,"duration_ms":109928,"concrete_test":"Re-run the Section 4.2 simulation with oracle likelihood-ratio weights (not uLSIF estimates) under two settings: (i) bounded ratio, e.g., P1 = N(0,1), P2 = N(0.5,1); (ii) unbounded ratio, e.g., P1 = N(0,1), P2 = shifted t-distribution with 2 df. In each setting compare (a) lambda = 1 oracle weights, (b) oracle weights raised to lambda selected by the n_eff = 0.75n rule, and (c) unweighted forest, computing weighted OOB error and true P2 RMSE at n = 500, 1000, 2000. If the lambda-regularized OOB error diverges from true P2 risk, or if the unbounded-ratio oracle version fails to converge, the concern is confirmed; if both track P2 risk, the proof gap is harmless in practice.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central tuning claim rests on Proposition 1, which asserts that the weighted OOB error converges to the P2 generalization error. Two problems make this support fail for the implemented method. First, Section 3.2.3 replaces the estimated likelihood-ratio weights w(x) with w(x)^lambda, choosing lambda so that effective sample size equals 0.75n. Table 3 reports lambda values of 0.0084, 0.13, and 0.36. For lambda != 1, w(x)^lambda is not proportional to dP2*/dP1*(x), so Proposition 1's hypothesis w_N(x) = c dP2*/dP1*(x) + epsilon_N(x) is not satisfied by the actual weighting. Second, even granting weight consistency, Appendix B's bound on Var(S1,n) treats L(X_i)(mB_i(X_i)-Y_i)^2 as if its fourth moment under P1 is comparable to the residuals' fourth moment; this requires supx L(x) < infinity or E_{P1}[L^2] < infinity, neither of which is stated. The proof also contains an internal inconsistency: after showing S2,N -> 0 it concludes Var(S2,N) -> 1. Thus the consistency of the weighted OOB metric and the model-selection recommendation built on it are not established for the algorithm that is actually evaluated. The simulations in Figure 4 show the expected consequence: the weighted forest's advantage shrinks and reverses as the shift grows, in exactly the regime where the density ratio is hardest to estimate.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a weighted random forest procedure for prediction under covariate shift. The method estimates the likelihood ratio dP2*/dP1* between the test and training covariate distributions using uLSIF (Kanamori et al., 2009), then uses these weights both in the split-selection criterion and in the prediction stage of each tree, with a weighted quantile regression forest variant for prediction intervals. To stabilize the weights, Section 3.2.3 replaces w(X) with w(X)^lambda so that the effective sample size equals a pre-specified n0, and Section 3.3 introduces a weighted out-of-bag error for model tuning. Proposition 1 claims that this weighted OOB error converges in probability to the P2 generalization error. The paper also proposes a quantile-regression-based missing-data imputation scheme. The method is evaluated on five synthetic data-generating models with varying shift severity, comparing weighted and unweighted forests, and on a hurricane power-outage forecasting application with six held-out storms.","tokens_in":23171,"tokens_out":8962,"duration_ms":91987,"significance":"If the claimed properties hold, the paper makes a practically useful contribution: it extends random forests to covariate shift with a relatively simple modification, and the hurricane outage application is a credible motivating problem. A notable strength is the design of the simulation in Section 4.1, which compares the learned uLSIF weights against oracle weights and thereby isolates the reweighting mechanism from density-ratio estimation error. The paper also provides a theoretical target, Proposition 1, for justifying weighted OOB model selection, and it makes concrete, falsifiable predictions about performance as a function of shift magnitude. However, the theoretical support as currently stated does not cover the regularized weights actually used in the experiments, and the empirical sections lack uncertainty quantification. With those issues addressed, the paper would be a solid contribution to the covariate-shift and random-forest literatures.","major_comments":[{"comment":"Proposition 1 does not apply to the weighting scheme actually implemented and evaluated. The proposition assumes w_N(x)=c dP2*/dP1*(x)+epsilon_N(x), but Section 3.2.3 replaces the estimated weights with w(X_i)^lambda, choosing lambda so that the effective sample size equals n0=0.75n; Table 3 reports lambda values of 0.0084, 0.1305, and 0.3602, for which the weight vector is not proportional to the likelihood ratio. Consequently, the tuning rationale in Section 3.3 -- that minimizing the weighted OOB error should select a good model for P2 -- is not supported by Proposition 1 for the regularized estimator used in the simulations and the hurricane study. Please either extend the consistency result to ESS-regularized weights or explicitly restrict the claim to the un-regularized estimator.","section":"Section 3.2.3, Section 3.3, Appendix B"},{"comment":"The proof of Proposition 1 has gaps that need repair. The variance bound for S_{1,n} requires a boundedness or finite-moment condition on the likelihood ratio, e.g., sup_x dP2*/dP1*(x)<infinity or a finite second moment under P1; the displayed step bounding E_{P1}[L(X)(mB-Y)^4] by max(E_P[W^4], E_P[Z^4]) is not justified as written because the change of measure is not controlled. In addition, after arguing that S_{2,N} -> 0, the text concludes Var(S2,N) -> 1; the limit should be 0. These issues matter because Proposition 1 is the stated theoretical basis for using the weighted OOB error in model selection.","section":"Appendix B"},{"comment":"The simulation results are reported without any measure of variability. Figure 4 and Tables A2-A6 show only point estimates averaged over the 150 runs, so the reader cannot assess whether differences such as Model 1 at lambda=1.5 (weighted RMSE 1.214 vs unweighted 1.404) are meaningful. The same tables also show that the weighted forest loses its advantage at high shift levels in Models 3 and 4, e.g., Table A4 at lambda=1.357 and higher, and the coverage plots show weighted intervals falling below the 0.80 nominal level in that regime. Please add standard errors, intervals, or statistical comparisons, and discuss explicitly the range of shift magnitudes in which the method is beneficial.","section":"Section 4.3, Figure 4, Tables A2-A6"}],"minor_comments":[{"comment":"The notation in Proposition 1 is inconsistent: it samples (X_i,Y_i)_{i=1}^{n+m} but then defines N, n, and m=sum Z_i without clearly relating n and m to N. Please clarify the sample-size notation.","section":"Proposition 1, Section 3.3"},{"comment":"The hurricane results are mixed: the weighted model improves the score for Harvey, Irma, and Matthew, but for Sandy, Nate, and Arthur the weighted model is worse on most reported metrics, including score. The text should state this balance explicitly and discuss possible reasons, since the hurricane application is central to the paper's motivation.","section":"Section 5, Table 3"},{"comment":"The statement that n0 is chosen in (1,n) conflicts with the later choice n0=0.75n; please specify whether n0 is an integer or a proportion and how the search for lambda is implemented.","section":"Section 3.2.3"},{"comment":"Equation (3.6) is hard to parse: the displayed convergence appears to reverse the roles of the estimator and the limiting CDF. Rewriting this step would improve clarity.","section":"Section 3.4, Equation (3.6)"},{"comment":"The lambda column is reported for unweighted rows as well as weighted rows; since unweighted forests do not use lambda, this is confusing and should be clarified in the table caption or by leaving those entries blank.","section":"Table 3"},{"comment":"The caption says \"Fitted vs Predicted\" but the figure appears to plot observed versus predicted values; please correct the wording.","section":"Figure 1 caption"}],"recommendation":"major_revision","confidential_remarks":"For the editor: the manuscript is worth pursuing after revision. The central idea is sensible and the oracle-weight comparison in Section 4.1 is a good design choice, but the main theoretical result currently overclaims relative to the implemented estimator, and the proof needs correction. The empirical sections would be considerably stronger with uncertainty quantification and a more balanced discussion of the hurricane results. I have no concerns about novelty or citation practices."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Tom — quick take on arXiv:1908.09967. The core idea is worth having: instead of retraining a forest for each test point as in case-specific random forests, they estimate the density ratio once via uLSIF and feed those weights into split selection, node prediction, quantile forest weighting, and a weighted OOB error. That combination is new to me, and the paper ships R code and data, so the empirical pieces are checkable. Under moderate covariate shift the weighted forest does reduce RMSE in the simulations, and the oracle-versus-learned weight comparison in Figure 3 is a nice sanity check.\n\nThe soft spots are mostly in the theory. Proposition 1 assumes the estimated weights are proportional to the true likelihood ratio up to a vanishing error term, but the implemented method replaces the weights with w^lambda, choosing lambda so the effective sample size hits 0.75n. For lambda != 1, w^lambda is not proportional to the target ratio, so the theorem's hypothesis isn't satisfied by the algorithm being evaluated. The proof also has an internal slip: after showing S2,N -> 0 it concludes Var(S2,N) -> 1 (surely meant 0), and the variance bound on S1,n implicitly needs a bounded likelihood ratio or a finite P1-moment of L, which is not stated. These are fixable, but as written the tuning rationale is not established.\n\nThe simulations are supportive but the text overstates them. No error bars are shown over the 150 runs, and the advantage shrinks and reverses at larger shift. In the hurricane application, the weighted method improves the score for Irma, Matthew, Nate, and Sandy, but is worse for Harvey and Arthur — and the text says Harvey improves, which Table 3 contradicts. Coverage also tends to be lower for the weighted forest, and that deserves a more careful discussion than a brief dismissal.\n\nWho gets value: anyone working on covariate shift with tree ensembles, and forecasters chasing extreme-event performance. The paper deserves a serious referee. I'd recommend major revision: fix the proposition's hypotheses and proof, add error bars or per-run summaries, and correct the summary of the real-data results. The method itself is sensible and the reproducibility is a real plus.","headline":"A genuinely useful and reproducible weighting scheme for random forests under covariate shift, held back by a consistency proof that doesn't cover the implemented weights and by empirical claims that run ahead of the tables.","tokens_in":23641,"tokens_out":3727,"would_cite":true,"duration_ms":38535,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":["62G08","62G09","62H30"],"pacs":[],"model":"deepseek-v4-flash","headline":"Weighting random-forest splits and predictions by the estimated test-to-training likelihood ratio makes the forest target the shifted test distribution.","keywords":["random forests","importance sampling","covariate shift","likelihood ratio estimation","uLSIF","hurricane power outage forecasting","quantile regression forests","out-of-bag error"],"falsifier":"On data where $P_1$ and $P_2$ are known, compare a forest weighted by the true oracle ratio $dP_2^*/dP_1^*$ with one weighted by the uLSIF estimates across increasing shift strength, for example the Dirichlet simulation with $\\lambda$ from 1 to 1.5. If the oracle-weighted forest tracks the $P_2$ generalization error while the estimated-weight forest diverges from it, the reweighting idea is sound and the fragility is in the weight estimator; if the oracle-weighted forest also fails, the covariate-shift assumption $P(Y\\mid X)$ unchanged is the point of failure.","tokens_in":22619,"feed_emoji":"🌀","tokens_out":15508,"duration_ms":128330,"temperature":0.7,"pith_summary":"Standard supervised learning validates models on a test set drawn from the same distribution as training, but real problems often involve a different target distribution. This paper proposes Locally Optimized Random Forests, which reweight both the split-selection and prediction stages of every tree by an estimate of the likelihood ratio $dP_2^*/dP_1^*$ between the test and training covariate distributions, so the forest behaves as if trained on the target distribution. The motivating application is forecasting county-level power outages during hurricanes, where rare, severe storms are exactly the cases standard validation handles worst. The paper claims the weighted out-of-bag error converges in probability to the generalization error under the test distribution, giving a model-selection criterion that does not need labeled test data. If right, the method offers a data-driven way to adapt a broadly used predictive model to extreme events.","feed_headline":"Reweighted forests adapt to covariate shift without target labels","feed_subtitle":"Splits and predictions are weighted by the estimated test-to-training density ratio, targeting rare severe storms.","key_machinery":"The load-bearing object is the weighted node distribution $\\tilde P_{A,w}$, which places weight $w_i$ on observation $(X_i,Y_i)$ inside node $A$ and renormalizes. All tree computations, the split-quality functional $T_{j,z}(\\tilde P_{A,w})$ and the prediction functional $T_1(\\tilde P_{A^*(x),w})$, are evaluated against this distribution, so importance weighting enters the structure of the tree, not just the bootstrap resampling. The weights come from unconstrained least-squares importance fitting (uLSIF), which models the ratio as a nonnegative linear combination of Gaussian kernels $\\sum_k \\alpha_k K_\\sigma(X, x_k)$, regularized by an $\\ell^1$ penalty and smoothed by selecting a power $\\lambda$ that fixes the effective sample size at a pre-chosen $n_0 = 0.75n$. The effective sample size is $n_{\\mathrm{eff}} = (\\sum_i w_i)^2 / \\sum_i w_i^2$, so fixing it at $n_0$ prevents a few observations from dominating.","core_discovery":"Under the covariate-shift model $P_i(X,Y)=P(Y\\mid X)P_i^*(X)$, the paper's central claim is that replacing the empirical node distribution inside a random forest with the weighted distribution $\\tilde P_{A,w} = \\sum_i w_i \\mathbb{I}(X_i\\in A)\\delta_{(X_i,Y_i)} / \\sum_j w_j \\mathbb{I}(X_j\\in A)$, where $w_i \\propto \\hat{\\ell}(X_i)$ estimates $dP_2^*/dP_1^*$, makes both the split criterion $L(j,z)$ and the terminal-node prediction $T(x;D)$ approximate the target distribution $P_2$ instead of the training distribution $P_1$. The weights are learned from unlabeled test covariates, making the procedure semi-supervised. Proposition 1 states that the weighted out-of-bag error converges in probability to the $P_2$ generalization error, so tuning hyperparameters by minimizing weighted out-of-bag error selects a model for the target distribution. The paper also extends the weighting to quantile regression forests for prediction intervals and supplies a quantile-regression imputation step that preserves the training covariate distribution.","pith_inferences":["Because the weighted node distribution is defined without assuming a particular splitter, the same two-stage recipe could be tested in gradient-boosted trees or other recursive ensemble methods, an extension the paper does not explore.","A practical diagnostic would be to compare the effective sample size after uLSIF with the effective sample size of the oracle weights; when these diverge, the weight estimator is the fragile component rather than the forest.","The proof of Proposition 1 assumes nonnegative responses and bounded fourth moments, so applying the method to zero-inflated or censored outage counts is a natural stress test.","Weighted out-of-bag error could be paired with conformal prediction under covariate shift to produce calibrated intervals for extreme storms, a neighbouring development the paper does not develop."],"forward_implications":["Weighted out-of-bag error can be used to tune $mtry$ and other hyperparameters when the test distribution differs from training, without labeled test data (Proposition 1).","The weighting extends to quantile regression forests, giving prediction intervals that stay near nominal coverage while growing narrower under covariate shift.","In the simulations, the weighted forest improves RMSE and score over an unweighted forest for moderate shifts, with the largest gains in Models 1 and 2.","Gains shrink as the shift grows: both weighted and unweighted forests degrade with increasing $\\lambda$, and the weighted forest can underperform on large shifts.","The quantile-regression imputation procedure generates missing covariates from the full conditional distribution, preserving the training covariate distribution so the density-ratio weights remain meaningful."],"supporting_citations":[{"why":"Defines random forests, the base model whose split and prediction stages the paper reweights.","marker":"(Breiman, 2001)"},{"why":"Introduces the covariate-shift model and weighting the log-likelihood, the framework the paper operates in.","marker":"(Shimodaira, 2000)"},{"why":"Provides the unconstrained least-squares importance fitting (uLSIF) estimator that produces the likelihood-ratio weights.","marker":"(Kanamori et al., 2009)"},{"why":"Supplies quantile regression forests, which the paper adapts to weighted prediction intervals.","marker":"(Meinshausen, 2006)"},{"why":"Contributes importance-weighted cross-validation and the weight-shrinkage idea behind the effective-sample-size regularization.","marker":"(Sugiyama et al., 2007)"},{"why":"Proposes case-specific random forests with pointwise resampling weights, the main contrast for the paper's distribution-level weighting.","marker":"(Xu et al., 2016)"},{"why":"Defines the hurricane outage response and predictor set used in the application.","marker":"(Pasqualini et al., 2017)"},{"why":"Provides the ranger implementation used as the unweighted baseline in simulations.","marker":"(Wright and Ziegler, 2015)"}],"fun_headline_variants":["Forests reweighted for hurricane power outage forecasts","Reweighted forests handle extreme storm outages","Semi-supervised forest adapts to target distribution","Weighted random forests beat covariate shift","Forests learn target density without labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benefit of the method depends on the estimated weights being close to the true likelihood ratio $dP_2^*/dP_1^*$; Proposition 1 assumes the sup-norm estimation error tends to zero, but the paper does not verify this on the hurricane data and its simulations show the weighted forest's advantage erodes and can reverse as the shift grows.","fun_headline_variants_meta":{"raw":{"variants":["Forests reweighted for hurricane power outage forecasts","Reweighted forests handle extreme storm outages","Semi-supervised forest adapts to target distribution","Weighted random forests beat covariate shift","Forests learn target density without labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000669,"raw_usage":{"total_tokens":3074,"prompt_tokens":994,"completion_tokens":2080,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":610,"completion_tokens_details":{"reasoning_tokens":2011}},"tokens_in":610,"tokens_out":2080,"duration_ms":15268,"temperature":1.0,"reasoning_tokens":2011,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T10:57:26.701123+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On data where $P_1$ and $P_2$ are known, compare a forest weighted by the true oracle ratio $dP_2^*/dP_1^*$ with one weighted by the uLSIF estimates across increasing shift strength, for example the Dirichlet simulation with $\\lambda$ from 1 to 1.5. If the oracle-weighted forest tracks the $P_2$ generalization error while the estimated-weight forest diverges from it, the reweighting idea is sound and the fragility is in the weight estimator; if the oracle-weighted forest also fails, the covariate-shift assumption $P(Y\\mid X)$ unchanged is the point of failure.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the unconstrained least-squares importance fitting (uLSIF) estimator that produces the likelihood-ratio weights."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies quantile regression forests, which the paper adapts to weighted prediction intervals."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Contributes importance-weighted cross-validation and the weight-shrinkage idea behind the effective-sample-size regularization."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Defines the hurricane outage response and predictor set used in the application."},{"cited_title":"ranger: A Fast Implementation of Random Forests for High Dimensional Data in C++ and R","cited_arxiv_id":"1508.04409","evidence_quote":"Provides the ranger implementation used as the unweighted baseline in simulations."}],"review_version":1}