{"id":"0b56eb99-0ee6-446c-bee4-d6f3516c1a71","arxiv_id":"2505.05958","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"When incomes are artificially removed in eight different patterns, random forest models predict poverty most accurately in most, but not all, scenarios.","lead":"This paper simulates missing income data in a real survey and checks which of eight statistical and machine learning models best predicts the true poverty rate. The finding that random forest is the most consistent performer, but not always the winner, gives practitioners a benchmark for choosing prediction tools when surveys are incomplete.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's 'true' poverty rates rest on an area-level non-response reweighting (Section 5); if income-dependent non-response persists within areas, all rankings in Tables 5-8 are measured against a biased counterfactual.","rationale":"Read in good faith: the paper is a comparative benchmark, not a claim of universal dominance, and its stated conclusion 'most but not all scenarios' is appropriately hedged. The design is transparent and the tables are detailed. The weakest link is not the model comparison machinery but the construction of the ground truth. The paper needs the reweighted ENNVM sample to be free of income bias for its absolute errors to be meaningful and for rankings to be interpretable as performance against the true poverty rate. That condition is not externally verified; it comes from an area-level non-response correction. Because the paper itself reviews evidence that income non-response is MNAR and tail-heavy, the assumption that area-level weights fully remove bias is not safe without a check. This is an external validity and identification concern, not an internal inconsistency. The reader identified the same assumption, so I agree. The proposed test—recomputing the benchmark under alternative, more flexible corrections—would settle whether the concern is material. Since the verdict was already conditional on this issue, my read does not change it.","tokens_in":25186,"tokens_out":6591,"duration_ms":71835,"concrete_test":"Reconstruct the benchmark two ways from the raw ENNVM 2007 data: (1) use the original surveyed incomes with no non-response correction as truth; (2) estimate response propensity flexibly from all available income predictors (household size, education, occupation, urban/rural, age) rather than area alone, form inverse-probability weights, and recompute true poverty rates at the 5/25/50/75 percentiles. Rerun Table 6 for the eight models under both truths. If rct remains first under MAR-MNAR at the 25% and 50% poverty lines, and rcn remains second overall, the reweighting concern does not change the practical conclusion. If rankings shift materially (for example, pct or ecn overtakes rct), the paper's headline is conditional on an unvalidated correction.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that random forest is the most consistent and accurate predictor—is measured entirely against a counterfactual constructed in Section 5. Observed incomes from the Morocco ENNVM are reweighted by the inverse of unit non-response probabilities estimated from non-response rates by geographical area, and the paper asserts this makes the sample 'statistically free from any income bias.' All poverty rates used as truth in Tables 5-8 are computed from this reweighted sample, so every model error and every ranking inherits the validity of that correction. The load-bearing assumption is that unit non-response is ignorable conditional on geographical area. The literature the paper itself cites (Lillard et al. 1986; Bollinger et al. 2019; Hlasny and Verme 2021) documents that income non-response is typically MNAR and concentrated in the tails, often within strata. If, within a given region, non-response depends on income—for example, richer households are less willing to report income—weights based only on area-level response rates do not remove the bias. The 'true' poverty rates at the 25th and 75th percentile lines, where tail errors matter most, would then be off, and the relative performance of rct/rcn in Table 6 could change. The original non-response rate is below 2%, so the magnitude may be small, but the experiment has no genuinely known counterfactual; it has a reweighted approximation whose accuracy is asserted rather than validated.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a controlled experiment for comparing poverty prediction models under missing income data. Using the 2007 Morocco ENNVM, the author reweights observed incomes by the inverse of estimated area-level unit non-response probabilities to construct a benchmark sample claimed to be free of income bias, and treats poverty rates computed from this sample as the true counterfactual. Incomes are then artificially deleted according to eight missing-data patterns (MCAR shares 5-95%, MARpure, MAR-MNAR, MNARpure), and eight models—OLS, logit, random forest, elastic net, and neural network, each in continuous and categorical variants—are trained on the observed part and used to predict poverty among the missing part. Performance is evaluated with confusion-matrix metrics and predicted-versus-true poverty rates at the 5th, 25th, 50th, and 75th percentile poverty lines, with robustness exercises on model specification, probability cutpoints, error-term adjustment, and grid-search calibration. The central claim is that random forest is the most consistent and accurate model under most but not all scenarios, with the categorical random forest dominating the baseline and the continuous random forest second overall.","tokens_in":25510,"tokens_out":8459,"duration_ms":77112,"significance":"Controlled experiments with known missingness mechanisms are rare in this literature, and the paper's comparative framework—three steps (modeling, prediction, classification) and confusion-matrix objective functions—is a useful contribution. The paper's practical message, if confirmed, is actionable: default to random forest, prefer categorical models on average, avoid simple OLS without error-term correction, and tune ML models only when computational cost can be borne. The strength of the design is that the missing-data mechanism is fully known by construction, all eight models use the same predictors and a common evaluation protocol, and the robustness checks cover poverty lines, specifications, cutpoints, and grid search. The paper does not ship code or replication files, but it names the Stata commands and the public data source.","major_comments":[{"comment":"The validity of every 'true' poverty rate in Tables 5-8 rests wholly on the assertion that inverse-probability reweighting by area-level unit non-response rates yields a sample 'statistically free from any income bias' (Section 5, first paragraph). This is equivalent to assuming unit non-response is ignorable conditional on geographical area. The paper itself cites evidence (Lillard et al. 1986; Bollinger et al. 2019; Hlasny and Verme 2021) that income non-response is typically MNAR and tail-concentrated, often within strata; if non-response depends on income within areas, the reweighting is incomplete, and all model rankings in Tables 5-8 are measured against a biased counterfactual. The reported unit non-response is below 2%, which limits the potential magnitude, but the paper provides no validation of the reweighted benchmark against an external source and no sensitivity analysis with alternative response models. Please add such a validation or a bounding exercise that shows how large within-area income-dependent non-response must be to change the rank ordering of the models.","section":"Section 5"},{"comment":"Each missing-data pattern is realized by a single corruption draw, and the rankings in Tables 6 and 7 are computed from that one draw. The paper thus reports no sampling variation of the corruption process; a different draw with the same pattern and share could rank the models differently, especially for the 5% and 95% missing-data cases where the test sets are small. The t-statistics in Table 5 test the equality of predicted and true poverty rates; they do not provide standard errors for the model rankings. Please replicate the corruption procedure (e.g., 100-1000 draws per scenario), report the distribution of predicted rates and of model ranks, and base the 'most consistent' conclusion on those distributions rather than on a single realization.","section":"Section 6.2 and Table 6"}],"minor_comments":[{"comment":"The notation is inconsistent; 'cWi', 'cβ1', and 'bPi' should be 'W-hat_i', 'beta-hat_1', and 'P-hat_i', and the intercept is missing in Eq. (3).","section":"Section 4.1, Eqs. (3)-(4)"},{"comment":"The column header for the categorical random forest appears as 'rct rct r' and 'rct rctr' in different rows; please correct the header to 'rct' and 'rctr' consistently.","section":"Table 6"},{"comment":"The error-adjusted OLS procedure is only described verbally as 'replicating the same technique used in cross-survey imputations'; please provide the explicit formula or algorithm so that Table 8 can be replicated.","section":"Section 7.2"},{"comment":"The grid-search evaluation reports max, mean, and standard deviation of accuracy, but the text describes one random 50/50 split; please clarify whether the mean and standard deviation are across multiple splits, across grid configurations, or across something else.","section":"Section 7.4"},{"comment":"The variable 'household size<5' is described as 'associated with income' for the MAR-MNAR pattern; please report the actual income difference or correlation to justify the label.","section":"Section 5"},{"comment":"The concluding bullet that 'simple logit models and random forest models should be preferred' is stronger than the baseline in Table 5, where the logit (pct) ranks fourth in accuracy; please qualify the recommendation with the parsimonious-specification and computational-conditioning context.","section":"Section 8"}],"recommendation":"major_revision","confidential_remarks":"The first footnote states that this paper is already published in the World Bank Economic Review (DOI 10.1093/wber/lhae044). If this submission is intended for publication elsewhere, the editor should clarify whether this is consistent with the journal's prior-publication policy. Also, no replication code is provided; given the Stata-command-based analysis, a do-file would materially strengthen the paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper gives applied poverty economists a genuinely useful testbed: it takes a real survey (Morocco ENNVM), corrupts it with known missing-data patterns, and compares eight prediction models against the poverty rate from the uncorrupted data. The central finding—random forest is the most consistent but not universally best—is credible and matches what you'd expect from the broader ML literature. The authors are explicit that no model dominates, which is a point worth taking seriously.\n\nWhat's new is the controlled counterfactual itself. They argue, reasonably, that previous work compared imputation strategies without knowing the true poverty rate. Here, because missingness is artificially imposed, all confusion-matrix metrics can be computed directly against a known target. The framework separating continuous from categorical dependent variable models is clean and reproducible. The robustness checks—poverty lines, missingness patterns, model calibration, probability thresholds—are more thorough than most papers in this space. I particularly appreciate the finding that simple OLS is ineffective unless you add back the error term; that echoes the cross-survey imputation literature and is worth citing.\n\nSoft spots, in proportion. The weakest link is the \"true\" counterfactual itself. The uncorrupted data are not genuinely missing-free; they are a survey with under 2% unit non-response, reweighted by the inverse probability of response estimated from area-level response rates. The paper asserts this makes the sample \"statistically free from any income bias.\" That is the part that needs scrutiny. If income non-response is MNAR within areas—as the paper's own cited literature says it often is—the reweighting is incomplete, and all poverty rates used as truth inherit that bias. At the 25th and 75th percentile lines, where tail errors matter, the ranking of models could shift. The non-response rate is small, so the bias is unlikely to be large, but the paper treats the counterfactual as exact rather than approximate.\n\nSecond, the scenarios are single corruption draws. There are no repeated draws or standard errors around the rankings, so we cannot tell whether the difference between, say, random forest and the runner-up is signal or noise. This is a minor complaint for a benchmark, but it means \"most consistent\" should be read as \"best in this one draw.\"\n\nFinally, no replication package is provided. Since the architecture is the main contribution, code would have made the paper more durable.\n\nWho it's for: practitioners choosing a model when incomes are missing, and researchers designing imputation benchmarks. It deserves a serious referee; even if the counterfactual is imperfect, the experiment is well designed and the results are informative. I'd accept it for peer review and push for sensitivity analysis around the reweighting step, or at minimum a more guarded claim about being free from income bias.","headline":"A useful controlled benchmark for poverty prediction models, but the 'true' counterfactual is a reweighted approximation and the rankings rest on single draws.","tokens_in":25981,"tokens_out":2972,"would_cite":true,"duration_ms":29556,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"In a controlled experiment that deletes incomes from a real survey, random forest models predict poverty more consistently and accurately than OLS, logit, elastic net, and neural network models under most—but not all—missing-data scenarios.","keywords":["poverty prediction","missing data","random forest","imputation","poverty rate","MNAR","machine learning","model comparison"],"falsifier":"Reconstruct true incomes for the full set of households from administrative tax or social-security records, repeat the eight corruption patterns, and rank the eight models against that externally verified poverty rate. If random forest no longer ranks first for the MAR-MNAR pattern at the 5, 25, and 50 percent poverty lines, the headline finding is an artifact of the reweighted benchmark rather than a property of the models.","tokens_in":25002,"feed_emoji":"🌲","tokens_out":9891,"duration_ms":93880,"temperature":0.7,"pith_summary":"The paper aims to answer a practical question: when some household incomes are missing from a survey, which prediction model should be used to estimate poverty? It constructs a benchmark dataset by taking a real household survey and reweighting observed incomes so that the sample is treated as free of income bias, then deletes incomes according to eight patterns that mimic real surveys: five random shares (5 to 95 percent), a pure missing-at-random pattern, a mixed pattern where missingness depends partly on observed traits and partly on income itself, and a pattern where missingness depends on income only. Eight models are scored against the known true poverty rate and the full table of correct and incorrect classifications: ordinary least squares (OLS), logit, random forest, elastic net, and neural networks, each in a continuous-income and a categorical-poverty version. Random forest with a categorical outcome ranks first on most objective functions and on the most common mixed missing-data pattern at most poverty lines, while random forest with a continuous outcome ranks second overall and best matches the aggregate poverty rate. The paper also reports that simple OLS without error-term correction is a poor default, that OLS improves sharply when the model error is added back post-estimation, and that fine-tuned elastic nets and neural networks approach random forest only with expensive grid searches.","feed_headline":"Random forest wins most missing-income poverty tests","feed_subtitle":"Researchers can default to random forest when survey incomes go missing, though no model wins in every setting.","key_machinery":"The load-bearing machinery is a controlled corruption experiment on a reweighted real sample. The observed survey is reweighted by the inverse of estimated non-response probabilities, creating a benchmark that the paper treats as statistically free of income bias; from that benchmark the paper deletes incomes in eight patterns, so the true poverty rate and every household's true status remain known while models see only the corrupted sample. The comparison framework is a three-step procedure—modeling, prediction, classification—that lets continuous income models and categorical poverty models be evaluated on the same four-cell confusion-matrix objective functions. Random forest is a decision-tree ensemble that averages many trees built on random subsamples and random variable subsets; the paper shows that its predicted income distribution tracks the tails of the true distribution better than the other models, and its predicted probabilities concentrate near 0 and 1, which stabilizes the poor/non-poor split across probability cut-points.","core_discovery":"The central claim is that random forest is the most reliable default among the eight poverty-prediction models tested. The categorical random forest model wins nearly all confusion-matrix objective functions—sensitivity, specificity, precision, accuracy, and preference-weighted versions—and it is the top model for the MAR-MNAR scenario that the paper considers the most common and problematic, at poverty lines of 5, 25, and 50 percent; the continuous random forest takes second place overall and first in matching the aggregate poverty rate at the baseline. The paper attributes this to tail behavior: random forest is the only model whose predicted income distribution stays close to the true distribution in both tails, and in categorical form it pushes predicted probabilities toward 0 and 1, leaving few households near the classification cut-point. The paper is explicit that no model dominates: at a 75 percent poverty line random forest wins only about half the missing-data patterns, with very few predictors all models perform alike, and an OLS model corrected post-estimation with the empirical error term performs competitively at the population level.","pith_inferences":["Out-of-survey targeting is the untested extension: if random forest's advantage comes from tail accuracy, proxy-means-test applications, where all incomes are missing, should inherit that advantage, but the paper does not run that experiment.","The reweighted benchmark is not externally verified; comparing the same eight models against administrative income records for the same households would test whether the ranking survives a truth that does not depend on the reweighting model.","Neural network results shift with the order of input variables, which our reading suggests means practitioners should average rankings over variable orderings before choosing a neural network.","The paper's parsimonious-models result implies a decision rule the author does not state: use simple logit or corrected OLS when few predictors are available, and reserve random forest for settings with rich predictor sets."],"forward_implications":["Researchers estimating poverty from surveys with missing incomes can treat random forest as the safest default without needing to know whether the missingness is random or income-driven.","Simple OLS should be avoided unless the model error term is corrected post-estimation; corrected OLS becomes competitive for the population poverty rate but cannot support household-level classification metrics.","Categorical dependent-variable models tend to beat continuous models on average, but a continuous random forest remains the best choice for matching the aggregate poverty rate.","Fine-tuned elastic nets and neural networks can approach random forest accuracy, but only with full grid searches that cost hours of computing time, so the practical edge of random forest is its default-specification performance.","No single model is guaranteed to win: the best model can change with the poverty line, the missing-data pattern, the objective function, and the number of predictors available."],"supporting_citations":[{"why":"supplies the MCAR/MAR/MNAR taxonomy that defines the eight missing-data patterns the experiment corrupts the data with.","marker":"Rubin and Little 2020"},{"why":"provides the model-based reweighting approach used to turn the observed survey into the bias-free benchmark sample.","marker":"Rosenbaum 1987"},{"why":"supplies the inverse-probability weighting adjustment used when constructing the counterfactual poverty rate.","marker":"Kim and Kim 2007"},{"why":"shows how unit non-response probabilities can be estimated and corrected, motivating the reweighting strategy and the MNAR emphasis.","marker":"Korinek et al. 2007"},{"why":"documents that nonresponse rises with income, the empirical basis for treating missing incomes as MNAR and concentrating on tail errors.","marker":"Lillard et al. 1986"},{"why":"documents earnings nonresponse concentrated in the tails, which the paper uses to justify tail-sensitive evaluation and the MAR-MNAR scenario.","marker":"Bollinger et al. 2019"},{"why":"introduces the model-error correction for OLS predicted values that Section 7.2 applies to make the welfare model competitive.","marker":"Elbers et al. 2003"},{"why":"summarizes the cross-survey imputation error-correction method that the paper contrasts with machine learning in robustness tests.","marker":"Dang et al. 2019"}],"fun_headline_variants":["Best bet for missing incomes? Random forest","Random forest wins most poverty predictions","Missing income data? Random forest is safest","Random forest beats rivals, but not always"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The benchmark 'true' poverty rate is built by reweighting observed incomes with estimated inverse non-response probabilities; if that correction misses any income bias, every model error and every ranking in the paper is measured against a distorted truth.","fun_headline_variants_meta":{"raw":{"variants":["Best bet for missing incomes? Random forest","Random forest wins most poverty predictions","Missing income data? Random forest is safest","Random forest beats rivals, but not always"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000613,"raw_usage":{"total_tokens":2802,"prompt_tokens":847,"completion_tokens":1955,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":463,"completion_tokens_details":{"reasoning_tokens":1902}},"tokens_in":463,"tokens_out":1955,"duration_ms":13487,"temperature":1.0,"reasoning_tokens":1902,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T22:52:01.084004+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Reconstruct true incomes for the full set of households from administrative tax or social-security records, repeat the eight corruption patterns, and rank the eight models against that externally verified poverty rate. If random forest no longer ranks first for the MAR-MNAR pattern at the 5, 25, and 50 percent poverty lines, the headline finding is an artifact of the reweighted benchmark rather than a property of the models.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the MCAR/MAR/MNAR taxonomy that defines the eight missing-data patterns the experiment corrupts the data with."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"provides the model-based reweighting approach used to turn the observed survey into the bias-free benchmark sample."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"supplies the inverse-probability weighting adjustment used when constructing the counterfactual poverty rate."},{"cited_title":"Mistiaen, and M","cited_arxiv_id":null,"evidence_quote":"shows how unit non-response probabilities can be estimated and corrected, motivating the reweighting strategy and the MNAR emphasis."},{"cited_title":"Smith, and F","cited_arxiv_id":null,"evidence_quote":"documents that nonresponse rises with income, the empirical basis for treating missing incomes as MNAR and concentrating on tail errors."},{"cited_title":"Hirsch, C","cited_arxiv_id":null,"evidence_quote":"documents earnings nonresponse concentrated in the tails, which the paper uses to justify tail-sensitive evaluation and the MAR-MNAR scenario."},{"cited_title":"Lanjouw, and P","cited_arxiv_id":null,"evidence_quote":"introduces the model-error correction for OLS predicted values that Section 7.2 applies to make the welfare model competitive."},{"cited_title":"Jolliffe, and C","cited_arxiv_id":null,"evidence_quote":"summarizes the cross-survey imputation error-correction method that the paper contrasts with machine learning in robustness tests."}],"review_version":1}