REVIEW 2 major objections 6 minor 47 references
Predicting Poverty
T0 review · 2 major / 6 minor · reviewed 2026-08-15 · deepseek-v4-flash
Pith's one-line read In a controlled experiment that deletes incomes from a real survey, random forest models predict poverty more consistently and accurately than OLS, logit, elastic net, and neural network models under most—but not all—missing-data scenarios.
desk verdict A useful controlled benchmark for poverty prediction models, but the 'true' counterfactual is a reweighted approximation and the rankings rest on single draws. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a controlled corruption experiment on a reweighted real sample. The observed survey is reweighted by the inverse of estimated non-response probabilities, creating a benchmark that the paper treats as statistically free of income bias; from that benchmark the paper deletes incomes in eight patterns, so the true poverty rate and every household's true status remain known while models see only the corrupted sample. The comparison framework is a three-step procedure—modeling, prediction, classification—that lets continuous income models and categorical poverty models be evaluated on the same four-cell confusion-matrix objective functions. Random forest is a decision-tree ensemble that averages many trees built on random subsamples and random variable subsets; the paper shows that its predicted income distribution tracks the tails of the true distribution better than the other models, and its predicted probabilities concentrate near 0 and 1, which stabilizes the poor/non-poor split across probability cut-points.
What would settle it
Reconstruct true incomes for the full set of households from administrative tax or social-security records, repeat the eight corruption patterns, and rank the eight models against that externally verified poverty rate. If random forest no longer ranks first for the MAR-MNAR pattern at the 5, 25, and 50 percent poverty lines, the headline finding is an artifact of the reweighted benchmark rather than a property of the models.
Extended reading notes
Core claim
The central claim is that random forest is the most reliable default among the eight poverty-prediction models tested. The categorical random forest model wins nearly all confusion-matrix objective functions—sensitivity, specificity, precision, accuracy, and preference-weighted versions—and it is the top model for the MAR-MNAR scenario that the paper considers the most common and problematic, at poverty lines of 5, 25, and 50 percent; the continuous random forest takes second place overall and first in matching the aggregate poverty rate at the baseline. The paper attributes this to tail behavior: random forest is the only model whose predicted income distribution stays close to the true distribution in both tails, and in categorical form it pushes predicted probabilities toward 0 and 1, leaving few households near the classification cut-point. The paper is explicit that no model dominates: at a 75 percent poverty line random forest wins only about half the missing-data patterns, with very few predictors all models perform alike, and an OLS model corrected post-estimation with the empirical error term performs competitively at the population level.
Load-bearing premise
The benchmark 'true' poverty rate is built by reweighting observed incomes with estimated inverse non-response probabilities; if that correction misses any income bias, every model error and every ranking in the paper is measured against a distorted truth.
Editorial extensions
If this is right
- Researchers estimating poverty from surveys with missing incomes can treat random forest as the safest default without needing to know whether the missingness is random or income-driven.
- Simple OLS should be avoided unless the model error term is corrected post-estimation; corrected OLS becomes competitive for the population poverty rate but cannot support household-level classification metrics.
- Categorical dependent-variable models tend to beat continuous models on average, but a continuous random forest remains the best choice for matching the aggregate poverty rate.
- Fine-tuned elastic nets and neural networks can approach random forest accuracy, but only with full grid searches that cost hours of computing time, so the practical edge of random forest is its default-specification performance.
- No single model is guaranteed to win: the best model can change with the poverty line, the missing-data pattern, the objective function, and the number of predictors available.
Reading between the lines
- Out-of-survey targeting is the untested extension: if random forest's advantage comes from tail accuracy, proxy-means-test applications, where all incomes are missing, should inherit that advantage, but the paper does not run that experiment.
- The reweighted benchmark is not externally verified; comparing the same eight models against administrative income records for the same households would test whether the ranking survives a truth that does not depend on the reweighting model.
- Neural network results shift with the order of input variables, which our reading suggests means practitioners should average rankings over variable orderings before choosing a neural network.
- The paper's parsimonious-models result implies a decision rule the author does not state: use simple logit or corrected OLS when few predictors are available, and reserve random forest for settings with rich predictor sets.
Signed reviews
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a controlled experiment for comparing poverty prediction models under missing income data. Using the 2007 Morocco ENNVM, the author reweights observed incomes by the inverse of estimated area-level unit non-response probabilities to construct a benchmark sample claimed to be free of income bias, and treats poverty rates computed from this sample as the true counterfactual. Incomes are then artificially deleted according to eight missing-data patterns (MCAR shares 5-95%, MARpure, MAR-MNAR, MNARpure), and eight models—OLS, logit, random forest, elastic net, and neural network, each in continuous and categorical variants—are trained on the observed part and used to predict poverty among the missing part. Performance is evaluated with confusion-matrix metrics and predicted-versus-true poverty rates at the 5th, 25th, 50th, and 75th percentile poverty lines, with robustness exercises on model specification, probability cutpoints, error-term adjustment, and grid-search calibration. The central claim is that random forest is the most consistent and accurate model under most but not all scenarios, with the categorical random forest dominating the baseline and the continuous random forest second overall.
Significance. Controlled experiments with known missingness mechanisms are rare in this literature, and the paper's comparative framework—three steps (modeling, prediction, classification) and confusion-matrix objective functions—is a useful contribution. The paper's practical message, if confirmed, is actionable: default to random forest, prefer categorical models on average, avoid simple OLS without error-term correction, and tune ML models only when computational cost can be borne. The strength of the design is that the missing-data mechanism is fully known by construction, all eight models use the same predictors and a common evaluation protocol, and the robustness checks cover poverty lines, specifications, cutpoints, and grid search. The paper does not ship code or replication files, but it names the Stata commands and the public data source.
major comments (2)
- [Section 5] The validity of every 'true' poverty rate in Tables 5-8 rests wholly on the assertion that inverse-probability reweighting by area-level unit non-response rates yields a sample 'statistically free from any income bias' (Section 5, first paragraph). This is equivalent to assuming unit non-response is ignorable conditional on geographical area. The paper itself cites evidence (Lillard et al. 1986; Bollinger et al. 2019; Hlasny and Verme 2021) that income non-response is typically MNAR and tail-concentrated, often within strata; if non-response depends on income within areas, the reweighting is incomplete, and all model rankings in Tables 5-8 are measured against a biased counterfactual. The reported unit non-response is below 2%, which limits the potential magnitude, but the paper provides no validation of the reweighted benchmark against an external source and no sensitivity analysis with alternative response models. Please add such a validation or a bounding exercise that shows how large within-area income-dependent non-response must be to change the rank ordering of the models.
- [Section 6.2 and Table 6] Each missing-data pattern is realized by a single corruption draw, and the rankings in Tables 6 and 7 are computed from that one draw. The paper thus reports no sampling variation of the corruption process; a different draw with the same pattern and share could rank the models differently, especially for the 5% and 95% missing-data cases where the test sets are small. The t-statistics in Table 5 test the equality of predicted and true poverty rates; they do not provide standard errors for the model rankings. Please replicate the corruption procedure (e.g., 100-1000 draws per scenario), report the distribution of predicted rates and of model ranks, and base the 'most consistent' conclusion on those distributions rather than on a single realization.
minor comments (6)
- [Section 4.1, Eqs. (3)-(4)] The notation is inconsistent; 'cWi', 'cβ1', and 'bPi' should be 'W-hat_i', 'beta-hat_1', and 'P-hat_i', and the intercept is missing in Eq. (3).
- [Table 6] The column header for the categorical random forest appears as 'rct rct r' and 'rct rctr' in different rows; please correct the header to 'rct' and 'rctr' consistently.
- [Section 7.2] The error-adjusted OLS procedure is only described verbally as 'replicating the same technique used in cross-survey imputations'; please provide the explicit formula or algorithm so that Table 8 can be replicated.
- [Section 7.4] The grid-search evaluation reports max, mean, and standard deviation of accuracy, but the text describes one random 50/50 split; please clarify whether the mean and standard deviation are across multiple splits, across grid configurations, or across something else.
- [Section 5] The variable 'household size<5' is described as 'associated with income' for the MAR-MNAR pattern; please report the actual income difference or correlation to justify the label.
- [Section 8] The concluding bullet that 'simple logit models and random forest models should be preferred' is stronger than the baseline in Table 5, where the logit (pct) ranks fourth in accuracy; please qualify the recommendation with the parsimonious-specification and computational-conditioning context.
Circularity Check
No significant circularity: the model comparison is benchmarked against a counterfactual constructed before and independently of the prediction models.
full rationale
The paper's central comparison is self-contained and non-circular. In Section 5, the 'true' counterfactual poverty rates are constructed by reweighting observed Morocco ENNVM incomes by the inverse of area-level unit non-response probabilities, and this reweighted sample is fixed before any of the eight prediction models are estimated. The models in Section 6 are then fitted to artificially corrupted versions of that same sample, and their predictions are compared against the pre-existing reweighted counterfactual. No prediction model contributes to the construction of the benchmark; the ranking is therefore not fitted into existence. The same-author citations (e.g., Hlasny and Verme 2018a, 2018b, 2021; Dang and Verme 2022; Verme and Gigliarano 2019) are used as background literature on non-response patterns and measurement methods, not as an imported theorem that forces the paper's conclusion. The reweighting technique itself is attributed to Rosenbaum (1987), Kim and Kim (2007), and Korinek et al. (2006, 2007), which are independent sources with stated assumptions. The statement that the reweighted sample is 'expected to be statistically free from any income bias' is an assumption about the quality of the counterfactual; it is not a step in which a prediction target is defined in terms of the models being tested. Whether area-level weights fully correct income-dependent non-response within areas is an external validity concern, not evidence of circularity. Even the baseline design choice of setting the poverty line at the median, which makes the true poverty rate 50% by construction, does not make the prediction exercise circular: the models' predicted poverty rates range from 43.09 to 50.48 in Table 5, so the task retains real discriminatory content. Overall, no equation or fitted parameter is reused as its own prediction target, and no load-bearing argument reduces to a self-citation.
Assumptions & free parameters
free parameters (3)
- Baseline ML hyperparameters =
RF: iterations=100, depth=0, mtry=sqrt(p); EN: alpha=0; NN: two hidden layers of 100 neurons, learning rate 0.1, no…
- Default probability cutpoint for categorical models =
0.5
- Poverty lines =
5th, 25th, 50th, 75th percentiles of true income
assumptions (5)
- domain assumption Re-weighting by inverse response probabilities removes non-response bias from the survey.
- domain assumption The eight corruption patterns span the missing-data mechanisms relevant to income surveys.
- domain assumption The covariate set is fully observed and sufficient for prediction.
- domain assumption Stata implementations of rforest, elasticnet and mlp2 correctly instantiate the model families.
- domain assumption Ignoring sampling weights and adult-equivalence scales does not distort relative model performance.
Cite this review
Pith. "Pith review of Predicting Poverty." pith.science (2026). https://pith.science/paper/P3IQKPIT
@misc{pith2026250505958,
author = {Pith},
title = {Pith review of: Predicting Poverty},
year = {2026},
howpublished = {\url{https://pith.science/paper/P3IQKPIT}},
note = {Machine review of arXiv:2505.05958}
}
read the original abstract
Poverty prediction models are used to address missing data issues in a variety of contexts such as poverty profiling, targeting with proxy-means tests, cross-survey imputations such as poverty mapping, top and bottom incomes studies, or vulnerability analyses. Based on the models used by this literature, this paper conducts a study by artificially corrupting data clear of missing incomes with different patterns and shares of missing incomes. It then compares the capacity of classic econometric and machine learning models to predict poverty under different scenarios with full information on observed and unobserved incomes, and the true counterfactual poverty rate. Random forest provides more consistent and accurate predictions under most but not all scenarios.
Figures
Reference graph
Works this paper leans on
-
[1]
Abelson, B., k. R. Varshney, and J. Sun (2014). Targeting direct cash transfers to the extremely poor. Proceedings of the 20th ACM SIGKDD international conference on knowledge discovery and data mining\/ , 1563--1572
work page 2014
- [2]
-
[3]
Aiken, E. L., G. Bedoya, J. E. Blumenstock, and A. Coville (2023). Program targeting with machine learning and mobile phone data: Evidence from an anti-poverty intervention in afghanistan. Journal of Development Economics\/ 161 , 103016
work page 2023
-
[4]
Andree, B. (2021). Estimating food price inflation from partial surveys. World Bank Policy Research Working Paper\/ 9886
work page 2021
-
[5]
Andree, B., A. Chamorro, A. Kraay, P. Spencer, and D. Wang (2020). Predicting food crises. World Bank Policy Research Working Paper\/ 9412
work page 2020
-
[6]
Atkinson, A., T. Piketty, and E. Saez (2011). Top incomes in the long run of history. Journal of Economic Literature\/ 49 , 3--71
work page 2011
-
[7]
Baker, J. L. and M. E. Grosh (1994). Poverty reduction through geographic targeting: How well does it work? World Development\/ 22\/ (7), 983--995
work page 1994
-
[8]
Blattman, C., N. Fiala, and S. Martinez (2013, 12). Generating Skilled Self-Employment in Developing Countries: Experimental Evidence from Uganda * . The Quarterly Journal of Economics\/ 129\/ (2), 697--752
work page 2013
Show all 47 references
-
[9]
Cadamuro, and R
Blumenstock, J., G. Cadamuro, and R. On (2015). Predicting poverty and wealth from mobile phone metadata. Science\/ 350\/ (626), 1073--1076
2015
-
[10]
Hirsch, C
Bollinger, C., B. Hirsch, C. M. Hokayem, and J. Ziliak (2019). Trouble in the tails? what we know about earnings nonresponse 30 years after lillard, smith, and welch. Journal of Political Economy\/ 127\/ (5), 2143 -- 2185
2019
-
[11]
Ravallion, and D
Brown, C., M. Ravallion, and D. van de Walle (2018). A poor means test? econometric targeting in africa. Journal of Development Economics\/ 134\/ (C), 109--124
2018
-
[12]
Burkhauser, R. V., K. Corinth, J. Elwell, and J. Larrimore (2023). Evaluating the success of the war on poverty since 1963 using an absolute full-income poverty measure. Journal of Political Economy\/ 0\/ (ja), null
2023
-
[13]
Calvo, C. and S. Dercon (2013). Vulnerability to individual and aggregate poverty. Social Choice and Welfare\/ 41\/ (4), 721--740
2013
-
[14]
Chen, S. and M. Ravallion (2010, 11). The Developing World is Poorer than We Thought, But No Less Successful in the Fight Against Poverty* . The Quarterly Journal of Economics\/ 125\/ (4), 1577--1625
2010
-
[15]
Grosh, and J
Coady, D., M. Grosh, and J. Hoddinott (2004). Targeting of Transfers in Developing Countries: Review of Lessons and Experience . The World Bank
2004
-
[16]
Cowell, F. and M. Victoria-Feser (1996a). Poverty measurement with contaminated data: A robust approach. European Economic Review\/ 40 , 1761--1771
1996
-
[17]
and M.-P
Cowell, F. and M.-P. Victoria-Feser (1996b). Robustness properties of inequality measures. Econometrica\/ 64 , 77--101
1996
-
[18]
D'Alessio, G. and I. Faiella (2002). Non-response behaviour in the bank of italy’s survey of household income and wealth. Banca D'Italia: Temi di discussione\/ (462)
2002
-
[19]
D'Alessio, G. and A. Neri (2015). Income and wealth sample estimates consistent with macro aggregates: some experiments. Banca D'Italia: Questioni di Economia e Finanza, Occasional Papers\/ (272)
2015
-
[20]
Jolliffe, and C
Dang, H., D. Jolliffe, and C. Carletto (2019, July). Data Gaps, Data Incomparability, And Data Imputation: A Review Of Poverty Measurement Methods For Data‐Scarce Environments . Journal of Economic Surveys\/ 33\/ (3), 757--797
2019
-
[21]
Dang, H.-A. and P. Verme (2022). Estimating poverty for refugee populations: Can cross-survey imputation methods substitute for data scarcity? Journal of Population Economics\/ (forthcoming)
2022
-
[22]
(2005, 05)
Deaton, A. (2005, 05). Measuring Poverty in a Growing World (or Measuring Growth in a Poor World) . The Review of Economics and Statistics\/ 87\/ (2), 395--395
2005
-
[23]
Ezzrari, R
Douidich, M., A. Ezzrari, R. van der Weide, and P. Verme (2016). Estimating quarterly poverty rates using labor force surveys: A primer. World Bank Economic Review\/ 30\/ (3), 475--500
2016
-
[24]
Lanjouw, and P
Elbers, C., J. Lanjouw, and P. Lanjouw (2003). Micro--level estimation of poverty and inequality. Econometrica\/ 71\/ (1), 355--364
2003
-
[25]
Glewwe, P. (1991). Investigating the determinants of household welfare in cote d'ivoire. Journal of Development Economics\/ 35\/ (2), 307--337
1991
-
[26]
Ceriani, and P
Hlasny, V., L. Ceriani, and P. Verme (2021). Bottom incomes and the measurement of poverty and inequality. Review of Income and Wealth\/ doi.org/10.1111/roiw.12535
2021 doi
-
[27]
Hlasny, V. and P. . Verme (2018a). Top incomes and the measurement of inequality in egypt. World Bank Economic Review\/ 32\/ (32), 428--455
2018
-
[28]
Hlasny, V. and P. . Verme (2021). The impact of top incomes biases on the measurement of inequality in the united states. Oxford Bulletin of Economics and Statistics\/ (https://doi.org/10.1111/obes.12472)
2021 doi
-
[29]
Hlasny, V. and P. Verme (2018b). Top incomes and inequality measurement: A comparative analysis of correction methods using the eu silc data. Econometrics\/ 6\/ (2), 1--21
2018
-
[30]
Burke, M
Jean, N., M. Burke, M. Xie, M. Davis, D. B. Lobell, and S. Ermon (2016). Combining satellite imagery and machine learning to predict poverty. Science\/ 353\/ (6301), 790--794
2016
-
[31]
Jenkins, S. (2017). Pareto models, top incomes and recent trends in uk income inequality. Economica\/ 84\/ (334), 261--289
2017
-
[32]
Kim, J. K. and J. J. Kim (2007). Nonresponse weighting adjustment using estimated response probability. The Canadian Journal of Statistics / La Revue Canadienne de Statistique\/ 35\/ (4), 501--514
2007
-
[33]
Mistiaen, and M
Korinek, A., J. Mistiaen, and M. Ravallion (2006). Survey nonresponse and the distribution of income. The Journal of Economic Inequality\/ 4\/ (1), 33--55
2006
-
[34]
Mistiaen, and M
Korinek, A., J. Mistiaen, and M. Ravallion (2007). An econometric method of correcting for unit nonresponse bias in surveys. Journal of Econometrics\/ 136\/ (1), 213--235
2007
-
[35]
Smith, and F
Lillard, L., J. Smith, and F. Welch (1986). What do we really know about wages? the importance of nonreporting and census imputation. Journal of Political Economy\/ 94\/ (3), 489--506
1986
-
[36]
Mcbride, L. and A. Nichols (2018). Retooling poverty targeting using out-of-sample validation and machine learnin. World Bank Economic Review\/ 32\/ (3), 531--550
2018
-
[37]
Stinson, and E
Moore, J., L. Stinson, and E. Welniak (1997). Income measurement error in surveys: A review. Census Working Papers\/ (SM97-05)
1997
-
[38]
Morduch, J. (1994). Poverty and vulnerability. American Economic Review\/ 84\/ (2), 221--25
1994
-
[39]
Piketty, T. and E. Saez (2003, 02). Income Inequality in the United States, 1913–1998* . The Quarterly Journal of Economics\/ 118\/ (1), 1--41
2003
-
[40]
Rosenbaum, P. R. (1987). Model-based direct adjustment. Journal of the American Statistical Association\/ 82\/ (398), 387--394
1987
-
[41]
Rubin, D. B. (1976). Inference and missing data. Biometrika\/ 63\/ (3), 581--592
1976
-
[42]
Rubin, D. B. and R. J. A. Little (2020). Statistical Analysis with Missing Data, 3rd Edition . John Wiley and Sons
2020
-
[43]
(2006, 05)
Sala-i Martin, X. (2006, 05). The World Distribution of Income: Falling Poverty and … Convergence, Period* . The Quarterly Journal of Economics\/ 121\/ (2), 351--397
2006
-
[44]
Tarozzi, A. and A. Deaton (2009). Using census and survey data to estimate poverty and inequality for small areas. The Review of Economics and Statistics\/ 91\/ (4), 773--792
2009
-
[45]
Verme, P. and C. Gigliarano (2019). Optimal targeting under budget constraints in a humanitarian context. World Development\/ (119)
2019
-
[46]
Gigliarano, C
Verme, P., C. Gigliarano, C. Wieser, K. Hedlund, M. Petzoldt, and M. Santacroce (2016). The Welfare of Syrian Refugees: Evidence from Jordan and Lebanon . Washington DC: World Bank
2016
-
[47]
Wodon, Q. (1997). Targeting the poor using roc curves. World Development\/ 25\/ (12), 2083--2092
1997
Reviewed August 15, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.