REVIEW 5 major objections 5 minor 28 references
In a synthetic country-level dataset, PM2.5 is the dominant predictor of respiratory disease rates and air-quality classification; removing it reveals how accuracy alone can mislead.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-01 19:13 UTC pith:3ZLQ7F7T
load-bearing objection A transparent, well-structured methodological demo on a synthetic dataset; the PM2.5-dominance finding is likely baked into the generator, and without generator code the paper can't be independently verified. the 5 major comments →
Interpretable Machine Learning for Air Pollution and Respiratory Health Prediction: A Socioeconomic Subgroup Analysis
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
On its own terms, the paper establishes that, in the Global Climate-Health Impact Tracker dataset—a synthetic collection of 14,100 country-week records—PM2.5 concentration is the dominant predictor of respiratory disease rate in a trained XGBoost model, and that a binary air-quality classification task derived from AQI achieves high balanced accuracy (~0.93) when PM2.5 is included but falls to ~0.82 when it is removed. The paper uses nested cross-validation to compare nine regression and nine classification models, and SHAP values to attribute predictions. It also reports that subgroup SHAP analysis shows PM2.5 contributes more strongly to predictions for lower-middle-income countries. The a
What carries the argument
The central mechanism is the target-proxy relationship between PM2.5 and the AQI-derived air-quality label, combined with SHAP (Shapley Additive exPlanations—a game-theoretic tool that quantifies each feature's contribution to individual predictions) as the attribution framework. The paper uses SHAP to rank feature importance, a 'remove PM2.5' sensitivity test to expose the model's dependence on that proxy, and income-level SHAP comparisons to detect subgroup differences. The nested cross-validation (outer 10-fold for evaluation, inner 5-fold for hyperparameter tuning) provides the evaluation backbone that separates model selection from performance estimation.
Load-bearing premise
The load-bearing premise is that the synthetic Global Climate-Health Impact Tracker dataset embeds realistic climate-health structure; if the data generator built the respiratory-rate target directly from PM2.5, the dominant-predictor finding is a simulation artifact rather than a discovery.
What would settle it
Obtain the dataset's generation script (documented as synthetic) and check the target construction: if respiratory disease rate is a deterministic function that includes PM2.5 (e.g., rate = c + a·PM2.5 + noise), the central finding is a tautology. Alternatively, run the identical workflow on real, traceable country-level PM2.5 and respiratory-health data with leave-one-country-out validation and see whether PM2.5 remains dominant.
If this is right
- Within this dataset, linear and regularized linear models match or beat complex ensembles for respiratory disease prediction, so a simpler model suffices.
- Air-quality classification accuracy drops from about 0.93 to about 0.82 when PM2.5 is removed, showing the task leans on a target-proxy feature.
- Aggregate regression error is similar across income groups (RMSE ~9.7–10.1), but SHAP attribution for PM2.5 is roughly 60% higher in lower-middle-income countries, revealing group-level differences in model behavior.
- Reducing features from 17 to 12 based on SHAP rankings changes balanced accuracy by only about 0.0002, indicating low-ranked predictors are nearly redundant.
Where Pith is reading between the lines
- The same target-proxy audit could apply to any classification target derived from an environmental index (e.g., heat alerts, water-quality alerts), where the index's own components may leak into features.
- If the generator built the target from PM2.5, the 'dominant predictor' result is expected; the paper's transferable value is the workflow, not the specific finding.
- The subgroup SHAP difference may reflect country composition or label imbalance rather than a genuine socioeconomic mechanism; leave-one-country-out validation would test this.
- Repeating the workflow on real monitoring and health-surveillance data with temporal and grouped validation is a direct next step, and would either confirm or overturn the PM2.5 dominance.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. This paper presents an interpretable machine-learning workflow for predicting respiratory disease rates (regression) and air-quality status (binary classification) using the Global Climate-Health Impact Tracker, a synthetically generated weekly country-level dataset with 14,100 records from 25 countries (2015-2025). Nine regression and nine classification models are compared under observation-level nested cross-validation. The authors report that PM2.5 is the dominant predictor of respiratory disease rate; that AQI-derived classification accuracy (about 0.93 balanced accuracy) drops substantially when PM2.5 is removed; that SHAP attributions shift to GDP, precipitation, and healthcare access in the no-PM2.5 setting; and that subgroup SHAP analysis shows a larger PM2.5 contribution in lower-middle-income countries. The paper is explicitly framed as a methodological demonstration and repeatedly disclaims causal or epidemiological interpretation (Highlights; Section 7). Key concerns: the synthetic generator's structural equations are not provided; the Abstract omits the synthetic nature of the data; subgroup classification numbers do not reconcile with aggregate results; and SHAP is computed on a full-data retrained model rather than the CV-evaluated models.
Significance. If the workflow is sound, the paper offers a useful template for auditing climate-health machine-learning models: nested CV with explicit within-dataset framing, target-proxy sensitivity analysis for index-derived labels, and subgroup evaluation in the spirit of fairness-aware ML. The authors deserve credit for reporting the PM2.5-label correlation (r=0.810), acknowledging that observation-level folds do not establish country/temporal generalization, and flagging the independence violation in fold-wise t-tests. The substantive significance is nonetheless limited by the data: the headline claims describe a synthetic file whose generation rules are undocumented, so the PM2.5-dominance and subgroup results are currently indistinguishable from artifacts of the generator. The target-proxy sensitivity demonstration and the subgroup audit template survive, but the regression findings need generator provenance or explicit reframing.
major comments (5)
- [Section 3, Section 5.4, Abstract] Central claim — 'PM2.5 concentration was the dominant predictor of respiratory disease rate' (Section 5.4, Abstract) — rests on a generator whose structural equations are never given. Section 3 concedes the dataset is synthetic and that the documentation 'does not provide complete variable-level provenance or a fully reproducible procedure.' If respiratory_disease_rate was constructed from pm2_5, the SHAP ranking in Figure 2 and the regression results in Table 3 simply restate the generation rule. The synthetic nature appears only in Section 3, not in the Abstract. Required: disclose the generator's structural equations/code (or a documented source), or reframe the PM2.5 claims as 'within this file' throughout the Abstract and Conclusion; ideally validate against a generator with known ground truth.
- [Section 5.2 (Table 5) vs Section 5.5 (Table 8)] Internal inconsistency. XGBoost's mean balanced accuracy is 0.9280 in the main analysis (Table 5), but Table 8 (also 'XGBoost classifier') reports 0.8765/0.8661/0.8557 for the three income groups. The pooled subgroup values imply an overall balanced accuracy of about 0.866, which matches neither Table 5 nor the no-PM2.5 result (0.817, Table 6). The text does not state whether subgroup predictions come from the CV models, a full-data retrained model, per-subgroup models, or a different feature set. This unexplained gap undermines the subgroup classification claims, including the sensitivity/specificity asymmetry discussed in Section 5.5. Please specify the exact evaluation protocol and report reconciled numbers.
- [Section 4 (SHAP methodology), Section 5.4] SHAP values are computed from a single model 'retrained on the full dataset' with inner-CV hyperparameters (Section 4), while all performance claims come from the outer folds of nested CV. The attributions therefore describe a model never evaluated on held-out data; linking performance (Sections 5.1-5.3) to attributions (Sections 5.4-5.5) assumes the full-data model behaves like the CV models. The SHAP-based feature-selection check (Section 5.4: 17-to-12 features, 0.8171-to-0.8169) also does not state whether it was performed inside the CV loop; if done on a single full-data fit, 'retained nearly all predictive performance' is not an unbiased estimate. Recommend computing attributions on the CV models (e.g., averaged over folds) or on an explicit held-out test set, and reporting the feature-reduction protocol.
- [Section 5.5, Figure 5(c)] For the regression subgroup SHAP using Lasso, the model is a single global linear model with a fixed PM2.5 coefficient. Per-group mean |SHAP| for PM2.5 (12.47 vs 7.75/7.82) therefore largely reflects differences in the PM2.5 distribution (distance from the global mean in standardized units) across income groups, not a change in model 'reliance.' Section 6 hedges ('could arise from predictor distributions'), but the Abstract repeats 'PM2.5 contributed more strongly to predictions in lower-middle-income countries' as a finding. For a linear model this should be stated as a distributional consequence (e.g., decompose the group difference or report beta*SD(x) per group), and 'relied more heavily' should be reserved for nonlinear models.
- [Abstract, Section 5.3, Section 7] The classification sensitivity result — balanced accuracy drops from about 0.93 to about 0.82 when PM2.5 is removed — is largely predetermined by label construction: the target is a thresholded AQI and PM2.5 is a primary AQI constituent (the paper itself reports r=0.810 with the binary label). The paper acknowledges this in Sections 5.3 and 7, which is commendable. However, the Abstract presents 'performance decreased substantially when PM2.5 was removed' as a headline finding without noting that it is an expected consequence of the target definition; combined with the undocumented synthetic generator, the no-PM2.5 numbers carry no real-world information. Suggest presenting this explicitly as a methodological illustration of target-proxy checking, and stating so in the Abstract.
minor comments (5)
- [Section 5.1, Table 4] The text says significant differences were found in 22 of the 36 model pairs, but the wins in Table 4 sum to 25 (5+4+4+4+3+2+2+1+0). Each significant pair awards exactly one win, so the text and the table cannot both be correct. Please reconcile.
- [Sections 5.2 and 5.3] Typographical errors: 'F = 152.54 ,p < 0.001' and 'F=16.55, p < 0.001' are missing the closing parenthesis after the p-value, and the formatting of 'F = 78.05' in Section 5.1 should be made consistent.
- [Section 3 vs Section 5.5] Income level is first ordinally encoded and then removed from the feature set during correlation-based reduction, yet Section 5.5 subgroups by income level. Clarify that the removal applies only to model inputs and that subgroup grouping uses the raw income-level column; otherwise the narrative reads as contradictory.
- [Data Availability Statement] Making processed data and code available 'upon reasonable request' is weak for a methods-demonstration paper. A public repository containing the code, the preprocessing pipeline, and ideally the synthetic generator specification would allow readers to separate the workflow contribution from dataset artifacts. Also, the Kaggle dataset reference (Gokhale 2025) should include an access date.
- [Section 7 (Limitations)] Section 7 correctly flags the independence violation in fold-wise paired t-tests and suggests the Nadeau-Bengio adjusted t-test. Actually applying the correction (or reporting its results) would substantiate the pairwise-comparison claims rather than leaving them as a stated limitation for future work.
Circularity Check
AQI-derived classification makes PM2.5 dominance self-definitional; regression result is not independently verifiable but not shown circular.
specific steps
-
self definitional
[Section 3 (Data and Study Design); Section 5.3 (Sensitivity Analysis Without PM2.5)]
"The binary label was constructed from the air quality index using a threshold of 100 ... PM2.5 was examined separately because it is closely related to AQI and strongly associated with the derived label ... PM2.5 concentration was correlated with the binary air-quality label at r = 0.810, quantifying the strength of this target-proxy relationship."
The classification target y is defined as I(AQI > 100), and PM2.5 is a primary component of AQI; hence PM2.5 is built into the label by definition. The reported high balanced accuracy with PM2.5 included and the drop when PM2.5 is removed therefore restate the label-construction rule rather than test an external prediction. The paper labels this 'target-proxy dependence' and is transparent, but the abstract and highlights still present the decrease ('performance decreased substantially when PM2.5 was removed') as a result of the analysis, so this component of the claimed derivation reduces by construction.
full rationale
Only the classification task exhibits a demonstrable circular step. The binary label is a thresholded AQI, and PM2.5 is an input to AQI, so the PM2.5 SHAP dominance and the balanced-accuracy drop are consequences of how the target is defined; the paper's own r = 0.810 confirms the engineered association. The regression claim ('PM2.5 was the dominant predictor of respiratory disease rate') is not demonstrably circular from the text: the target respiratory_disease_rate is not shown to be defined in terms of PM2.5, and no generator equations are given, so a specific reduction cannot be exhibited. That is a data-provenance/verifiability limitation, not a proven circular identity. There are no load-bearing self-citations; the methodological references (SHAP, nested CV) are external and standard. Because one of the two central reported predictions (the classification sensitivity result) is forced by label construction, the overall circularity score is 6 rather than 0-2.
Axiom & Free-Parameter Ledger
free parameters (3)
- AQI threshold for binary air-quality label =
100 (AQI > 100 = unhealthy)
- Correlation threshold for feature reduction =
0.85
- SHAP-based feature-set reduction size =
17 -> 12 features in the no-PM2.5 setting
axioms (3)
- ad hoc to paper The synthetic dataset generator produces realistic climate-health relationships.
- domain assumption Record-level cross-validation folds are independent enough for performance estimation.
- domain assumption SHAP values from a full-data retrained XGBoost model represent the models compared in cross-validation.
read the original abstract
Air pollution and climate-related stressors are increasingly important concerns for respiratory health, especially in settings with unequal environmental exposure and healthcare capacity. This study evaluates an interpretable machine learning framework for predicting respiratory disease rates and air-quality status using structured country-level weekly data. Two supervised learning tasks were considered: regression of respiratory disease rate per 100,000 population and binary classification of air-quality status. Nine regression models and nine classification models were compared using nested cross-validation. Model interpretation was conducted using SHAP values, and subgroup analysis was performed across income levels and geographic regions. The results showed that PM2.5 concentration was the dominant predictor of respiratory disease rate, with linear and regularized linear models achieving the strongest regression performance. For air-quality classification, models achieved high balanced accuracy when PM2.5 was included, but performance decreased substantially when PM2.5 was removed, indicating strong dependence on pollutant-related information. SHAP analysis showed that, without PM2.5, socioeconomic and meteorological variables such as GDP per capita, precipitation, and healthcare access became more influential. Subgroup analysis showed similar aggregate regression error across income groups, but PM2.5 contributed more strongly to predictions in lower-middle-income countries. These results show that model accuracy alone is not sufficient for climate-health prediction. Interpretable models can help identify dominant pollution-related signals, test whether results depend on key pollutant variables, and show whether prediction patterns differ across socioeconomic groups.
Reference graph
Works this paper leans on
-
[3]
“Artificial Intelligence Platform to Predict Children’s Hospital Care for Respiratory Disease Using Clinical, Pollution, and Climatic Factors.” Journal of Global Health 15 (July): 04207. https://doi.org/10.7189/jogh.15.04207. Cappelli, Francesco, Gianfranco Castronuovo, Salvatore Grimaldi, and Vito Telesca
-
[6]
Global Climate-Health Impact Tracker (2015-2025)
“Global Climate-Health Impact Tracker (2015-2025).” https://www.kaggle.com/datasets/sohumgokhale/global-climate-health-impact-tracker-2015-
2015
-
[7]
Machine Learning for Urban Air Quality Analytics: A Survey
“Machine Learning for Urban Air Quality Analytics: A Survey.” arXiv:2310.09620. Preprint, arXiv, October
-
[11]
Rojas, Juan C., John Fahrenbach, Sonya Makhni, et al
https://doi.org/10.48550/arXiv.1811.12808. Rojas, Juan C., John Fahrenbach, Sonya Makhni, et al
-
[14]
Houdou, Anass, Imad El Badisy, Kenza Khomsi, et al
https://doi.org/10.48550/arXiv.2310.09620. Houdou, Anass, Imad El Badisy, Kenza Khomsi, et al
-
[15]
A Survey on Bias and Fairness in Machine Learning
“A Survey on Bias and Fairness in Machine Learning.” ACM Computing Surveys 54 (6): 1–35. https://doi.org/10.1145/3457607. Mhasawade, Vishwali, Yuan Zhao, and Rumi Chunara
-
[17]
“Air Pollution Particulate Matter (PM2.5) Prediction in South African Cities Using Machine Learning Techniques.” Frontiers in Artificial Intelligence 6 (October): 1230087. https://doi.org/10.3389/frai.2023.1230087. Patel, Pranshu, Swara Patel, Kanish Shah, et al
arXiv 2023
-
[18]
“A Systematic Study on PM2.5 and PM10 Concentration Prediction in Air Pollution Using Machine Learning and Deep Learning Model.” Environmental Chemistry and Ecotoxicology 7: 1401–15. https://doi.org/10.1016/j.enceco.2025.07.001. Rajkomar, Alvin, Michaela Hardt, Michael D. Howell, Greg Corrado, and Marshall H. Chin
-
[20]
Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning
“Model Evaluation, Model Selection, and Algorithm Selection in Machine Learning.” arXiv:1811.12808. Preprint, arXiv, November
-
[22]
Framework for Integrating Equity Into Machine Learning Models
“Framework for Integrating Equity Into Machine Learning Models.” Chest 161 (6): 1621–27. https://doi.org/10.1016/j.chest.2022.02.001. Saminathan, S., and C. Malathy
-
[23]
Ensemble-Based Classification Approach for PM2.5 Concentration Forecasting Using Meteorological Data
“Ensemble-Based Classification Approach for PM2.5 Concentration Forecasting Using Meteorological Data.” Frontiers in Big Data 6 (June): 1175259. https://doi.org/10.3389/fdata.2023.1175259. Schuch, Helena Silveira, Mariane Furtado, Gabriel Ferreira Dos Santos Silva, Ichiro Kawachi, Alexandre D. P. Chiavegatto Filho, and Hawazin W. Elani
arXiv 2023
-
[24]
Fairness of Machine Learning Algorithms for Predicting Foregone Preventive Dental Care for Adults
“Fairness of Machine Learning Algorithms for Predicting Foregone Preventive Dental Care for Adults.” JAMA Network Open 6 (11): e2341625. https://doi.org/10.1001/jamanetworkopen.2023.41625. Telesca, Vito, and Maríca Rondinone
arXiv 2023
-
[25]
Lundberg, Scott M., Gabriel Erion, Hugh Chen, et al
https://doi.org/10.48550/arXiv.1705.07874. Lundberg, Scott M., Gabriel Erion, Hugh Chen, et al
-
[27]
“Practical Considerations and Applied Examples of Cross- Validation for Model Development and Evaluation in Health Care: Tutorial.” JMIR AI 2 (December): e49023. https://doi.org/10.2196/49023. Yehuala, Tirualem Zeleke, Bezawit Melak Fente, Sisay Maru Wubante, and Nebiyu Mekonnen Derseh
-
[28]
“Exploring Machine Learning Algorithms to Predict Acute Respiratory Tract Infection and Identify Its Determinants among Children under Five in Sub-Saharan Africa.” Frontiers in Pediatrics 12 (November): 1388820. https://doi.org/10.3389/fped.2024.1388820. Appendix A. Hyperparameter Search Spaces Model Task Parameters Search Space Best Value Linear Regressi...
arXiv 2024
-
[91]
https://doi.org/10.1186/1471-2105-7-91. Wilimitis, Drew, and Colin G. Walsh
-
[867]
Chin, Marshall H., Nasim Afsar-Manesh, Arlene S
https://doi.org/10.3390/ijerph21070867. Chin, Marshall H., Nasim Afsar-Manesh, Arlene S. Bierman, et al
-
[1276]
Ku, Yunseo, Soon Bin Kwon, Jeong-Hwa Yoon, Seog-Kyun Mun, and Munyoung Chang
https://doi.org/10.3390/bioengineering12111276. Ku, Yunseo, Soon Bin Kwon, Jeong-Hwa Yoon, Seog-Kyun Mun, and Munyoung Chang
-
[2017]
A Unified Approach to Interpreting Model Predictions
“A Unified Approach to Interpreting Model Predictions.” arXiv:1705.07874. Preprint, arXiv, November
-
[2018]
Ensuring Fairness in Machine Learning to Advance Health Equity
“Ensuring Fairness in Machine Learning to Advance Health Equity.” Annals of Internal Medicine 169 (12): 866–72. https://doi.org/10.7326/M18-1990. Raschka, Sebastian
doi:10.7326/m18-1990 1990
-
[2020]
From Local Explanations to Global Understanding with Explainable AI for Trees
“From Local Explanations to Global Understanding with Explainable AI for Trees.” Nature Machine Intelligence 2 (1): 56–67. https://doi.org/10.1038/s42256-019-0138-9. Mehrabi, Ninareh, Fred Morstatter, Nripsuta Saxena, Kristina Lerman, and Aram Galstyan
-
[2021]
Machine Learning and Algorithmic Fairness in Public and Population Health
“Machine Learning and Algorithmic Fairness in Public and Population Health.” Nature Machine Intelligence 3 (8): 659–66. https://doi.org/10.1038/s42256-021-00373-4. Morapedi, Tshepang Duncan, and Ibidun Christiana Obagbuwa
-
[2022]
“Machine Learning Models for Predicting the Occurrence of Respiratory Diseases Using Climatic and Air- Pollution Factors.” Clinical and Experimental Otorhinolaryngology 15 (2): 168–76. https://doi.org/10.21053/ceo.2021.01536. Lundberg, Scott, and Su-In Lee
arXiv 2021
-
[2023]
“Guiding Principles to Address the Impact of Algorithm Bias on Racial and Ethnic Disparities in Health and Health Care.” JAMA Network Open 6 (12): e2345050. https://doi.org/10.1001/jamanetworkopen.2023.45050. Gokhale, Sohum
arXiv 2023
-
[2024]
“Interpretable Machine Learning Approaches for Forecasting and Predicting Air Pollution: A Systematic Review.” Aerosol and Air Quality Research 24 (1): 230151. https://doi.org/10.4209/aaqr.230151. Im, Chaeyeong, Wonji Kim, and Heesoo Kim
-
[2025]
Prognosis of Air Quality Index and Air Pollution Using Machine Learning Techniques
“Prognosis of Air Quality Index and Air Pollution Using Machine Learning Techniques.” Scientific Reports 15 (1): 25890. https://doi.org/10.1038/s41598-025-11260-y. Ardali, Faezeh, Mwembezi A. Nyelele, and Gerald M. Knapp
-
[2385]
https://doi.org/10.1038/s41598-024-85089-2. Cabral-Miranda, William, Cauê Beloni, Felipe Lora, Rogério Afonso, Thales Araújo, and Fátima Fernandes
-
[4864]
Varma, Sudhir, and Richard Simon
https://doi.org/10.3390/s25154864. Varma, Sudhir, and Richard Simon
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.