{"id":"7d707bfb-b8d6-4e39-afa3-a1d356727804","arxiv_id":"2506.16929","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"The authors report 99% LSTM accuracy for predicting neonatal death in São Paulo, but the result is undermined by preprocessing before the data split, class imbalance, and missing code and data.","lead":"This paper trained seven machine learning and deep learning models on 1.4 million birth and death records from São Paulo to predict whether a newborn dies within 28 days. It reports that an LSTM model reached 99% accuracy, but the evaluation methods have serious flaws that make the headline number unreliable.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 99% LSTM accuracy is not a valid estimate of real-world performance because the evaluation protocol does not reflect natural class prevalence, so the central claim is unsupported.","rationale":"The reader's weakest-assumption analysis identifies the central flaw: applying NearMiss before the train/test split invalidates the reported accuracy as an estimate of real-world performance. My stress-test read agrees and sharpens the point by noting that the deep-learning models are evaluated under the opposite condition—full imbalanced data—where 99% accuracy is exactly what a trivial majority-class predictor achieves. Either way, the reported Table 1 metrics cannot distinguish genuine predictive skill from artifacts of the evaluation protocol. The paper has strengths worth acknowledging: a large public administrative dataset, seven standard classifiers, and a relevant public-health question. However, routine application of standard tools does not compensate for the absence of a valid held-out test set, confidence intervals, confusion matrices, or code/data for independent verification. The central claim is therefore unsupported as written, and the REJECT verdict is appropriate. No additional concern needs to be raised; the evaluation-protocol issue is sufficient and load-bearing.","tokens_in":10522,"tokens_out":3011,"duration_ms":36515,"concrete_test":"Re-run the evaluation on the original São Paulo birth-records data with a strict protocol: split 70:30 first, apply NearMiss (if at all) only to the training portion, and leave the test set untouched at natural prevalence. Compute accuracy, precision, recall, F1, and PR-AUC for all seven models on this same held-out test set. If LSTM's accuracy is not materially above the majority-class baseline or if XGBoost or random forest match or beat it on PR-AUC, the headline claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that LSTM predicts neonatal death with 99% accuracy and is the best model for real-life use—rests entirely on Table 1, but the evaluation protocol cannot support that number. Section 2.3 applies NearMiss undersampling before the 70:30 train/test split described in Section 2.4, so the test set used for the classical ML models is artificially balanced rather than reflecting the roughly 1% neonatal-death prevalence in São Paulo. Accuracy computed on such a test set is not the accuracy a deployed model would achieve. For the deep learning models, Section 2.6 says NearMiss was 'evaded' and the full dataset was used; on the full imbalanced data, a trivial always-survive classifier already achieves approximately 99% accuracy, so the LSTM's reported 99% accuracy is indistinguishable from the majority-class baseline unless recall and precision are demonstrated on a natural-prevalence test set. The paper reports only rounded aggregate metrics, with no confusion matrices, confidence intervals, PR-AUC, or calibration checks, and no code or data are provided. Thus the 99% figure is not established as genuine predictive skill, and the conclusion that LSTM is most suitable for real-life use does not follow.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies six machine learning and deep learning models—logistic regression, KNN, support vector classifier, XGBoost, random forest, CNN, and LSTM—to 1,427,906 birth records from São Paulo to predict whether a newborn dies within 28 days. It reports that XGBoost and random forest achieve 94% accuracy and that LSTM achieves 99% accuracy, concluding that LSTM is the most suitable model for real-life neonatal mortality prediction. The central evaluation, however, is invalid for the stated real-world claim: for the classical models, NearMiss undersampling is applied before the 70:30 train/test split, so the test set is artificially balanced rather than representative of natural prevalence; for the deep learning models, the reported metrics on a highly imbalanced population are not interpretable without confusion matrices and natural-prevalence evaluation. The manuscript also lacks code, a correct data citation, hyperparameters, architecture details, confidence intervals, and any form of external validation.","tokens_in":10791,"tokens_out":6180,"duration_ms":69063,"significance":"If the 99% accuracy claim were properly validated, the work would have practical significance for early-warning systems in neonatal care, and the use of a 1.4-million-record population dataset is a clear strength. The paper also compares several standard algorithms, which is a useful framing. However, the reported results are not established as genuine predictive skill: the evaluation protocol cannot support the headline accuracy, and the manuscript does not provide the reproducibility artifacts (code, data, detailed experimental configuration) needed to verify or repair the claim. The potential significance is therefore not realized in the present manuscript.","major_comments":[{"comment":"NearMiss undersampling is applied to the entire dataset in Section 2.3, before the 70:30 train/test split described in Section 2.4. This means the test set used for logistic regression, KNN, SVC, XGBoost, and random forest has artificially equal class proportions rather than the natural prevalence of neonatal death. Accuracy, precision, recall, and F1 reported in Table 1 for these models are therefore not estimates of real-world performance on actual newborns. The split must be performed before any balancing, with NearMiss applied only to the training portion and the test set left untouched.","section":"§2.3 and §2.4"},{"comment":"For the deep learning models, Section 2.6 states that NearMiss was 'evaded' and the models were trained on the complete dataset. In a population where neonatal deaths are rare, an always-survive classifier already achieves approximately 99% accuracy, so the LSTM's reported 99% accuracy is not, by itself, evidence of predictive skill. The manuscript says confusion matrices were generated but does not show them, and no PR-AUC, recall at natural prevalence, or calibration metrics are reported. The claim that LSTM is 'most suitable' for real-life use requires evaluation on a test set with natural class prevalence and requires reporting true-positive and false-positive rates, not only rounded aggregate scores.","section":"§2.6 and Table 1"},{"comment":"No confidence intervals, cross-validation results, or statistical significance tests are reported for any model, and no hyperparameters, network architectures, optimizer settings, or training epochs are provided. Without these details, the observed differences between models—for example, 94% for XGBoost/random forest versus 99% for LSTM—cannot be interpreted as meaningful or reproducible. The conclusion that LSTM is the best model for deployment is therefore unsupported by the evidence presented.","section":"§3 and Table 1"},{"comment":"The dataset is described as 'secondary information on Children's births and deaths in the city of São Paulo' but is cited to reference [4], which is a WHO fact sheet on newborn mortality, not a dataset source. The actual provenance, version, and access information for the São Paulo data are missing. This is a load-bearing reproducibility issue because the entire study depends on this dataset.","section":"§2.1 and reference [4]"}],"minor_comments":[{"comment":"Subsections 3.6.1 through 3.6.7 appear under Section 2.6 but are numbered as if they belong to Section 3; they should be renumbered as 2.6.1 through 2.6.7.","section":"§2.6"},{"comment":"The phrase '26.693 babies out of 1,000 births' should read '26.693 per 1,000 births', and 'Macro Trades' should be 'Macrotrends'.","section":"Abstract"},{"comment":"The text says 'All the column descriptions are given in Table 1', but Table 1 contains model comparison results, not column descriptions; the column-description table appears to be missing or mislabeled.","section":"§2.1"},{"comment":"The recall formula is written as TP/(TP + F); the denominator should be TP + FN. Similar notation ambiguities appear in the true-positive and false-positive rate definitions.","section":"§3"},{"comment":"The flowchart shows NearMiss applied before the split for all models, but Section 2.6 says NearMiss was not used for deep learning; the flowchart should distinguish the two pipelines.","section":"Figure 1"},{"comment":"The accuracy comparison graph does not include error bars or any measure of variance, so the visual comparison is not informative.","section":"Figure 9"},{"comment":"The text says the split 'randomly chooses the columns for test and training'; it should say rows or samples.","section":"§2.4"},{"comment":"There are numerous typographical and grammatical issues, including 'logical regression', 'nearest neighborhood', 'Near Miss' versus 'nearmiss' inconsistency, and 'using LSTM of the model shall be best suited'; a thorough language edit is needed.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is substantially below the standard of a serious journal. The evaluation protocol is invalid for the real-world claim, and the missing data citation, code, and detailed experimental configuration make the results unverifiable. I chose major_revision rather than reject because the central flaws are in principle fixable by re-running the experiments with a proper split-before-balancing protocol and reporting confusion matrices on a natural-prevalence test set. However, the required changes amount to a full reanalysis, and if the editor's policy is to reject papers whose central claim is not supported by the reported evaluation, rejection is also defensible."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Nothing in this paper made me re-evaluate anything, but the dataset is genuinely large (1.4M birth records from São Paulo) and the authors do cite the relevant prior ML work on neonatal mortality in other countries. Those are the two things the paper has going for it.\n\nWhat is actually new is just the São Paulo context. Every algorithm is off-the-shelf, and earlier studies the authors cite (Ethiopia, Rwanda, India, Bangladesh, Iran) already applied similar methods to similar problems. So novelty is thin.\n\nThe soft spots are not minor. For the ML models, NearMiss undersampling is applied before the 70:30 split (Sections 2.3 and 2.4). That means the test set is balanced, not representative of the real population where neonatal death is rare. The reported 93–94% accuracy is therefore a property of the test distribution, not a measure of real-world skill. For the deep learning models, the paper says NearMiss was evaded and the full imbalanced dataset was used (Section 2.6). On a dataset with roughly 1% positive examples, predicting 'survive' for everyone already gives ~99% accuracy, so the LSTM's 99% figure is indistinguishable from a majority-class baseline unless recall, precision, or PR-AUC on the minority class are reported. They aren't. There are no confusion matrices, no confidence intervals, no calibration checks, no external validation, and no code or data.\n\nI'm not accusing anyone of hiding anything—the paper openly describes the pipeline—but the description is enough to see why the headline claim doesn't hold. The conclusion that LSTM is 'most suitable' for real-life use does not follow from the evidence.\n\nWho benefits from reading this? Someone wanting a reminder that balancing before splitting invalidates a test set, or that accuracy on imbalanced data can be misleading. As a substantive contribution, it's not there.\n\nMy recommendation: this does not deserve a serious referee. The topic is important, but the flaw is load-bearing and the paper would need a full re-analysis to be salvageable. Desk reject with a clear explanation would be fair.","headline":"The paper's headline 99% LSTM accuracy is not supported by its evaluation: balancing before splitting and majority-class accuracy on an imbalanced dataset make the central claim an artifact.","tokens_in":11317,"tokens_out":4503,"would_cite":false,"duration_ms":49168,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an LSTM neural net trained on 1.4 million São Paulo birth records predicts newborn death within 28 days with 99% accuracy, the best of seven models tested.","keywords":["neonatal mortality prediction","infant mortality","machine learning","deep learning","LSTM","XGBoost","random forest","São Paulo birth records"],"falsifier":"Re-run the pipeline on the original 1,427,906 records, splitting 70:30 before any balancing, and compute LSTM accuracy, precision, recall, and ROC-AUC on the natural-prevalence test set; if accuracy falls substantially below 99% or recall on the rare death class drops sharply, the paper's real-world performance claim is falsified.","tokens_in":10380,"feed_emoji":"👶","tokens_out":8192,"duration_ms":87375,"temperature":0.7,"pith_summary":"The paper sets out to find the most accurate machine-learning or deep-learning model for predicting whether a newborn will die within 28 days, using 1,427,906 birth and death records from São Paulo between 2012 and 2018. Across seven algorithms, the long short-term memory network (LSTM) reaches 99% accuracy, while XGBoost and random forest each reach 94%; the authors conclude that LSTM is the model best suited to real-life use as an early-warning tool. The practical point is that a nearly error-free predictor would let clinicians direct extra care to at-risk babies and mothers. The paper also reports that the classical models were trained on a NearMiss-undersampled dataset to handle the rarity of deaths, while the deep learning models were trained on the full dataset.","feed_headline":"LSTM predicts newborn death within 28 days at 99% accuracy","feed_subtitle":"On 1.4 million São Paulo birth records, the model beats XGBoost and random forest by five points.","key_machinery":"The load-bearing object is the long short-term memory (LSTM) network, a recurrent-neural-network variant that selectively retains patterns over long input sequences; here it is applied to 23 tabular birth-record features to classify the binary label. The auxiliary mechanism for the classical baselines is NearMiss undersampling, which balances the overwhelmingly common 'survived' class against the rare 'death' class before the 70:30 train/test split. The LSTM's 99% result carries the paper's claim that the deep model is the best choice for deployment, while the 94% results from XGBoost and random forest provide the comparison that makes the gap visible.","core_discovery":"On the authors' own terms, the central discovery is that an LSTM classifier trained on the full São Paulo dataset predicts the 'Death within 28 Days' label with 99% accuracy, 99% precision, 99% recall, and a 0.99 F1-score. This is reported as beating a CNN (98%) and the best classical models, XGBoost and random forest (94% each). The paper interprets these numbers to mean that when a newborn's recorded parameters are passed in, the model returns the correct outcome 99 times out of 100, making LSTM the recommended choice for deciding whether precautionary care is needed. The conclusion extends this to say that such intelligent models could help doctors and health workers be aware and prepared ahead of time.","pith_inferences":["My inference: the reported 99% accuracy almost certainly overstates real-world performance, because the test set was built after NearMiss undersampling and therefore contains far more deaths than the actual São Paulo newborn population; a natural-prevalence test set would be the real check.","My inference: because the input features are tabular rather than explicitly sequential, the LSTM's apparent edge may come from model capacity or regularization rather than from its sequence memory; comparing it to a carefully tuned feed-forward network or a tabular transformer would separate those explanations.","My inference: the same pipeline could be turned into a risk score computed at birth registration, and the approach could be applied to other Brazilian state birth cohorts to test whether the 99% result transfers."],"forward_implications":["A hospital or public-health agency in São Paulo could use the LSTM as a screening step: a newborn's record would be flagged as at-risk, prompting earlier care.","The five-percentage-point gap over XGBoost and random forest suggests the deep model learns patterns the tree ensembles miss, and if reproduced it justifies the added complexity of training an LSTM.","Because the dataset spans 1.4 million births over seven years, the model's training signal is large enough to support subgroup analysis by birth weight, gestational age, or maternal characteristics.","The reported 99% recall implies very few missed deaths in the balanced test setting, which matters for a screening tool where a false negative is the costliest error."],"supporting_citations":[{"why":"The paper cites this as the source of the São Paulo 2012-2018 birth and death records, the 1,427,906-row dataset on which all models are trained and tested.","marker":"[4]"},{"why":"A prior ensemble model reported 94.39% accuracy for neonatal and infant mortality, providing the closest comparison point against which the paper's 99% LSTM result is positioned.","marker":"[7]"},{"why":"The paper cites this for the LSTM architecture whose 99% accuracy is the central result of the study.","marker":"[25]"}],"fun_headline_variants":["99% accuracy: LSTM flags at-risk newborns in Sao Paulo","LSTM predicts neonatal death at 99% accuracy on 1.4M births","LSTM outperforms traditional ML for newborn mortality risk","Neonatal death prediction: LSTM hits 99% on Sao Paulo data","Using LSTM to spot at-risk newborns: 99% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracy assumes that the balanced test set created by undersampling, with roughly equal numbers of survivors and deaths, represents the real São Paulo newborn population well enough for 99% accuracy to describe real-world use.","fun_headline_variants_meta":{"raw":{"variants":["99% accuracy: LSTM flags at-risk newborns in Sao Paulo","LSTM predicts neonatal death at 99% accuracy on 1.4M births","LSTM outperforms traditional ML for newborn mortality risk","Neonatal death prediction: LSTM hits 99% on Sao Paulo data","Using LSTM to spot at-risk newborns: 99% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000865,"raw_usage":{"total_tokens":3734,"prompt_tokens":916,"completion_tokens":2818,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":532,"completion_tokens_details":{"reasoning_tokens":2723}},"tokens_in":532,"tokens_out":2818,"duration_ms":22684,"temperature":1.0,"reasoning_tokens":2723,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T23:37:25.914140+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the pipeline on the original 1,427,906 records, splitting 70:30 before any balancing, and compute LSTM accuracy, precision, recall, and ROC-AUC on the natural-prevalence test set; if accuracy falls substantially below 99% or recall on the rare death class drops sharply, the paper's real-world performance claim is falsified.","supporting_citations":[{"cited_title":"Newborn Mortality,","cited_arxiv_id":null,"evidence_quote":"The paper cites this as the source of the São Paulo 2012-2018 birth and death records, the 1,427,906-row dataset on which all models are trained and tested."},{"cited_title":"Violence detection in automated video surveillance: recent trends and comparative studi es,","cited_arxiv_id":null,"evidence_quote":"The paper cites this for the LSTM architecture whose 99% accuracy is the central result of the study."}],"review_version":1}