{"id":"d9fa01e9-cb3e-4974-b5e8-19b7d879595e","arxiv_id":"2506.22129","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":2.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"A stacking ensemble of standard classifiers with SMOTE achieves 96% test accuracy on 3-class earthquake damage grade prediction from building attributes.","lead":"This paper compares machine learning and deep learning models to predict building damage grades from the 2015 Gorkha earthquake dataset, and reports that a stacking ensemble reaches 96% test accuracy. A generalist might read it to see whether off-the-shelf ML methods can support post-disaster triage and resource allocation.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The claimed 96% test accuracy for the Stacking Ensemble is unverifiable because the paper never specifies whether SMOTE and undersampling were applied before or after the train/test split, and the result tables do not label train versus test.","rationale":"I read the paper as attempting to demonstrate that a stacking ensemble of standard classifiers achieves a specific test accuracy on a public earthquake damage dataset. The strongest claim is a number, and for that number to be meaningful the evaluation protocol must be valid. The reader's weakest assumption is exactly the load-bearing point: the manuscript never says where the train/test split occurs or whether SMOTE and resampling were confined to training data. My reading of Sections 2.6, 4.1, 5.1, and 6 confirms that this condition is missing. The tables compound the problem by reporting an 'accuracy' column that is identical across class-specific tables and by never labeling train versus test, while the stated 99.9% training accuracy appears only in the conclusion. This is not a claim that the result is impossible; a clean split could conceivably yield high accuracy, but the current text provides no way to verify it. I see no reason to change the reader's rejection, and I would keep the verdict as reject unless the authors supply a precise split-before-resampling protocol, the holdout confusion matrix, and ideally code. The underlying methods are standard and the dataset is public, which means the missing protocol is fixable, but on the present manuscript the central claim is not established.","tokens_in":7348,"tokens_out":5147,"duration_ms":61667,"concrete_test":"Re-implement the described pipeline on the Gorkha Nepal building damage dataset. Use a stratified 80/20 split before any preprocessing; apply SMOTE, RandomUnderSampler, Isolation Forest, and SelectKBest only inside the training split, tune the Stacking meta-learner by nested cross-validation, and evaluate once on the untouched holdout. Compare this holdout accuracy to the reported 96% and also run a control pipeline in which resampling is applied before the split. If the no-leakage holdout accuracy is materially below 96%, or if the leakage control reproduces roughly 96%, the Section 6 claim is an artifact of protocol leakage. Report a per-class confusion matrix on the untouched holdout as part of the check.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is the Section 6 statement that the Stacking Ensemble achieves 96% test accuracy and 99.9% training accuracy. For that claim to hold, the test set must be a clean holdout untouched by training and untouched by resampling. The manuscript never establishes this. Section 4.1 lists LabelEncoder, Isolation Forest anomaly removal, SMOTE with RandomUnderSampler, and SelectKBest feature selection as preprocessing steps before any mention of validation or a holdout split. Section 2.6 describes generic k-fold cross-validation but gives no k, no split ordering, and no nested evaluation procedure. Tables 3-5 report per-class precision, recall, F1-score, and an 'accuracy' column, but no table header states whether these are train, validation, or test numbers, and the 'accuracy' values are identical for each algorithm across the three class-specific tables. The conclusion's 99.9% training accuracy appears nowhere in the results tables. If SMOTE is applied before splitting, synthetic samples derived from all original data can appear in both training and test partitions, creating near-duplicate test points and inflating every metric; RandomUnderSampler applied before splitting can also distort the test distribution. Without a stated split protocol, a consistent holdout definition, or code, the reported 96% cannot be parsed as an honest test accuracy, and the comparative claim over ten algorithms is unsupported. This is an internal specification gap in the evaluation protocol, not merely a disagreement with field consensus.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a pipeline for classifying building damage grades into three classes using the 2015 Gorkha earthquake dataset, combining SMOTE, RandomUnderSampler, Isolation Forest, and SelectKBest feature selection with base classifiers (Logistic Regression, Decision Tree, Random Forest, GBM, AdaBoost, LightGBM, XGBoost), ensemble methods (Voting, Bagging, Stacking), and deep models (FFN and KAN). The central claim, stated in Section 6, is that a Stacking Ensemble achieves 96% test accuracy and 99.9% training accuracy, outperforming ten other algorithms. The paper provides mathematical formulations for SMOTE, loss functions, ensemble methods, and evaluation metrics, but the experimental evaluation is critically underspecified.","tokens_in":7606,"tokens_out":5033,"duration_ms":51036,"significance":"If the 96% test accuracy claim were validated, it would suggest that a stacking ensemble of standard classifiers is a highly accurate tool for post-earthquake building damage triage, which could be practically useful for disaster response. The paper uses a publicly available real-world dataset and explores a broad set of models, including a comparison between classical ML and newer KAN architectures. However, the evaluation protocol is not described with sufficient detail to verify the central quantitative result, and no code, error bars, or statistical comparisons are provided; as it stands, the significance of the claimed result cannot be assessed.","major_comments":[{"comment":"The preprocessing pipeline described in Section 4.1 applies LabelEncoder, Isolation Forest, SMOTE with RandomUnderSampler, and SelectKBest before any mention of a train/test split, while Section 2.6 defines only generic k-fold cross-validation with no specified k and no nested procedure. This creates a leakage risk: if SMOTE or undersampling is applied before splitting, synthetic samples derived from all original data can appear in both training and test partitions, inflating every reported metric. The paper never states where the split occurs, so the claimed 96% test accuracy in Section 6 cannot be identified as coming from an untouched holdout set. This is a load-bearing gap that directly undermines the central claim.","section":"Section 4.1"},{"comment":"Tables 3, 4, and 5 are labeled as per-class performance for Class 0, Class 1, and Class 2, but the 'Accuracy' column contains identical values for each algorithm across all three tables (e.g., Stacking Classifier is 0.96 in every table). Accuracy is a global metric, so labeling these tables as per-class accuracy is incoherent, and no header or text states whether the numbers are train, validation, or test results. In addition, the conclusion's 99.9% training accuracy appears nowhere in the results tables, so the Section 6 claim cannot be traced to any reported result.","section":"Tables 3-5"},{"comment":"The comparison across the ten algorithms in Section 5 is presented without error bars, confidence intervals, repeated runs over random seeds, or any statistical significance test. In Tables 3-5, most strong models are clustered at accuracy 0.95-0.96, and the differences between the Stacking Classifier and Random Forest or LightGBM are within one percentage point, which is likely within noise. The assertion in Section 6 that the Stacking Ensemble 'stands out' among ten algorithms is therefore unsupported by the evidence presented.","section":"Section 5"},{"comment":"The FFN and KAN results in Sections 5.2.1 and 5.2.2 report train and test accuracies (94%/92.4% for FFN, 96.44%/94.39% for KAN) but give no description of the train/test split, whether SMOTE was applied before or after splitting, or the hyperparameter search protocol for these deep models. The KAN's 96.44% training accuracy is also quoted in the conclusion as an example of 'strong generalization,' but without a clear evaluation protocol it is impossible to determine whether the reported test accuracies are honest holdout results.","section":"Section 5.2"}],"minor_comments":[{"comment":"The text contains a typo in 'ANOV A F-test'; it should read 'ANOVA F-test'.","section":"Section 2.2"},{"comment":"The number of folds k in the k-fold cross-validation is never specified, and no nested cross-validation procedure is described for cases where feature selection or hyperparameter tuning is performed.","section":"Section 2.6"},{"comment":"The table headers contain spacing errors such as 'W eighted Avg', and the 'Macro Avg' and 'Weighted Avg' columns are not defined in Section 2.4.","section":"Tables 3-5"},{"comment":"The paper's outline in Section 1 says that Section 2 reviews related works, Section 3 presents methodology, and Section 4 presents results, but the actual sections are shifted: Section 2 is mathematical formulation, Section 3 is related works, Section 4 is methodology, and Section 5 is results; the outline should be corrected.","section":"Section 1"},{"comment":"Table 1 reports a count of 100 for every numerical feature, which is inconsistent with the categorical frequency counts in Table 2 (e.g., 254,477 occurrences) and with the presumably much larger dataset size; the sampling or subset used for the statistical summary should be clarified.","section":"Table 1"},{"comment":"Several references are duplicated or inconsistently formatted, including 'Lu and et al.' versus 'Min Lu and et al.', and 'Klusowski and Tian' appears as both 2023 and 2024 entries; also, the paper provides no link to code or data availability.","section":"References"}],"recommendation":"reject","confidential_remarks":"For the editor: the central result is unverifiable from the manuscript as written, and the evaluation protocol described in Section 4.1 risks data leakage that could invalidate all reported metrics. The paper would need a substantial revision with a clearly stated train/test split, resampling confined to the training set, corrected result tables, and ideally code or reproducible artifacts before it could be considered for publication. This is not a case of mere presentation issues."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Thanks for the report. I read the paper. Bottom line: this is a routine benchmark on the Ghimire et al. Gorkha dataset, and I wouldn't trust the headline 96% test accuracy because the paper never says where the train/test split is relative to SMOTE and undersampling, and the tables don't label train vs test. The stress test is right.\n\nWhat's actually here: the authors apply a standard pipeline—label encoding, anomaly removal, SMOTE + RandomUnderSampler, SelectKBest, then 10 classifiers and ensembles—to a published dataset. That's fine as a technical exercise. The positive side is they compare a reasonable set of models including KAN, which you don't see often in this literature, and they report per-class precision/recall/F1. The dataset is real and public, so someone could reproduce this if they had the code. That's about it.\n\nThe problems are load-bearing. The central claim—96% test accuracy for stacking—is not traceable. Section 4.1 lists SMOTE and undersampling before any mention of a holdout. If resampling happens before splitting, test points are contaminated by synthetic neighbors from the training set, and every number in the paper is inflated. The k-fold description in Section 2.6 is generic: no k, no nesting, no ordering. Tables 3–5 report the same 'accuracy' value for each algorithm across all three class tables, which makes no sense as per-class accuracy and suggests the numbers are copied from a single run, possibly training accuracy. The conclusion states 99.9% training accuracy for stacking, but that number doesn't appear in any table. No code, no baselines, no error bars. These aren't minor omissions; they make the headline result unverifiable.\n\nI also can't overlook the references. Many entries have the same suspicious page pattern (149:0402308x), author names like 'Min Lu and et al.', and 'Wei Zhang and et al.' These read like placeholders or fabricated citations. One of the authors' own papers is cited for SMOTE, which is odd. A reviewer would need to verify each one.\n\nWho is this for? Someone building a quick baseline for earthquake damage prediction might skim it, but they'd have to redo the whole pipeline themselves. As a research contribution, the paper is a parameter scan with an unclear evaluation. I'd desk reject it on the protocol gap alone; the reference list makes that decision easier. If the authors resubmit with a clear split-before-resampling statement, labeled tables, code, and a cleaned reference list, it could be a modest but citable benchmark. As is, I wouldn't send it to a serious referee.","headline":"A standard benchmark on a public dataset whose headline 96% accuracy is unverifiable because the evaluation protocol is missing and the references look unreliable.","tokens_in":8156,"tokens_out":2740,"would_cite":false,"duration_ms":28652,"reading_group":"no","serious_thinker":"no","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims a stacking ensemble of standard classifiers predicts earthquake damage grades with 96% test accuracy on the Gorkha Nepal building dataset, outperforming ten other algorithms.","keywords":["earthquake damage prediction","stacking ensemble","SMOTE","class imbalance","Gorkha earthquake dataset","XGBoost","machine learning","deep learning"],"falsifier":"Apply the pipeline with a train/test split fixed before SMOTE, undersampling, and feature selection, and compare test accuracy against a run where these steps see the full dataset; a large drop in the untouched-holdout run would show the 96% figure is inflated.","tokens_in":7109,"feed_emoji":"🏚️","tokens_out":5810,"duration_ms":52044,"temperature":0.7,"pith_summary":"This paper tries to show that a stacking ensemble of ordinary machine-learning classifiers can predict the damage grade of buildings after an earthquake with 96% test accuracy, using only building attributes such as age, construction materials, and floor type. The test bed is the dataset compiled after the 2015 Gorkha, Nepal, earthquake. The authors argue that handling class imbalance with SMOTE, selecting the top 20 features, and combining several classifiers through a meta-learner gives a practical tool for post-disaster triage and relief prioritization. If the result holds, a cheap, fast model could flag the most damaged buildings before inspectors reach them.","feed_headline":"Stacking ensemble hits 96% accuracy on quake damage grades","feed_subtitle":"Ten models face the 2015 Gorkha Nepal building data; the stacked classifier leads after SMOTE balancing.","key_machinery":"The load-bearing mechanism is the stacking ensemble: a set of base classifiers (Random Forest, Gradient Boosting, LightGBM, XGBoost, and others) each produce damage-grade predictions, and a meta-learner learns to combine those outputs into the final grade. Around it, the pipeline uses SMOTE (synthetic minority oversampling) plus RandomUnderSampler to rebalance the three damage classes, Isolation Forest to remove anomalous rows, and SelectKBest with the ANOVA F-test to keep the 20 most informative building features. The deep-learning alternatives are a feedforward neural network and a Kolmogorov-Arnold Network, which transforms each input dimension before combining them.","core_discovery":"The paper reports that a Stacking Classifier, which feeds predictions from Random Forest, GBM, LightGBM, XGBoost, and other base models into a meta-learner, achieves 96% test accuracy and 99.9% training accuracy, the best among the ten algorithms compared. Random Forest, GBM, LightGBM, Voting, and Bagging also reach roughly 96%, while Logistic Regression and AdaBoost lag. Among deep models, a feedforward network reaches 92.4% test accuracy and a Kolmogorov-Arnold Network reaches 94.39% test accuracy. The authors attribute the result to balancing the training data with SMOTE and RandomUnderSampler, removing anomalies with Isolation Forest, selecting the top 20 features by ANOVA F-test, and tuning hyperparameters.","pith_inferences":["A consequence the paper does not test: the reported 96% could be optimistic if resampling touched the test set, since the paper never states the split point; re-running with a strictly leakage-free split would settle it.","Because the dataset comes from one earthquake and one region, the pipeline's value for other seismic zones is an open question; a natural next experiment is applying it to damage data from a different earthquake.","The gap between 99.9% training accuracy and 96% test accuracy hints that the stacking model may be overfitting or memorizing, so a simpler model or stronger regularization might generalize just as well on new data.","One useful extension the paper leaves implicit: reporting per-class accuracy after balancing, since overall accuracy can hide poor performance on rare damage grades."],"forward_implications":["If the 96% test accuracy is genuine, post-earthquake teams could rank buildings by predicted damage grade from survey attributes alone, focusing field inspections on the most likely severe cases.","The same SMOTE-plus-ensemble pipeline is claimed to transfer to other natural calamities, giving a flexible modelling template for disaster response.","Feature selection points to building age, construction materials, foundation, roof, and ground-floor type as the attributes that matter most for seismic vulnerability.","The near-tie among stacking, Random Forest, GBM, and LightGBM suggests that a single strong tree-based model may capture most of the signal, with stacking adding a modest edge.","Deep learning models, while behind the best ensembles, still reach above 92% test accuracy, so the choice between ML and DL depends on interpretability and compute constraints."],"supporting_citations":[{"why":"Provides the post-2015 Gorkha Nepal building damage dataset that all models are trained and evaluated on.","marker":"Ghimire et al. [2022]"},{"why":"Cited as the source of the SMOTE-based balancing approach used to address class imbalance.","marker":"Yadav et al. [2023]"},{"why":"Demonstrates XGBoost's effectiveness on tabular earthquake damage data, the main baseline the ensemble must beat.","marker":"Lu et al. [2023]"},{"why":"Proposes a CNN approach for seismic damage prediction, the deep-learning baseline the paper contrasts with its ensembles.","marker":"Zhang and et al. [2023]"},{"why":"Supplies the scalable decision-tree method that motivates the tree-based classifiers used inside the ensembles.","marker":"Klusowski and Tian [2024]"}],"fun_headline_variants":["Stacked ML ensemble predicts quake damage at 96% accuracy","96% accuracy: stacking ensemble for earthquake damage grading","Stacking beats single models for quake damage grade prediction","Ensemble of 10 models nails earthquake damage grades at 96%","SMOTE-balanced stacking classifier tops 96% on Gorkha quake data"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that SMOTE and undersampling were applied only to the training portion, so the reported 96% test accuracy comes from a holdout set the balancing steps never saw; the paper does not say where that split occurs.","fun_headline_variants_meta":{"raw":{"variants":["Stacked ML ensemble predicts quake damage at 96% accuracy","96% accuracy: stacking ensemble for earthquake damage grading","Stacking beats single models for quake damage grade prediction","Ensemble of 10 models nails earthquake damage grades at 96%","SMOTE-balanced stacking classifier tops 96% on Gorkha quake data"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000636,"raw_usage":{"total_tokens":2921,"prompt_tokens":920,"completion_tokens":2001,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":536,"completion_tokens_details":{"reasoning_tokens":1911}},"tokens_in":536,"tokens_out":2001,"duration_ms":14497,"temperature":1.0,"reasoning_tokens":1911,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T22:09:14.381985+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Apply the pipeline with a train/test split fixed before SMOTE, undersampling, and feature selection, and compare test accuracy against a run where these steps see the full dataset; a large drop in the untouched-holdout run would show the 96% figure is inflated.","supporting_citations":[{"cited_title":"Effective ml-based quality of life prediction approach for dependent people in guardianship entities","cited_arxiv_id":null,"evidence_quote":"Cited as the source of the SMOTE-based balancing approach used to address class imbalance."},{"cited_title":"A cnn-based approach for earthquake damage prediction","cited_arxiv_id":null,"evidence_quote":"Proposes a CNN approach for seismic damage prediction, the deep-learning baseline the paper contrasts with its ensembles."},{"cited_title":"Large scale prediction with decision trees","cited_arxiv_id":null,"evidence_quote":"Supplies the scalable decision-tree method that motivates the tree-based classifiers used inside the ensembles."}],"review_version":1}