{"id":"2fd67db8-cf20-49d7-8d05-3d21713037b8","arxiv_id":"2501.16900","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A grid-search-tuned ensemble framework with new temperature and humidity difference features claims 93.8% accuracy for next-day rainfall prediction, but the evaluation may be optimistically biased by target-informed preprocessing.","lead":"This paper proposes RAINER, a preprocessing and model-selection pipeline for predicting next-day rain from Australian weather records, and reports that Random Forest with hand-constructed temperature and humidity difference features reaches about 94% accuracy. A generalist might read it as a recipe for applying grid search and ensemble voting to tabular weather data, but the evaluation protocol raises concerns about inflated performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Target leakage from pre-split balancing and full-data PCA/feature selection likely inflates the reported test accuracy.","rationale":"The reader's weakest assumption identifies the same load-bearing concern: target leakage through pre-split balancing, full-data PCA, and correlation-based feature selection. The manuscript explicitly describes these steps before the split, so the concern is concrete and grounded in the text. If the leakage is real, every test metric in Table 3 is optimistically biased, and the 'state-of-the-art' claim loses support. The concrete test I propose would settle the matter by rerunning the pipeline with preprocessing confined to the training fold. Since the reader already recommends rejection and my analysis supports that recommendation, I keep the verdict unchanged. I do not see a more fundamental flaw: the paper's systematic model comparison would be useful if the numbers were trustworthy, and the leakage issue is the single most load-bearing threat to that trust.","tokens_in":35699,"tokens_out":3469,"duration_ms":32039,"concrete_test":"Re-run the RAINER pipeline with a leakage-free protocol: split the raw data 80/10/10 first, then fit imputation, outlier capping, feature construction and correlation analysis, balancing, and PCA on the training fold only, and apply the fitted transforms to the test fold. Compare the Random Forest accuracy and AUC under 'Selected + Constructed Features' against Table 3; if the gap exceeds 2 percentage points, the reported gains are not trustworthy as generalization estimates.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim of state-of-the-art performance rests on test metrics that are not honest out-of-sample estimates. The paper's preprocessing pipeline uses test-set labels and test-set feature values before the 8:1:1 split. Section 4.4.1 states that the dataset is balanced by preserving all positive samples and randomly selecting an equal number of negative samples; this balancing is applied to the full dataset before the split described in Section 5. Section 4.4.2 shows PCA biplots for the balanced dataset, indicating PCA is fit on the full balanced data. Section 4.2 computes correlations with RainTomorrow on the full data to justify feature construction (Figure 13). Consequently, the test fold is not independent: its labels influence which features are kept, how the data are balanced, and how PCA components are derived. Any grid-search or model selection then operates in a feature space already shaped by the test distribution. This leaks target information into every reported number in Table 3, including the headline 93.8% accuracy and 93.8% AUC for Random Forest under 'Selected + Constructed Features'. If the pipeline were rerun with all data-dependent steps confined to the training fold, the reported metrics would likely drop, undermining the 'state-of-the-art' claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents RAINER, a rainfall-prediction framework for the Australian Bureau of Meteorology dataset. The pipeline removes high-missingness features, imputes missing values, caps outliers, constructs temperature/humidity difference features, balances the class distribution, applies PCA, and then trains a broad set of classifiers and neural networks with grid-search tuning and voting ensembles. The main empirical claim, stated in the abstract and Section 6.2, is that this framework achieves state-of-the-art results, with Random Forest reaching 93.8% accuracy and 93.8% AUC under the 'Selected + Constructed Features' strategy.","tokens_in":35898,"tokens_out":6624,"duration_ms":63414,"significance":"The paper has useful breadth: it compares many models, including recent architectures such as KAN, and systematically explores feature-engineering strategies and hyperparameter settings on a well-known public dataset. If the evaluation were valid, the model comparison and the emphasis on grid-search tuning and ensemble voting would be a useful reference for practitioners. However, the load-bearing evaluation protocol is not a valid out-of-sample assessment: class balancing, feature selection, and PCA are performed on the full dataset before the train/test split, so the test labels and test feature values influence the reported metrics. In addition, no external or published baseline is used, so the state-of-the-art claim is unsupported. The reported numbers in Table 3 cannot currently be interpreted as honest estimates of predictive performance.","major_comments":[{"comment":"The class-balancing step is applied to the full dataset before the 8:1:1 split. The text states that 'all positive samples are preserved, and an equal number of negative samples are randomly selected from the remaining data.' Because this selection uses RainTomorrow labels from the entire dataset, the eventual test fold is not independent: its class ratio is forced to 50/50, and its composition depends on labels that should be unseen. Every test metric in Table 3, including the headline 93.8% accuracy and AUC of Random Forest under 'Selected + Constructed Features', is therefore not a valid out-of-sample estimate. The protocol must split first, then balance only the training portion, and evaluate on an untouched test set.","section":"Section 4.4.1 and Section 5"},{"comment":"Feature selection and dimensionality reduction use the full dataset. The correlation weights in Figure 13 are computed against RainTomorrow on all records and are used to justify retaining and constructing features; PCA is fit on the full and on the full balanced dataset, and the PCA-based feature strategies in Table 3 are evaluated on components learned from data that include the test fold. The mean/mode imputation statistics and outlier-capping thresholds in Sections 4.1.1 and 4.1.2 also appear to be estimated from the full dataset. All such data-dependent steps must be learned from the training fold only and then applied to the test fold; otherwise test information leaks into feature and model selection.","section":"Sections 4.2, 4.4, and 4.4.2"},{"comment":"The claim of 'state-of-the-art results' is unsupported by the experiments. Table 3 compares only models run in this paper; there is no comparison with previously published results on the same Australian Bureau of Meteorology dataset, no external baseline, and no defined protocol (such as repeated runs with variance or a fixed public benchmark) against which 'state-of-the-art' can be judged. A valid comparison to published work is needed before any SOTA claim can be made.","section":"Abstract and Section 6.2"},{"comment":"The 8:1:1 split ratio is selected based on the authors' 'pre-exploration' experiments comparing different split ratios. If the same test fold was used to choose this ratio, then the test set has been used for model selection. The paper should clarify whether the comparison was performed on a separate validation set and, if not, use nested validation or a fixed hold-out that is never used for any design choice.","section":"Section 5"}],"minor_comments":[{"comment":"The record count is inconsistent: Section 3 states 145,460 records, while Section 4.4 states n = 200,000 daily records. The number of features is also given as 19 in Section 4.1 and as 17 in Section 4.4; these figures should be reconciled.","section":"Sections 3 and 4.4"},{"comment":"Equations (1), (2), and (3) are not connected to the analysis: Equation (1) is a simple linear regression, Equation (2) is Bayes' rule, and Equation (3) is a set notation for a dataset. None is used or tested; either use them substantively or remove them.","section":"Section 4.2"},{"comment":"The term 'correlation weights' is not defined. The paper should state how the values (e.g., 13.44% for MaxDifferenceTemp) are computed and how they support the feature-selection decision.","section":"Figure 13"},{"comment":"The outlier-capping thresholds (Rainfall above 3.2 mm, WindSpeed9am above 55 km/h, WindSpeed3pm above 57 km/h) are given without justification. For Rainfall, a cap of 3.2 mm appears to remove much of the dynamic range relevant to the target variable; the choice should be justified or tested.","section":"Section 4.1.2"},{"comment":"Table 3 contains stray markup characters such as colons and dotted underlines and is very hard to read; it should be regenerated with clean formatting.","section":"Table 3"},{"comment":"The axis labels in Figures 10 and 11 contain garbled Unicode tokens such as '/uni00000014'; the figures should be regenerated.","section":"Figures 10 and 11"}],"recommendation":"reject","confidential_remarks":"The central empirical claim is invalidated by the leakage issue and the absence of external baselines. I would encourage the authors to re-run the full pipeline with a leakage-free protocol, evaluate on an untouched test set, and compare against published results on the same dataset before resubmitting."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The paper is a wide benchmark of ML models on the standard BoM 'Rain in Australia' dataset, with a feature-engineering pipeline (missing-value imputation, outlier capping, constructed temperature/humidity differences, PCA, class balancing) and grid-search tuning. The headline claim — 93.8% accuracy and AUC for Random Forest under the 'Selected + Constructed Features' strategy — should not be read as an honest out-of-sample result. The evaluation leaks target information: the data are balanced by under-sampling negatives before the 8:1:1 split (Section 4.4.1), PCA is fitted on the full balanced dataset (Section 4.4), and the constructed features are justified by correlations with RainTomorrow computed on all records (Figure 13). The test fold is therefore not independent of the decisions that shape the model inputs.\n\nWhat is genuinely useful is the systematic comparison of many model families — LDA, LASSO, random forest, KAN, transformer ensembles, and others — under one pipeline. For a practitioner deciding which classifiers to try on tabular weather data, the relative rankings are a reasonable starting point, and the preprocessing steps are documented in enough detail to reproduce.\n\nThe soft spots beyond leakage: no external baselines, so the 'state-of-the-art' claim is unsupported; no code or data released; no error bars or repeated runs; and Table 3 has internal inconsistencies (the LSTM AUC column looks misaligned). The free parameters — outlier thresholds, balancing ratio, number of PCA components — are chosen without sensitivity analysis, so we do not know how robust the rankings are.\n\nIf the authors re-run the pipeline with all data-dependent steps confined to the training fold and benchmark against published results, this could become a solid empirical reference. In its current form, the central result fails, so I would reject it. But the breadth of the benchmark makes it worth a major revision rather than a desk-reject: the leakage is fixable, and the comparative table would be useful once the numbers are honest.","headline":"Broad but leaky benchmark; headline results are not out-of-sample.","tokens_in":36478,"tokens_out":5196,"would_cite":false,"duration_ms":43642,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Grid-searched random forest hits 93.8 percent accuracy on next-day rain classification.","keywords":["rainfall prediction","feature engineering","grid search","ensemble learning","random forest","principal component analysis","voting classifier","Kolmogorov-Arnold Networks"],"falsifier":"Build the identical pipeline but confine outlier thresholds, feature-construction choices, PCA transformation, and under-sampling to the training fold alone, then evaluate on the untouched test fold; if accuracy and AUC fall materially below 93.8 percent, the paper's core claim depends on test information leaking into the model-building process.","tokens_in":35480,"feed_emoji":"🌧️","tokens_out":3914,"duration_ms":33201,"temperature":0.7,"pith_summary":"The paper proposes RAINER, a rainfall-prediction pipeline that combines outlier capping, missing-value imputation, engineered features such as temperature and humidity differences, PCA-based dimensionality reduction, grid-search hyperparameter tuning, and voting ensembles across weak and deep classifiers. Its central claim is that this systematic pipeline achieves top results on the Australian Bureau of Meteorology dataset, with a tuned Random Forest reaching 93.8 percent accuracy and 93.8 percent AUC for next-day rain classification. The authors argue that for structured, binary weather data, carefully preprocessed simple classifiers outperform newer deep architectures such as transformers and Kolmogorov-Arnold Networks.","feed_headline":"Random forest hits 93.8% accuracy for next-day rain","feed_subtitle":"On Australian Bureau of Meteorology data, tuned RF and voting ensembles beat neural nets like KAN and LSTM.","key_machinery":"The load-bearing mechanism is the preprocessing pipeline applied before any model is trained: high-missingness columns are dropped, outliers in Rainfall, WindSpeed9am and WindSpeed3pm are capped, missing values are mean- or mode-imputed, and new features are constructed as the maximum difference between daily temperature and humidity readings, with the original columns removed. PCA then reduces the 17-dimensional feature space and the dataset is balanced by randomly under-sampling negative labels. Grid search, guided by a preliminary ROC-based exploration of parameter ranges, tunes every classifier, and voting ensembles combine three models at once.","core_discovery":"On the paper's own terms, the discovery is that the combination of feature construction and systematic grid search lets a Random Forest classifier beat every other method on the benchmark, including neural-network ensembles. The reported numbers put RF at 93.8 percent accuracy and 93.8 percent AUC under the 'Selected + Constructed Features' strategy, with voting ensembles like DT+LR+RF achieving a 97.3 percent recall alongside high precision.","pith_inferences":["The headline figures likely depend on target-informed preprocessing being done before the 8:1:1 split, so a fully nested pipeline that learns thresholds, PCA, and balancing only from the training fold would probably report lower test metrics.","A natural test is to apply the same pipeline to other weather datasets with stronger seasonal or geographic structure, where the constructed 'difference' features may matter more or less.","The comparison would be stronger with an explicit baseline of the same models trained on raw features alone, without any target-aware construction."],"forward_implications":["If the reported numbers hold, a tuned Random Forest is the best published next-day rain classifier on this dataset.","The results imply that domain-informed constructed features preserve predictive information that PCA-only reductions discard.","They also imply that deep sequence models and Kolmogorov-Arnold Networks add computation without accuracy gains on this structured tabular task.","The pipeline's voting ensembles show that precision-recall trade-offs can be stabilised by combining divergent classifiers.","The feature-engineering recipe could transfer to other binary tabular forecasting tasks."],"supporting_citations":[{"why":"Provides the principal component analysis method used to reduce the 17-dimensional feature space.","marker":"[84]"},{"why":"Supplies the t-SNE method used to validate that the processed features separate rain and no-rain classes.","marker":"[83]"},{"why":"Introduces Kolmogorov-Arnold Networks, the advanced neural baseline that RAINER compares against.","marker":"[45]"},{"why":"Provides the imbalanced-learn toolbox used for under-sampling the negative class to balance the dataset.","marker":"[82]"},{"why":"Prior rainfall prediction work using dimensionality reduction that RAINER extends with its feature-construction pipeline.","marker":"[39]"}],"fun_headline_variants":["Rain prediction: tuned random forest tops neural nets","Grid search boosts rain forecast to 93.8% accuracy","Random forest wins rain battle with 93.8% AUC","Voting ensemble recalls 97.3% of rainy days","RAINER framework: RF beats KAN and LSTM"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"Every preprocessing and feature-selection decision, including which features are most correlated with next-day rain, the PCA fit, and the class balancing, is computed on the full dataset before the data is split into training and test sets.","fun_headline_variants_meta":{"raw":{"variants":["Rain prediction: tuned random forest tops neural nets","Grid search boosts rain forecast to 93.8% accuracy","Random forest wins rain battle with 93.8% AUC","Voting ensemble recalls 97.3% of rainy days","RAINER framework: RF beats KAN and LSTM"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000656,"raw_usage":{"total_tokens":2949,"prompt_tokens":836,"completion_tokens":2113,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":452,"completion_tokens_details":{"reasoning_tokens":2030}},"tokens_in":452,"tokens_out":2113,"duration_ms":15734,"temperature":1.0,"reasoning_tokens":2030,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T05:44:38.203710+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Build the identical pipeline but confine outlier thresholds, feature-construction choices, PCA transformation, and under-sampling to the training fold alone, then evaluate on the untouched test fold; if accuracy and AUC fall materially below 93.8 percent, the paper's core claim depends on test information leaking into the model-building process.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the principal component analysis method used to reduce the 17-dimensional feature space."},{"cited_title":"van der Maaten, G","cited_arxiv_id":null,"evidence_quote":"Supplies the t-SNE method used to validate that the processed features separate rain and no-rain classes."},{"cited_title":"Lemaitre, F","cited_arxiv_id":null,"evidence_quote":"Provides the imbalanced-learn toolbox used for under-sampling the negative class to balance the dataset."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior rainfall prediction work using dimensionality reduction that RAINER extends with its feature-construction pipeline."}],"review_version":1}