{"id":"6f99dfb2-dade-4d89-b107-aa9c8ebdde85","arxiv_id":"1908.07805","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"Models trained with forward feature selection guided by spatial cross-validation drop harmful geolocation predictors and yield more reliable spatial maps than standard approaches.","lead":"Spatial predictions from machine learning models can look excellent on paper but fail on new locations when geolocation variables like latitude and longitude are included. This two-case-study paper shows that spatially aware variable selection, not just spatially aware validation, is needed to produce trustworthy maps.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Spatial FFS models are evaluated on the same CV folds used for selection, so the reported gains (Kappa 0.55→0.56, RMSE 1.25→1.20) may be selection bias; an unbiased nested spatial CV estimate is needed.","rationale":"The reader's weakest assumption (LAI time lag) is a data-quality concern, but it mainly affects one of two case studies; the LULC demonstration would still support the central claim about geolocation overfitting. A more load-bearing issue is that the quantitative evidence for spatial variable selection—the small improvements in Table 3—is produced by evaluating the selected models on the same spatial folds used to select them. This selection bias directly undermines the paper's conclusion that spatial FFS improves statistical spatial performance. The paper even acknowledges the statistical similarity (Section 4.3), so the entire claim that spatial variable selection is 'essential' hinges on the biased numbers and subjective visual inspection. A nested spatial CV would settle whether the improvement is real. This is a methodological flaw, not a disagreement with consensus. I therefore keep the reader's CONDITIONAL verdict unchanged, with the additional condition that nested CV be reported.","tokens_in":18893,"tokens_out":6576,"duration_ms":63942,"concrete_test":"Recompute models 4a/4b using nested spatial CV: partition the study area into, e.g., 10 outer spatial blocks (or leave-one-cluster-out for LAI). Within each outer training set, rerun FFS with spatial CV on inner folds to select variables, then evaluate the resulting model on the outer test block. Aggregate the outer-fold predictions and compare the resulting Kappa/RMSE with the full model's spatial CV performance. If the nested estimate shows no improvement (ΔKappa ≈ 0, ΔRMSE ≈ 0), the statistical claim in Table 3 is selection bias.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim that spatial variable selection improves spatial prediction rests on the performance of models 4a/4b (Section 2.3, Table 3). For these models, forward feature selection (FFS) is run with spatial cross-validation (Section 2.3.2), and the selected variable set is then retrained and evaluated using the same spatial CV folds. This is a classic selection-bias error: the left-out folds were already used to choose the variables, so the reported spatial performance (Kappa 0.56, RMSE 1.20) is not an unbiased estimate of what the selected model would achieve on genuinely new spatial locations. The improvement over the full model is tiny (0.01 Kappa, 0.05 RMSE) and could disappear under a correct nested resampling design. The visual artifact argument remains, but the quantitative support for 'improved statistical spatial model performance' (abstract) is invalidated by the double use of the test folds. Without a nested spatial CV (outer folds for evaluation, inner folds for FFS), the claim that spatial variable selection is essential in addition to spatial validation is not supported by the statistical results.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper argues that standard machine learning practice for spatial prediction, which typically ignores spatial autocorrelation during model building, produces models that reproduce training data but fail when predicting beyond training locations. The authors use two case studies (land use/land cover classification and leaf area index regression) with Random Forests to compare random versus spatial cross-validation and several variable selection strategies, including recursive feature elimination, forward feature selection with random CV, and forward feature selection with spatial CV. They report that spatial CV strongly lowers estimated performance compared to random CV, that geolocation predictors such as latitude and longitude dominate variable importance and create visible artefacts in prediction maps, and that spatial forward feature selection removes these predictors and yields qualitatively more plausible maps with slightly better spatial CV statistics.","tokens_in":19098,"tokens_out":5838,"duration_ms":59696,"significance":"If the central claim is supported, the paper makes a useful methodological contribution: it demonstrates that spatial variable selection, not only spatial validation, should be part of spatial machine learning pipelines. The study is well positioned in the recent literature on spatial CV and benefits from two independent case studies, a negative control (random FFS and RFE) that performs worse, and publicly available code on GitHub. The visual artefact evidence from geolocation predictors is compelling and should be reproducible by readers. However, the quantitative support for the central claim is weakened by a selection-bias issue in the evaluation of the spatial FFS models and by a temporal mismatch in the LAI reference data, as detailed below.","major_comments":[{"comment":"The spatial performance of the spatial-FFS models is not an unbiased estimate because the same spatial CV folds are used both to select the variables and to evaluate the final model. In Section 2.3.2, FFS with spatial CV uses all folds to choose the variable subset; the subsequent re-training and spatial CV evaluation of models 4a/4b then tests on folds that already influenced the variable selection. This is a classic selection-bias error. The reported gains from spatial FFS are small (Kappa 0.55 to 0.56; RMSE 1.25 to 1.20; global R2 0.58 to 0.63) and are not accompanied by significance tests or confidence intervals, so they cannot support the abstract's claim of 'improved statistical spatial model performance.' Please re-run the analysis with nested spatial CV (outer folds for evaluation, inner folds for FFS), or, if the visual pattern assessment is the primary evidence, state that explicitly and temper the statistical claim accordingly.","section":"Section 2.3.2 and Tables 2-3, models 4a/4b"},{"comment":"The LAI reference data are from lidar acquired in 2010, while the Sentinel-2 predictors are from May 2017. The authors justify this by stating that no major management occurred, but natural forest growth, phenology, and small-scale disturbance over seven years can change LAI and canopy structure. If the lidar-derived LAI does not represent the forest state at the time of the Sentinel-2 image, the training labels are incorrect, making the reported RMSE and R2 values in Table 3 and the LAI-based conclusions in Sections 3.3 and 4.2 unreliable. Please either use temporally matching reference data, restrict the LAI case study to a sensitivity analysis, or clearly frame the LAI results as illustrative with this caveat prominently stated.","section":"Section 2.2.1 and Table 3"},{"comment":"The discussion states that the model using all predictors and the model using spatially selected variables 'perform equally well, statistically speaking' under spatial CV, while the abstract claims that spatial variable selection leads to 'improved statistical spatial model performance.' Given the small and untested differences in Tables 2 and 3, these two statements are in tension. The quantitative claim should be based on nested CV estimates (or equivalent unbiased evaluation) or explicitly downgraded to a qualitative/visual claim, so that the paper's conclusion is internally consistent.","section":"Section 4.3 and Abstract"}],"minor_comments":[{"comment":"The acronyms 'CV', 'FFS', and 'RFE' are used in Figure 4 before they are defined in the text; please define them in the figure caption or introduce them before the figure.","section":"Figure 4 and Section 2.3"},{"comment":"The colors used to indicate the 11 folds are not easily distinguishable in grayscale or for color-deficient readers; please use a colorblind-safe palette or add direct labels to the points.","section":"Figure 6"},{"comment":"The per-fold R2 values for spatial CV are near zero for all models, and although the text explains this as a consequence of low within-fold variability, the table itself is misleading without that caveat; consider reporting only global R2 or adding a footnote to the table.","section":"Table 3"},{"comment":"Several references are preprints or non-peer-reviewed versions (e.g., Valavi et al. 2018 bioRxiv, Hengl et al. 2018 preprint); please update to the published versions where available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The two main concerns (selection bias in the spatial FFS evaluation and the temporal mismatch in the LAI reference data) are fixable through re-analysis or reframing, so I do not see a need for rejection. The self-citation of the authors' CAST package is appropriate given that the FFS method is their own and is not a circularity problem in this demonstration."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nYou should know about arXiv:1908.07805 if you work on spatial prediction with machine learning. It gives a clean, reproducible demonstration that including coordinates and elevation as predictors leads to visible artifacts in Random Forest maps, and that a forward feature selection run with spatial cross-validation drops those predictors and cleans up the maps. The paper is a follow-up to the authors' own FFS paper (Meyer et al. 2018), so it is not a new method. The new material is two case studies (LULC classification and LAI regression) plus a comparison against RFE and random FFS. That comparison is the real value: RFE, even with spatial CV, does not remove latitude/longitude because they rank highly in variable importance; random FFS makes the artifacts worse. These are useful negative controls, and the visual evidence is striking. Code and data links are included.\n\nWhere I part company with the abstract is the quantitative claim. Models 4a/4b are built by running FFS inside spatial CV and then evaluated on the same spatial CV folds. That is double use of the test folds. The reported improvements (Kappa 0.55 to 0.56, RMSE 1.25 to 1.20, R2 0.58 to 0.63) are selection-biased and should not be read as unbiased estimates of spatial predictive performance. The paper needs an outer/inner nested resampling scheme (outer spatial CV for evaluation, inner spatial CV for FFS) before claiming improved statistical spatial model performance. The visual argument is independent and survives, but the headline statistical gain is not established.\n\nThe LAI case study has a second, smaller caveat: lidar reference data from 2010 are paired with Sentinel-2 predictors from 2017. The authors disclose this and argue the forest was stable; that is plausible but it makes the LAI comparison a bit fragile. If the canopy changed in those seven years, the labels are noisy and the variable-selection comparison for the regression task is weakened.\n\nOverall this is a serious, readable paper that makes a practical point: spatial validation alone does not fix a model that is overfitting to geolocation variables; you need spatial feature selection too. The demonstration is convincing visually, and the negative controls are good. The statistical support needs repair, and the novelty is incremental. But it deserves a serious referee. I would send it to review with a request for nested spatial CV, confidence intervals on the performance differences, and a more cautious wording of the abstract.\n\nBest","headline":"A useful practical demonstration that spatial feature selection removes geolocation artifacts, but the statistical gains are overclaimed because selection and evaluation share the same folds.","tokens_in":19615,"tokens_out":3461,"would_cite":true,"duration_ms":33690,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Machine-learning models for spatial prediction can reproduce training data yet fail to map anywhere else, and the fix is to run predictor selection with spatial cross-validation.","keywords":["spatial cross-validation","variable selection","machine learning","Random Forests","spatial prediction","overfitting","remote sensing","leaf area index"],"falsifier":"On a spatially clustered dataset with coordinates and elevation as candidate predictors, train Random Forest with and without spatial forward feature selection and compare spatial block-CV predictions; if the full model shows no linear map artefacts and no worse spatial CV error than the spectrally selected model, the central claim that geolocation predictors cause spatial overfitting would fail.","tokens_in":18665,"feed_emoji":"🗺️","tokens_out":5238,"duration_ms":51476,"temperature":0.7,"pith_summary":"The paper argues that machine learning models trained for spatial prediction fail when the training data are spatially clustered and the predictor set includes variables that are strongly autocorrelated in space, such as latitude, longitude, or elevation. Standard random cross-validation can report near-perfect performance for such models, while spatial cross-validation reveals much lower skill, and the prediction maps show artificial linear features that trace back to the geolocation predictors. The paper claims that, in addition to spatial validation, a spatial variable selection—specifically forward feature selection run with spatial cross-validation—is needed to automatically remove these misleading predictors. In two remote-sensing case studies over the Marburg Open Forest, spatial variable selection removed coordinates and elevation, improved visual prediction patterns, and modestly improved spatial validation performance. If true, this means many published spatial machine-learning maps built with coordinates or other location-derived predictors and validated only randomly may reproduce training data while being unreliable beyond the training sites.","feed_headline":"Machine-learning maps need spatial variable selection","feed_subtitle":"Random validation hides overfitting that only spatially aware predictor choice removes.","key_machinery":"The load-bearing mechanism is forward feature selection (FFS) driven by spatial cross-validation. FFS starts from the two-predictor model with the best spatial validation score, then adds predictors one at a time only if they improve that score; variables that do not help spatial prediction are excluded. Because the cross-validation folds are spatial blocks or held-out clusters, a variable that only memorizes training-location patterns cannot earn a place in the model. This contrasts with Random Forest's internal out-of-bag importance and with recursive feature elimination, which rank variables by how well they fit the training data and therefore keep the very geolocation variables that cause spatial overfitting.","core_discovery":"The central discovery is that Random Forest models trained on spatially clustered field data treat highly autocorrelated geolocation variables (latitude, longitude, elevation) as if they carried environmental signal, when in fact they encode the locations of the training clusters. The result is a model that scores above 0.99 Kappa or an R-squared near 0.87 under random cross-validation but drops to 0.55 Kappa or an R-squared of 0.58 under spatial block cross-validation, and it produces maps with linear boundaries that do not match the underlying imagery. Recursive feature elimination does not fix this, because those misleading variables rank as the most important according to the algorithm's training-based importance measure. Forward feature selection evaluated with spatial cross-validation automatically drops the geolocation variables, leaves only spectral predictors (plus slope for land cover), removes the linear artefacts, and slightly raises spatial validation performance (Kappa 0.56; global R-squared 0.63 for leaf area index). The paper concludes that spatial validation alone is insufficient: the variable selection step itself must use spatial cross-validation.","pith_inferences":["The failure mode identified here should generalize to any flexible learner, not just Random Forests, whenever training samples are spatially clustered and predictors include smoothly varying location fields.","The benefit of spatial variable selection is likely to scale with the strength of spatial autocorrelation and the degree of clustering; on near-random sampling designs, the gap between random and spatial FFS should shrink, a prediction that could be tested by simulation.","Linear map artefacts caused by geolocation predictors could be turned into an automated diagnostic: scanning prediction maps for axis-aligned discontinuities as a cheap screen for spatial overfitting.","The same spatial-selection logic applies to temporal or spatio-temporal prediction, where date or time-index variables could play the role of latitude and longitude."],"forward_implications":["Spatial cross-validation should be part of every step of spatial machine-learning modelling, including predictor selection and hyperparameter tuning, not just final validation.","Recursive feature elimination, common in environmental mapping, will not remove geographically misleading predictors because it ranks variables by training importance.","Including coordinates or other location-derived fields as predictors in spatially clustered designs risks producing maps with artificial linear boundaries, even when overall accuracy statistics look acceptable under random validation.","Statistical spatial performance alone cannot distinguish a reliable spatial model from one with artefacts; visual inspection of prediction maps is needed in practice.","Spatial FFS changed the prediction maps considerably even where statistical performance changed only slightly, so quantitative metrics alone would miss the improvement."],"supporting_citations":[{"why":"Supplies the forward feature selection method with user-defined cross-validation and the spatio-temporal context that the paper extends to spatial-only prediction.","marker":"(Meyer et al., 2018)"},{"why":"Provides the spatial block cross-validation strategy used for the land-cover classification case study.","marker":"(Roberts et al., 2017)"},{"why":"Supplies the spatial block CV implementation and recommendation used to build the spatial folds.","marker":"(Valavi et al., 2018)"},{"why":"Defines the Random Forest algorithm that is the machine learning model throughout both case studies.","marker":"(Breiman, 2001)"},{"why":"Provides the general cross-validation and recursive feature elimination protocols that the paper adapts and critiques.","marker":"(Kuhn & Johnson, 2013)"},{"why":"Shows spatial leave-one-out cross-validation for variable selection in the presence of spatial autocorrelation, a key background for the spatial FFS approach.","marker":"(Le Rest et al., 2014)"},{"why":"Exemplifies the use of Euclidean distance fields as geolocation predictors that the paper argues cause spatial overfitting.","marker":"(Behrens et al., 2018)"},{"why":"Supports the conclusion that spatial dependencies in predictors harm spatial predictions of plant traits.","marker":"(Rocha et al., 2018)"}],"fun_headline_variants":["Spatial variable selection beats random CV in ML maps","Random Forest maps overfit without spatial predictor selection","Spatial CV alone won't fix overfit; add spatial variable selection","Geolocation predictors mislead Random Forest spatial maps","Spatial variable selection critical for reliable ML predictions"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The leaf-area-index demonstration assumes that lidar-derived leaf area index from 2010 is a valid reference for predictions built on 2017 Sentinel-2 imagery, because no major forest management occurred in between; if the canopy changed during those seven years, the regression labels would be partly wrong and the comparison of variable-selection strategies would be unreliable.","fun_headline_variants_meta":{"raw":{"variants":["Spatial variable selection beats random CV in ML maps","Random Forest maps overfit without spatial predictor selection","Spatial CV alone won't fix overfit; add spatial variable selection","Geolocation predictors mislead Random Forest spatial maps","Spatial variable selection critical for reliable ML predictions"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000249,"raw_usage":{"total_tokens":1594,"prompt_tokens":1035,"completion_tokens":559,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":651,"completion_tokens_details":{"reasoning_tokens":478}},"tokens_in":651,"tokens_out":559,"duration_ms":16620,"temperature":1.0,"reasoning_tokens":478,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-14T11:55:33.809285+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"On a spatially clustered dataset with coordinates and elevation as candidate predictors, train Random Forest with and without spatial forward feature selection and compare spatial block-CV predictions; if the full model shows no linear map artefacts and no worse spatial CV error than the spectrally selected model, the central claim that geolocation predictors cause spatial overfitting would fail.","supporting_citations":[],"review_version":1}