{"id":"68f2787e-c447-4bf9-bc3d-6062c6384844","arxiv_id":"2507.10993","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":3.0,"correctness_risk":"high","formal_verification":"none","parameter_count":3,"one_line_summary":"The paper reports 85% average accuracy for bird presence prediction using ResNet on satellite imagery and random forests on climate data, but the evaluation is undermined by synthetic absence labels and missing uncertainty.","lead":"A student project applies convolutional neural networks to satellite images and random forests to climate data to predict whether four bird species are present at a location. It reports roughly 85 percent accuracy, but the study's synthetic absence labels and missing error bars make the headline number shaky.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 85% accuracy claim rests on pseudo-absence labels generated by a 1.1 km buffer around eBird observations; these are not true absences, so accuracy and AUC measure label construction rather than habitat suitability.","rationale":"The reader's strongest claim and weakest assumption point to the same issue, and I agree that the pseudo-absence construction is the load-bearing weakness. The abstract's 85% figure is also not supported by the reported per-model results — the custom CNN is around 61% test accuracy and GBT can be as low as 73% — so the summary is selective even on its own terms. The deeper problem is that the evaluation would be invalid even if the numbers were consistent: binary classification with synthetic negatives derived from a 1.1 km buffer does not measure habitat suitability. The paper's own admission that pseudo-absence data might not accurately reflect where birds may be absent is decisive. Because no true absence data, spatial blocking, or uncertainty estimates are provided, the central claim is not established. The proposed concrete test, using eBird complete checklists as real absence points, would settle whether the 85% figure is an artifact. Since the reader already rejected on these grounds, this stress-test leaves the verdict unchanged.","tokens_in":10575,"tokens_out":4743,"duration_ms":53594,"concrete_test":"Use eBird complete checklists as a true absence benchmark: for each target species, extract checklists from the same region and season in which the species was not detected despite recorded survey effort, and use these as held-out negative locations (or retrain the models on them). Recompute random forest and ResNet accuracy and AUC on this benchmark. If average accuracy drops substantially below 85% or AUC approaches 0.5, the headline is an artifact of synthetic pseudo-absences. As a complementary check, run spatial block cross-validation on the original pseudo-absence dataset; if accuracy collapses under blocking, the model exploited spatial autocorrelation rather than habitat.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim — that both systems predict bird distributions with average 85% accuracy — depends on the absence labels being ecologically meaningful. They are not. In Methods, the paper generates pseudo-absent observations by checking whether a location is not too close to a known observed location for that bird within a 1.1 km radius. A point 1.2 km from a sighting, in identical habitat, is therefore labeled absent. That is not a true absence; it encodes eBird sampling effort and an arbitrary distance threshold. Each species is trained on 250 presence and 250 pseudo-absence points, so the model only has to separate known detections from arbitrary distant points. Since eBird detections cluster near accessible, well-surveyed areas and pseudo-absences are not matched for observer effort or habitat availability, the classifier can reach high accuracy by learning detection bias and geographic autocorrelation rather than habitat suitability. The paper itself concedes this in Results: 'our pseudo-absent dataset is being generated and might not accurately reflect where birds may be absent.' Every accuracy and AUC in Table 1 and the CNN section is computed against these synthetic negatives, so the 85% claim is not evidence of predictive skill for real species distributions. Even if the reported numbers were internally consistent, they would not establish the central claim.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes two approaches to predict bird species presence: a tabular pipeline using eBird occurrence data and WorldClim environmental rasters with random forest and gradient boosting classifiers, and CNN models (a fine-tuned ResNet-34 and a custom CNN) applied to Sentinel-2 satellite imagery. The datasets are constructed by pairing eBird presence records with pseudo-absence points generated by a 1.1 km buffer around observed locations, and the models are evaluated on a random 70/10/20 split. The paper reports roughly 85% test accuracy for the random forest models and 91% for the ResNet, but only 61% for the custom CNN, and the abstract claims that both systems predict bird distribution with an average accuracy of 85%.","tokens_in":10840,"tokens_out":4957,"duration_ms":52212,"significance":"If the reported results were valid, the paper would offer a useful comparison of deep learning and tree-based methods for bird habitat modeling, and it does provide pseudocode, a code repository, and clear descriptions of the data sources. However, the central claim is compromised by the use of synthetic pseudo-absence labels that are not true absences, a random split that ignores spatial autocorrelation, and inconsistent reporting of accuracy across the two systems. The paper also does not actually model migration or range shifts despite the title, because all inputs are static presence observations and current climate rasters with no temporal dimension. As a result, the reported accuracy and AUC numbers do not support the stated conclusions.","major_comments":[{"comment":"The abstract states that 'both systems predict the distribution of birds with an average accuracy of 85%,' but the reported numbers do not support this. Table 1 shows the random forest test accuracy averages about 85% across the four species, while the gradient boosting test accuracy averages about 81%, and Figure 7 reports the custom CNN at 61% accuracy and the ResNet at 91%. A truthful summary would need to distinguish among these models, and 'both systems' cannot be said to achieve 85%.","section":"Abstract; Results (Convolutional Neural Network)"},{"comment":"The pseudo-absence points are generated by selecting locations more than 1.1 km from any eBird observation for a species. This labeling is not a valid absence record: a location 1.2 km away can be identical habitat, and eBird observations are biased toward accessible, well-surveyed areas. Consequently, the classifier can separate known detections from arbitrary distant points by learning sampling bias and spatial autocorrelation rather than habitat suitability. The paper itself acknowledges in the Results section that the pseudo-absent dataset 'might not accurately reflect where birds may be absent.' Because every accuracy and AUC value in Table 1 and the CNN section is computed against these synthetic negatives, the reported performance does not establish predictive skill for real species distributions.","section":"Datasets (eBird Dataset); Statistical Models"},{"comment":"The random 70/10/20 split ignores spatial autocorrelation in the environmental rasters and satellite imagery. Nearby points share covariates and are not independent, so a random split inflates test performance. The paper cites blockCV (reference [24]) as a tool for spatially separated cross-validation but does not use it. Without spatial or environmental blocking, the test accuracy figures in Table 1 and Figure 7 cannot be taken at face value.","section":"Statistical Models (first paragraph)"},{"comment":"The title and framing promise modeling of habitat shifts and migration, but the experiments are static presence/absence classification at current locations. The dataset contains observation dates from eBird, yet the methods do not use time, and the WorldClim rasters represent current climate. There is no future climate scenario, no temporal split, and no prediction of range shifts. This discrepancy between the central claim and the scope of the experiments cannot be repaired by reporting additional metrics within the current design.","section":"Introduction; Conclusion"}],"minor_comments":[{"comment":"'Ovverall' is a typo for 'overall,' and the same section uses 'wholistically' instead of 'holistically.'","section":"Results (Convolutional Neural Network)"},{"comment":"The reference list contains duplicates: [12] and [32] are both Cutler et al. 2007, [14] and [25] are both He et al. 2015/2016, and [15] and [38] are both the SatBird paper; the citation style is also inconsistent between numeric and author-year formats.","section":"References"},{"comment":"Algorithm 6 does not specify the spatial dimensions after each convolutional block, so the feature map sizes that lead to the final 512-dimensional vector are unclear; a reader cannot reproduce the architecture without this detail.","section":"Algorithm 6"},{"comment":"The sentence 'For each species, we sample 250 presence and 250 pseudo-absence statistics respectively' should read '250 presence and 250 pseudo-absence observations.'","section":"Results (Tabular Data)"},{"comment":"The paper states that a threshold θ (usually 0.5) is applied on the validation set, but no threshold values or sensitivity analysis are reported, making it unclear how the threshold was tuned.","section":"Statistical Models (Tabular Model)"},{"comment":"Figure 7 is described as a comparison of ResNet and CNN across all performance metrics, but no confidence intervals or repeated-run statistics are provided, making it impossible to judge the stability of the 91% versus 61% gap.","section":"Results (Convolutional Neural Network)"}],"recommendation":"reject","confidential_remarks":"The manuscript has the character of a course project report rather than a complete research paper. The central result is not credible because the pseudo-absence labels and the random spatial split invalidate the evaluation. In addition, the claimed 'migration prediction' is not actually implemented. I do not see a path to acceptance within the scope of a standard journal paper, because fixing the evaluation would require a substantial redesign of the data labeling and experimental protocol, and would likely change the conclusions. The paper may be more suitable for a workshop or a preprint after those revisions."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nShort take: this is a straightforward application of ResNet transfer learning and random forests to predict bird presence from Sentinel-2 imagery and climate rasters. The abstract's claim that \"both systems predict the distribution of birds with an average accuracy of 85%\" is not supported by the results table. The custom CNN gets 61% test accuracy, the random forest ranges from 73% to 90% depending on species, and only the ResNet transfer model lands near 91%. The 85% figure is a selective summary that ignores the large performance gap between the two CNN approaches.\n\nWhat is actually new is not much. SatBird already combines Sentinel-2 imagery with eBird data for species distribution modeling, and the authors cite it. The contribution reduces to custom implementations of standard classifiers and a fairly obvious comparison between a pre-trained ResNet and a from-scratch CNN. That is incremental, which is fine as long as the evaluation is sound.\n\nThe paper has some positive features. The writing is clear, the pseudo-code is helpful, and the authors openly acknowledge the biggest weakness in their own results section: the pseudo-absence data \"might not accurately reflect where birds may be absent.\" They also flag the small dataset and the limitation of RGB-only bands. That candor is worth something.\n\nThe soft spots are load-bearing, though. The pseudo-absence points are generated by marking anything more than 1.1 km from an eBird observation as absent for that species. That is not a true absence; it encodes sampling effort and an arbitrary distance cutoff. Since eBird detections cluster near accessible areas, models can achieve high accuracy by learning detection bias and geographic autocorrelation rather than habitat suitability. The paper concedes this, but the central claim still rests on those labels. There are also no confidence intervals or repeated runs, no spatial blocking despite citing blockCV, and no comparison against the SatBird baseline they mention. The test set is small—250 presence and 250 pseudo-absence points per species, so roughly 100 test points per species.\n\nIs this worth a referee? I don't think so. The method is conventional, the evaluation does not support the headline, and the authors already know the main issue. It reads like a solid class project or a preliminary tech report. To become a serious paper, they would need real absence data or a well-justified pseudo-absence scheme, spatial cross-validation, uncertainty estimates, and an honest per-model summary. As it stands, I'd desk reject but encourage the authors to rework and resubmit.\n\nBest,\n[Your name]","headline":"Conventional bird-presence modeling whose headline accuracy is undercut by its own table and by pseudo-absence labels the authors themselves admit are questionable.","tokens_in":11337,"tokens_out":2876,"would_cite":false,"duration_ms":30879,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":false},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper argues that satellite imagery plus environmental rasters let two standard machine-learning models predict bird presence with about 85% average accuracy.","keywords":["bird species distribution modeling","convolutional neural networks","random forest","pseudo-absence","satellite imagery","climate data","habitat shift","eBird"],"falsifier":"Collect independent absence records for the same four species, for example repeated standardized point counts at locations the model labels absent, and compare the model's predictions against those real absences. If accuracy at locations the model calls absent falls far below the reported 80 to 90 percent, the pseudo-absence labels are not true absences and the reported performance is an artifact of label construction.","tokens_in":10341,"feed_emoji":"🐦","tokens_out":3944,"duration_ms":45746,"temperature":0.7,"pith_summary":"The paper tries to establish that bird habitat suitability can be predicted automatically from publicly available data: satellite images fed to a convolutional network and climate/topography rasters fed to a random forest. Using eBird observations for four North American species, it reports that both an image-based model and a tabular model reach roughly 85 percent average accuracy, with the ResNet-based vision model at about 91 percent and the random forest between 73 and 90 percent depending on species. If right, this offers a scalable alternative to field surveys for tracking range shifts under climate change. The paper also finds that the custom CNN trained from scratch underperforms, and that combining both data modalities is the natural next step.","feed_headline":"Two model types predict bird presence with 85 percent accuracy","feed_subtitle":"Satellite images feed a ResNet while climate rasters feed a random forest, mapping habitats for four bird species automatically.","key_machinery":"The argument is carried by two paired classifiers over the same locations. The image side is a transfer-learned ResNet-34 on Sentinel-2 RGB tiles, whose pretrained features are adapted through a dropout-equipped classification head to output species-presence probabilities; the tabular side is a random forest over elevation, precipitation, temperature, and coordinates, trained on presence points from eBird plus pseudo-absence points generated more than 1.1 km from any observation. The pseudo-absence construction is what turns presence-only citizen science data into a binary classification problem, and both classifiers share the same accuracy and AUC evaluation protocol.","core_discovery":"The central claim is that standard deep-learning and tree-based classifiers can map bird presence from two complementary representations of a location: Sentinel-2 RGB imagery and WorldClim environmental rasters. On four species sampled from eBird, a transfer-learned ResNet-34 reaches about 91% test accuracy and high AUC scores, while a random forest trained on latitude, longitude, elevation, precipitation, and temperature reaches roughly 80 to 88% test accuracy depending on species. The paper presents these results as evidence that the two approaches capture both macro-level climate factors and micro-level landscape features such as water bodies, forest edges, and urbanization, making the method suitable for forecasting habitat shifts.","pith_inferences":["The headline 85% is an average that blends the ResNet's roughly 91% accuracy with tabular accuracies near 80%; species-by-species reporting is more informative than the single average.","If eBird occurrence data are geographically biased toward accessible areas, the strong importance of longitude in tabular predictions may partly reflect sampling effort rather than biological migration behavior.","A temporal test that trains on earlier years and tests on later years would directly probe whether the models predict range shifts rather than only static habitat suitability."],"forward_implications":["With openly available satellite and climate data, presence maps for additional bird species could be generated quickly using the same two pipelines.","The ResNet result implies that pretrained image features transfer to satellite imagery even with a small ecological dataset, making transfer learning the practical route over training CNNs from scratch.","Feature importance shows longitude and proximity to coast dominate tabular predictions, so future migration models should weight geographic and climate drivers alongside visual landscape cues.","An integrated model that combines image and tabular inputs is the paper's explicitly proposed next step and would likely outperform either model alone.","The evaluation protocol of accuracy, AUC, confusion matrices, and ROC curves is portable to new species without requiring manual field observation."],"supporting_citations":[{"why":"Supplies the citizen-science bird occurrence data used for presence labels.","marker":"[17]"},{"why":"Supplies the climate and topography rasters used as tabular model features.","marker":"[18]"},{"why":"Supplies Sentinel-2 satellite imagery used as CNN input.","marker":"[19]"},{"why":"Provides the methodology for generating pseudo-absence points, on which the binary labels depend.","marker":"[20]"},{"why":"Provides the ResNet residual learning architecture used as the CNN backbone.","marker":"[14]"},{"why":"Frames the bird species distribution modeling task and serves as the comparative dataset baseline.","marker":"[38]"},{"why":"Provides the random forest classifier underlying the tabular approach.","marker":"[12]"}],"fun_headline_variants":["CNN plus tabular data maps bird habitats at 85% accuracy","Satellite imagery and climate data predict bird presence","Hybrid model forecasts bird migration via images and climate","Birds in focus: CNNs and tabular data hit 85% accuracy","Two-model system predicts bird distributions with 85% accuracy"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire accuracy and AUC story rests on treating locations more than 1.1 km from any eBird sighting as true bird absences; if birds can be present near those points or the citizen-science data is biased, the labels do not reflect reality.","fun_headline_variants_meta":{"raw":{"variants":["CNN plus tabular data maps bird habitats at 85% accuracy","Satellite imagery and climate data predict bird presence","Hybrid model forecasts bird migration via images and climate","Birds in focus: CNNs and tabular data hit 85% accuracy","Two-model system predicts bird distributions with 85% accuracy"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000154,"raw_usage":{"total_tokens":1144,"prompt_tokens":811,"completion_tokens":333,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":427,"completion_tokens_details":{"reasoning_tokens":248}},"tokens_in":427,"tokens_out":333,"duration_ms":4118,"temperature":1.0,"reasoning_tokens":248,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:20:04.717756+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Collect independent absence records for the same four species, for example repeated standardized point counts at locations the model labels absent, and compare the model's predictions against those real absences. If accuracy at locations the model calls absent falls far below the reported 80 to 90 percent, the pseudo-absence labels are not true absences and the reported performance is an artifact of label construction.","supporting_citations":[{"cited_title":"L., Wood, C","cited_arxiv_id":null,"evidence_quote":"Supplies the citizen-science bird occurrence data used for presence labels."},{"cited_title":"E., & Hijmans, R","cited_arxiv_id":null,"evidence_quote":"Supplies the climate and topography rasters used as tabular model features."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies Sentinel-2 satellite imagery used as CNN input."},{"cited_title":"H., and Thuiller, W","cited_arxiv_id":null,"evidence_quote":"Provides the methodology for generating pseudo-absence points, on which the binary labels depend."}],"review_version":1}