{"id":"90cf7685-5733-44ad-8cab-57942fc8ea8b","arxiv_id":"2412.12188","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"Using free satellite and survey data, machine learning predicts school internet connectivity in Botswana and Rwanda, with engineered features outperforming pre-trained location embeddings.","lead":"This study tests whether satellite and open geospatial data can predict whether a school has internet access, using pilots in Botswana and Rwanda. The authors find that combining engineered features with ground-based school data beats off-the-shelf location embeddings, reaching around 80% accuracy in Botswana and 73% in Rwanda.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Spatial leakage from overlapping 1,000m buffers likely inflates the reported 80%/0.81 and 73%/0.74 accuracies; the random school-level split cannot support the headline feasibility claim.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the evaluation relies on a random school-level split despite overlapping 1,000m satellite buffers, which couples training and test samples. The paper itself provides evidence of this overlap in the Kigali case study (Section 5) and Appendix C, making the concern concrete rather than hypothetical. I additionally flag that the buffer radius was selected using the same test set, compounding the optimism. Both issues directly affect the absolute accuracy and F1 values in the conclusion and the relative gain from auxiliary data in Table 5, so the central feasibility claim is not fully supported by the current evaluation. The paper has independent strengths: it releases a cleaned, labelled dataset and reproducible code, compares multiple location encoders fairly, and includes an honest limitations section. These strengths justify keeping the verdict as CONDITIONAL rather than moving to REJECT; the authors can address the concern with spatial cross-validation and a corrected buffer-selection protocol. The reader's verdict already requires such checks, so no change in verdict is needed.","tokens_in":1127,"tokens_out":740,"duration_ms":38560,"concrete_test":"Re-run the Table 5 experiments (e.g., GB, XGB, or SVM with auxiliary features for Botswana and Rwanda) using spatial cross-validation: cluster schools into spatially contiguous blocks (e.g., a 2km grid or DBSCAN with eps=2,000m), assign all schools in a block to the same fold, and report averaged test accuracy, F1, and false-positive rate. If the 80%/0.81 and 73%/0.74 results drop below the corresponding no-auxiliary baselines (68%/0.71 and 63%/0.72) or change the relative ranking of feature setups, the headline claim is an artifact of leakage. Additionally, repeat the buffer-size selection in Appendix C with a nested hold-out, so the 1,000m choice is evaluated on a held-out set rather than on the same test set used for final reporting.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.1 constructs engineered features from a 1,000m radius around each school. Section 5's Kigali case study and Appendix C acknowledge that nearby schools have overlapping buffers, with schools less than 400m apart having different connectivity labels. With a random 70/15/15 split (Section 4), a test school can have training schools within 2,000m, sharing almost identical MODIS, VIIRS, GHSL, population, and other features. The model can therefore memorize local feature patterns instead of learning a generalizable connectivity signal, inflating test accuracy, F1, and the apparent benefit of auxiliary data in Table 5. Additionally, Appendix C shows the 1,000m buffer was selected by comparing test-set performance across buffer sizes; because the same test set was used for this selection, even the choice of buffer radius is optimistically biased. The paper's central claim that ML with EO and ground-based auxiliary data is feasible for school connectivity screening rests on these absolute numbers, so this evaluation premise is load-bearing.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates whether machine learning with freely available Earth Observation (EO) data and ground-based auxiliary data can predict binary internet connectivity status of schools in Botswana and Rwanda. The authors construct engineered features from a 1,000m buffer around each school (MODIS, VIIRS nightlight, GHSL, population, distance to transmission lines, Ookla speedtest data), extract embeddings from four location-encoder models (SatCLIP, GeoCLIP, CSP, and their PhilEO VHR), and train six shallow classifiers. They report that engineered features outperform location-encoder embeddings, and that adding auxiliary school information improves performance, achieving 80% accuracy and 0.81 F1 in Botswana and 73% and 0.74 in Rwanda. They also present a Kigali case study and a feature importance analysis.","tokens_in":22054,"tokens_out":5637,"duration_ms":51710,"significance":"If the reported accuracies are reliable, the work would demonstrate a low-cost, open-data screening tool for school connectivity in low-resource settings, with potential operational value for Giga/Project Connect. The paper's strengths include the release of code and preprocessed data, the systematic comparison of four location encoders across six classifiers, reporting of variance over five runs for many experiments, and a candid discussion of limitations including label staleness and buffer overlap. However, the evaluation design currently precludes the absolute accuracy claims from being taken at face value, because the random school-level split does not separate spatially overlapping feature buffers and the 1,000m buffer radius was selected using the test set.","major_comments":[{"comment":"The evaluation premise is undermined by spatial leakage from overlapping 1,000m buffers. The paper's own Kigali case study (Section 5) notes schools less than 400m apart with different connectivity labels, and Appendix C shows overlapping buffers. With a random 70/15/15 split (Section 4), a test school can have training schools within 2,000m, so their engineered features (MODIS, VIIRS, GHSL, population, and distance-to-grid variables) are nearly identical. The model can therefore memorize local feature patterns rather than learn a generalizable connectivity signal, inflating the test accuracy, F1, and the apparent benefit of auxiliary data in Table 5. This is a load-bearing issue for the central feasibility claim, and should be addressed by spatial cross-validation (e.g., grouping schools into spatially disjoint clusters or excluding training samples within the buffer radius) and by reporting the degree of feature overlap between train and test sets.","section":"Section 3.1, Section 4, Section 5, Appendix C"},{"comment":"The 1,000m buffer radius was selected based on test-set performance. Section 3.1 states that the buffer extent was chosen because '1,000m yielded the best results for test set F1 score and accuracy,' and Appendix C confirms that the comparative analysis used the test set to select this hyperparameter. Using the same test set for both model selection and final evaluation makes the reported accuracies optimistically biased. The buffer radius should be selected on a validation set or via nested cross-validation, and the final test numbers should be reported only after that selection.","section":"Section 3.1, Appendix C, Table 7"},{"comment":"Label staleness is acknowledged but not quantified. The paper states that 'there is an undefined latency between connection and updated record,' meaning ground-truth labels may not reflect true current connectivity. Because the headline claims are absolute accuracy and F1 values (80%/0.81 in Botswana and 73%/0.74 in Rwanda, Table 5), this labeling noise—especially if it is spatially correlated—directly weakens the conclusion that the model predicts true connectivity. The paper should either quantify the expected label noise (e.g., by comparing against a sample of ground-validated schools) or temper the claims to 'predicting the current Project Connect database labels' rather than actual school connectivity.","section":"Section 6, Table 5"}],"minor_comments":[{"comment":"There are several typographical errors: 'digital literary' should be 'digital literacy' in the abstract, 'feasability' should be 'feasibility' in Section 2, 'datatset' should be 'dataset' in Section 3.2, and 'construing' in Section 5 appears to be a typo for 'preventing' or 'confusing.'","section":"Abstract, Section 2, Section 3.2, Section 5"},{"comment":"The reference 'N. atledge' should be 'N. Ratledge' (Nature paper 'Using machine learning to assess the livelihood impact of electricity access').","section":"References"},{"comment":"Table 5 reports no variance or confidence intervals for the auxiliary-data results, whereas the appendix (Tables 13-24) provides variance over five runs for the other feature spaces; the same variance reporting should be added for Table 5 to support the reliability of the headline numbers.","section":"Table 5"},{"comment":"In Table 3 (BW A), the SVM with SatCLIP-ViT16-L40 achieves accuracy 0.19, far below the majority-class baseline; this anomaly is not discussed and suggests a degenerate hyperparameter configuration, and it should be either explained or excluded from the comparison.","section":"Table 3"},{"comment":"The false-positive rate (FP) in Table 5 is not explicitly defined; the paper should state its denominator (e.g., FP/(FP+TN)) and how it is averaged across folds or runs.","section":"Table 5"},{"comment":"The feature importance analysis is based on a single model (Random Forest); the paper should note that these importances may not be stable across classifiers and should avoid over-generalizing from one model's ranking.","section":"Section 5, Figure 4"}],"recommendation":"major_revision","confidential_remarks":"The manuscript addresses an important application and the dataset release is valuable. The core issue is evaluation methodology rather than the prediction idea itself: a spatial cross-validation or buffer-disjoint split, together with an honest buffer-radius selection procedure, would substantially strengthen the claims. The limitations acknowledged in the paper (label staleness, buffer overlap) are pointed in the right direction but are not enough to mitigate the load-bearing evaluation problems. I would not recommend rejection, as these issues are fixable within the scope of a revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know. First, this is a useful feasibility study with a genuinely new open dataset for school connectivity in Botswana and Rwanda, and an honest limitations section. Second, the headline numbers (80%/0.81 and 73%/0.74) are likely inflated by spatial leakage: the 1,000m buffers around schools overlap, the random 70/15/15 split does not separate spatially, and the buffer radius itself was chosen on the test set. The central feasibility claim rests on those numbers, so the evaluation premise is shaky.\n\nWhat's new: no prior study has applied location encoders (SatCLIP, GeoCLIP, CSP, and the ESA PhilEO VHR precursor) to school internet connectivity, and the comparison of engineered EO features versus embeddings is a reasonable contribution. The dataset and code are released, which is real value. The finding that engineered features beat location embeddings across architectures is consistent, and the Kigali case study honestly shows where the method fails.\n\nSoft spots, in proportion. The stress-test note is correct: with schools less than 400m apart having different labels and buffers of 1,000m radius, training and test schools share nearly identical satellite-derived input features. That inflates accuracy and F1, and may exaggerate the apparent benefit of auxiliary school data, since some of those features (distance to LTE/UMTS/GSM stations) are almost direct signals. The buffer selection on test set is a separate, smaller but real problem. The authors acknowledge stale ground-truth labels themselves. None of this sinks the paper's qualitative message, but the absolute numbers should not be taken at face value for policy.\n\nWho it's for: anyone working on EO-based social good, especially in low-resource settings, and the Giga/UNICEF community. It doesn't reshape a field, but it provides a benchmark and a dataset.\n\nRecommendation: send to peer review. The evaluation needs spatial cross-validation (e.g., buffer-exclusion or cluster-based splits), and the abstract should soften the absolute claims. With those changes, it's a solid contribution.","headline":"Useful open dataset and honest feasibility study, but spatial leakage from overlapping buffers and test-set buffer selection inflate the headline accuracies.","tokens_in":22540,"tokens_out":1630,"would_cite":true,"duration_ms":15341,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"School internet access can be predicted from free satellite and survey data with 80% accuracy in Botswana and 73% in Rwanda, the paper claims.","keywords":["school internet connectivity","Earth Observation","machine learning","location encoders","satellite imagery","digital divide","Botswana","Rwanda"],"falsifier":"Re-run the same experiments with a spatial split that keeps all schools within 1,000 metres of each other in the same fold; if accuracy and F1 in Botswana and Rwanda drop substantially, or false-positive rates rise, relative to the reported 80%/0.81 and 73%/0.74, then the random split is the reason. A complementary check would be to field-validate predictions in a fresh district that contributed no schools to training.","tokens_in":1907,"feed_emoji":"🛰️","tokens_out":2310,"duration_ms":77041,"temperature":0.7,"pith_summary":"This feasibility study tries to establish that machine learning can predict whether a school has internet access using only free, openly available satellite imagery and survey data, at accuracies that are useful to governments planning digital infrastructure. Its best models combine satellite-derived features (land cover, night-time lights, settlement type, population, human modification, distance to power lines, and speed-test data) with auxiliary school records, reaching 80% accuracy and 0.81 F1 in Botswana and 73% accuracy and 0.74 F1 in Rwanda. The paper also reports that hand-crafted satellite features outperform all four tested location-encoder embeddings on this task, although adding embeddings helped slightly in Botswana. The authors present this as a practical, low-cost screening approach for low-resource settings and release cleaned, labelled datasets so other countries can replicate the pipeline.","feed_headline":"Satellite data spots offline schools with up to 80% accuracy","feed_subtitle":"A feasibility study in Botswana and Rwanda shows free Earth observation data can help governments target digital infrastructure aid.","key_machinery":"The load-bearing object is the engineered feature vector: for each school, summary statistics such as mean, variance, mode, max, min, and class percentages are computed inside a 1,000-metre satellite-imagery buffer around the school's coordinates, drawn from MODIS land cover, VIIRS night-time lights, the Global Human Settlement Layer, Gridded Population of the World, and Global Human Modification, together with distance to the nearest power-grid transmission line and Ookla mobile and fixed speed-test statistics. A location encoder is a neural network that turns a latitude/longitude coordinate into a vector embedding summarising the surrounding area from satellite imagery; the paper compares SatCLIP, GeoCLIP, CSP, and the new ESA PhilEO VHR Pre-cursor model as alternative or additive feature sources. Shallow classifiers (random forest, gradient boosting, SVM, logistic regression, XGBoost, and MLP) map these features to a binary connected/not-connected label. The decisive extension is auxiliary school data, which is what pushes accuracy up and false positives down.","core_discovery":"The central claim is that school internet connectivity can be predicted from multi-modal open geospatial data with practically useful accuracy in low-resource countries. Using a 1,000-metre satellite buffer around each school to compute features from MODIS land cover, VIIRS night-time lights, the Global Human Settlement Layer, gridded population, and global human modification, plus distance to power-grid lines and Ookla speed-test statistics, the best shallow classifiers outperform all four location-encoder embedding spaces on their own. Adding auxiliary school-level information (education level, distance to mobile network towers, one-hot administrative region, and school-age population counts) improves results further: accuracy and F1 reach 80% and 0.81 in Botswana and 73% and 0.74 in Rwanda, while false-positive rates drop from 48% to 19% in Botswana and 64% to 26% in Rwanda. The paper claims this demonstrates a viable, low-cost alternative to exhaustive school surveys, while its Kigali case study shows that the model struggles in dense urban areas where connected and unconnected schools sit within a few hundred metres of each other.","pith_inferences":["Editorial extension: The two-country result does not establish global generalization; the same pipeline should be tested in a geographically diverse set of low-resource countries, and the released dataset makes such a test straightforward.","Editorial extension: Because the 1,000-metre satellite windows around nearby schools overlap, the random 70/15/15 split may leak spatial context into the test set; a cluster-based or buffer-excluded split would be a sharper test of the reported 80% and 73% accuracy figures.","Editorial extension: The false-positive reduction from auxiliary data points toward a practical product: a prioritisation score combining satellite context with school records, which governments could update as new schools are connected.","Editorial extension: The underperformance of location encoders here suggests that for infrastructure tasks, frozen generic embeddings should be evaluated against simple engineered features before being adopted, rather than assumed to carry the needed socio-economic signal."],"forward_implications":["Governments and development agencies can produce national school-connectivity maps from free satellite and survey data in countries that cannot afford exhaustive ground surveys, and use those maps to prioritise infrastructure investment.","Adding basic administrative and school-record information to satellite features is the main lever for reducing false positives, the error that would waste resources by treating unconnected schools as connected.","The public release of cleaned school-connectivity labels and the feature-generation pipeline enables replication of the study in other low-resource countries without proprietary data.","Generic location-encoder embeddings alone are not a sufficient substitute for domain-engineered features in this task, suggesting that pretrained geographic foundation models need fine-tuning or higher-resolution inputs for connectivity prediction.","In dense urban settings, predictions for schools less than a few hundred metres apart are unreliable, so operational deployment would need to combine model outputs with local verification."],"supporting_citations":[{"why":"Supplies the school coordinates and binary connected/not-connected labels from Project Connect that every model is trained and evaluated on.","marker":"[UNICEF, 2023b]"},{"why":"Provides the global power-grid transmission-line map used to compute each school's distance to the nearest line, a top-ranking engineered feature.","marker":"[Arderne et al., 2020]"},{"why":"Provides the mobile and fixed broadband speed-test tiles used to build download, upload, latency, device-count, and distance features.","marker":"[Ookla LLC., 2023]"},{"why":"Supplies gridded sex-disaggregated school-age population counts used as auxiliary features to improve accuracy and lower false positives.","marker":"[Bondarenko et al., 2022]"},{"why":"Defines SatCLIP, the location-encoder model whose embeddings the paper evaluates and compares against engineered features.","marker":"[Klemmer et al., 2023]"},{"why":"Defines CSP, the contrastive spatial pre-training model trained on Functional Map of the World data used as a second location encoder.","marker":"[Mai et al., 2023]"},{"why":"Defines GeoCLIP, the location encoder chosen for its limited-data performance and compared in the study.","marker":"[Vivanco Cepeda et al., 2023]"},{"why":"Introduces the PhilEO VHR Pre-cursor model, the new geographically-aware foundation model whose embeddings are tested and combined with engineered features.","marker":"[Fibaek et al., 2024]"}],"fun_headline_variants":["Satellite data predicts school internet access with 80% accuracy","Space-based AI maps offline schools in Botswana and Rwanda","Free Earth data helps target school internet gaps","ML on satellite images finds schools without internet"],"cache_read_input_tokens":24832,"weakest_assumption_plain":"The paper's results assume that a random 70/15/15 split of schools into training, validation, and test sets is a fair test, even though the 1,000-metre satellite windows around nearby schools overlap, so the model may have effectively seen parts of a test school's surroundings during training.","fun_headline_variants_meta":{"raw":{"variants":["Satellite data predicts school internet access with 80% accuracy","Space-based AI maps offline schools in Botswana and Rwanda","Free Earth data helps target school internet gaps","ML on satellite images finds schools without internet"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000204,"raw_usage":{"total_tokens":1433,"prompt_tokens":1036,"completion_tokens":397,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":652,"completion_tokens_details":{"reasoning_tokens":336}},"tokens_in":652,"tokens_out":397,"duration_ms":4459,"temperature":1.0,"reasoning_tokens":336,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T15:51:38.102196+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same experiments with a spatial split that keeps all schools within 1,000 metres of each other in the same fold; if accuracy and F1 in Botswana and Rwanda drop substantially, or false-positive rates rise, relative to the reported 80%/0.81 and 73%/0.74, then the random split is the reason. A complementary check would be to field-validate predictions in a fresh district that contributed no schools to training.","supporting_citations":[{"cited_title":"Global fixed broadband and mobile network maps","cited_arxiv_id":null,"evidence_quote":"Provides the mobile and fixed broadband speed-test tiles used to build download, upload, latency, device-count, and distance features."},{"cited_title":"CSP: Self-Supervised Contrastive Spatial Pre-Training for Geospatial-Visual Representations","cited_arxiv_id":"2305.01118","evidence_quote":"Defines CSP, the contrastive spatial pre-training model trained on Functional Map of the World data used as a second location encoder."},{"cited_title":"Phileo bench: Evaluating geo-spatial foundation models, 2024","cited_arxiv_id":null,"evidence_quote":"Introduces the PhilEO VHR Pre-cursor model, the new geographically-aware foundation model whose embeddings are tested and combined with engineered features."}],"review_version":1}