{"id":"d062225f-420e-463c-88a8-bfc7be9b693e","arxiv_id":"2412.14870","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A weakly supervised satellite-imagery pipeline locates schools in ten African countries with AUPRC above 0.96 and generates nationwide candidate school maps.","lead":"This paper trains deep learning models to spot schools in satellite images and maps their likely locations across ten African countries using only image-level labels. It reports classification accuracy above 0.96 and a web tool for government validation of the predicted school points.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 'precise GPS coordinates' claim is unvalidated: CAM peak location is never compared with true school coordinates, and ROAD only measures classifier faithfulness.","rationale":"The reader's weakest_assumption exactly identifies the missing validation of CAM-derived coordinates. This is the most load-bearing concern because it targets the paper's central claim: that classification-level annotations suffice for GPS-level school localization. The classification results (AUPRC >0.96) and the deployment tool are useful, but they do not establish the coordinate claim. I find no internal evidence that would resolve the concern: the ROAD metric is a faithfulness test, and the Senegal comparison uses a 250 m tolerance, which is not 'GPS coordinate' precision. The proposed test using held-out school coordinates would directly settle the issue. I agree with the reader's conditional verdict and its required revisions; no verdict change is needed. I credit the paper for transparently discussing label noise, releasing code, and focusing negative sampling on built-up areas, but these do not substitute for coordinate-level evaluation.","tokens_in":12896,"tokens_out":3487,"duration_ms":29700,"concrete_test":"For a held-out test set in one pilot country (e.g., Senegal) where true school coordinates are known, run the full localization pipeline: compute the chosen CAM for each test image that contains a school, take the argmax pixel, convert to lat-lon, and compute great-circle distance to the known school coordinate. Report median, mean, and 90th percentile distance error, and compare against a simple baseline such as the image center or the centroid of the top-10% CAM region. If the median error exceeds roughly 40 m (the scale of a school compound) or the 90th percentile approaches the 250 m threshold used in the Senegal case study, the 'GPS coordinate' claim fails.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central novelty is weakly supervised coordinate-level localization. In 'School localization', the authors 'convert the xy coordinates of the most important pixel to a lat-lon coordinate to approximate the geographic coordinates of the school.' Yet no experiment measures the distance between these predicted coordinates and known school coordinates. Table 3 reports ROAD confidence drop, a faithfulness metric: it shows that removing high-attention pixels lowers classifier confidence, not that those pixels are geographically at the school. The Senegal comparison (Figure 5) matches model predictions to government points within 250 m, a threshold far coarser than 'the level of GPS coordinates'; it also compares counts, not per-prediction distance errors. Given the paper itself notes that schools are identified by 'playgrounds, track and field ovals, basketball courts, open fields, and grouped building structures', the argmax CAM pixel may fall on a court or field tens of meters from the school building. Without a direct spatial error measurement, the coordinate-level claim is unsupported and the downstream connectivity-cost estimates built on it inherit this uncertainty. This is the load-bearing assumption: if CAM peaks are systematically offset, the headline contribution collapses even though image-level AUPRC remains above 0.96.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript proposes a weakly supervised pipeline for mapping schools from high-resolution Maxar satellite imagery. It builds country-level training sets from government records, OpenStreetMap, Overture Maps, and random non-school sampling; fine-tunes ViT, SwinV2, and ConvNext classifiers and their ensemble; reports AUPRC values above 0.96 on test sets for ten African countries; uses class activation maps to convert the most important pixel into a predicted lat-lon coordinate; generates nationwide school predictions with an overlapping sliding window and a duplicate-aggregation step; evaluates attribution methods with the ROAD confidence-drop measure; and presents a Senegal case study comparing model predictions with government school data at a 250 m matching threshold. The paper also introduces an interactive web-based validation tool and makes code available on GitHub.","tokens_in":13141,"tokens_out":5995,"duration_ms":47514,"significance":"If the classification and localization claims hold, the pipeline would be practically valuable for school connectivity planning and would be a useful example of low-cost, weakly supervised geospatial mapping. The paper has clear strengths: it combines multiple public data sources, explicitly analyzes urban/rural performance differences, examines cross-country generalization, compares regional and local models, and releases code. However, the two headline claims are not established by the evidence as presented. The reported AUPRC numbers are affected by test-set-based model selection, and the claimed GPS-coordinate-level localization accuracy is never directly validated. The ROAD evaluation measures attribution faithfulness rather than geographic accuracy, and the Senegal comparison uses a coarse 250 m matching threshold, so it does not support the coordinate-level precision claim. These issues are central to the paper's contribution and require substantial revision.","major_comments":[{"comment":"The reported AUPRC values are not unbiased estimates because the same test set is used for model selection and final performance reporting. The text states that the authors 'select the model that achieves the highest AUPRC on the test set for each country,' and Table 2 then reports the performance of the selected models. This procedure makes the reported numbers optimistically biased because the test set has influenced the choice among architectures and ensembles. Please select models using the validation set (or nested cross-validation) and report test performance for the chosen model, or explicitly quantify the selection bias. This is load-bearing because the abstract's AUPRC-above-0.96 claim rests on these numbers.","section":"Results and Discussion, 'Image-level school classification'; Table 2"},{"comment":"The paper's central claim that the method 'can approximate the precise locations of schools at the level of GPS coordinates' is not supported by any direct spatial validation. The localization procedure converts the xy coordinate of the most important CAM pixel to a lat-lon coordinate, but no experiment compares predicted coordinates to known school coordinates. Table 3 reports ROAD confidence drop, which measures whether removing high-attention pixels lowers classifier confidence; it does not measure whether the highlighted pixel is geographically located at the school. The Senegal comparison in Figure 5 uses a 250 m matching distance and compares counts rather than per-prediction distance errors, so it does not establish coordinate-level accuracy. Because the paper itself notes that schools are identified by playgrounds, fields, and grouped building structures, the argmax CAM pixel could plausibly fall tens of meters from the actual school building. Please add a direct spatial evaluation, such as distance-to-nearest-known-school distributions at several thresholds and precision/recall as a function of distance, and report how the CAM peak relates to the school building.","section":"Methods, 'School localization'; Results and Discussion, 'Evaluating CAM methods'; Figure 5"}],"minor_comments":[{"comment":"The regional-model evaluation is inconsistent as written: the text says the regional model is tested on the individual test sets of each country, while the Table 2 caption says it is tested on the combined test sets, and Figure 3 compares regional and local AUPRC per country. Please clarify which evaluation was used, since this affects the regional-vs-local conclusion and the reproducibility of Figure 3.","section":"Generalizability experiments; Table 2 vs. Figure 3"},{"comment":"The country codes 'BW A' and 'RW A' contain stray spaces; please standardize them to 'BWA' and 'RWA'.","section":"Table 2"},{"comment":"The caption includes 'ROAD: -2.388' without explanation; please define this value, clarify why it is negative, or remove it.","section":"Figure 1"},{"comment":"The criterion for choosing the 'best-performing model' used to compute ROAD scores, while excluding the ensemble model, is not stated; please specify whether selection is by validation AUPRC, test AUPRC, or another metric, and use validation-based selection for consistency.","section":"Evaluating CAM methods"},{"comment":"The manual repositioning of school GPS coordinates based on Google Satellite Hybrid imagery should be documented quantitatively, including how many points were moved and by what distance, because this affects the label accuracy used for both training and evaluation.","section":"Data preprocessing"},{"comment":"The aggregation buffer radius (50 m, or 150 m for Botswana) and the F2-optimal threshold are free parameters that materially affect the nationwide counts; a sensitivity analysis for the Senegal case would help interpret the reported 6,983 matched, 2,050 unmatched government, and 5,369 unmatched prediction figures.","section":"Country-wide school predictions"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a reasonable applied contribution, but the two headline claims need substantial rework before publication. The AUPRC-above-0.96 claim is likely inflated by test-set-based model selection, and the coordinate-level localization claim lacks any direct spatial accuracy validation. If the authors cannot provide per-prediction distance errors, the paper should be reframed as image-level school detection with candidate generation rather than coordinate-level mapping. The GitHub code release is a positive factor, and the test-set selection issue is fixable by re-running evaluations with a proper validation/test split."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThis paper is a solid applied contribution to school mapping, but the headline coordinate-level claim doesn't hold up to scrutiny. The authors train image classifiers on 300m satellite tiles across 10 African countries, get AUPRC >0.96, then use CAMs to pick the most important pixel and convert that to a lat-lon. The classification pipeline is real: they use government data plus OSM/Overture, clean it, sample negatives from built-up areas (a good idea), and compare ViT/Swin/ConvNext plus an ensemble. The ensemble consistently beats individual models. The web tool for human-in-the-loop validation is genuinely useful for the Giga use case.\n\nWhat's new here is the scale and the combination: transformer ensembles plus built-up-area negative sampling applied to school mapping, with a ten-country evaluation. The cross-country generalization heatmap is also informative.\n\nThe soft spots are real and not minor. First, model selection is done on the test set: \"select the model that achieves the highest AUPRC on the test set for each country.\" That directly inflates the reported numbers. With no error bars and a single training run per model, the AUPRC differences between variants are within the noise. They should have selected on validation.\n\nSecond, and more important, the localization claim is unvalidated. The paper says they approximate \"the precise geographical coordinates\" of schools from CAMs, but never once compares a predicted coordinate to a known school coordinate. The ROAD metric measures whether removing high-attention pixels drops classifier confidence; it says nothing about whether that pixel is actually the school. The Senegal comparison uses a 250m matching threshold and compares counts, not per-prediction distance error. Since the paper itself notes that schools are often identified by playgrounds, courts, and open fields, the argmax CAM pixel can easily sit tens of meters off the building. For a connectivity program, that might be acceptable for field validation, but it's not \"GPS coordinate\" accuracy as claimed.\n\nThere's also no spatial cross-validation, though the authors cite Wadoux et al. to justify the random split, and the 300m spacing between points mitigates leakage somewhat. That is a minor concern compared to the two above.\n\nThis is worth serious peer review, not a desk reject. The problem is important, the pipeline is plausible, and the flaws are fixable. A referee should ask for (a) localization error measured against held-out true coordinates, (b) model selection on validation only, and (c) confidence intervals or multiple seeds. If those are addressed, this could become a solid reference for operational school mapping.\n\nI'd bring it to a reading group as a case study on evaluation pitfalls.","headline":"Useful applied school-mapping pipeline, but the GPS-level localization claim is unvalidated and the AUPRC numbers are inflated by test-set model selection.","tokens_in":13661,"tokens_out":3203,"would_cite":true,"duration_ms":26161,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Classification-level labels alone can locate schools at GPS coordinate precision, with ensembled ViT/CNN models scoring AUPRC above 0.96 across ten African countries.","keywords":["school mapping","satellite imagery","weakly supervised learning","class activation maps","vision transformers","explainable AI","universal school connectivity","Senegal"],"falsifier":"Hold out a set of tiles whose true school-building coordinates are known, compute the CAM argmax for each, convert it to lat-lon, and measure the geodesic distance to the true coordinate; if most predictions miss by more than roughly 100 meters, the GPS-level localization claim fails even if tile-level AUPRC stays above 0.96.","tokens_in":12692,"feed_emoji":"🛰️","tokens_out":8688,"duration_ms":66421,"temperature":0.7,"pith_summary":"This paper claims that nationwide school maps can be produced from satellite imagery using only low-cost, classification-level annotations, with no bounding boxes or pixel masks required. The authors train ensembles of vision transformers and convolutional networks to decide whether a 300-by-300-meter tile contains a school, reaching AUPRC values above 0.96 in ten African countries. They then read the coordinate of the most important pixel in a class activation map and convert it to lat-lon, approximating GPS-level school locations from image-level labels alone. If correct, this makes complete and current school location data affordable for low- and middle-income countries, directly supporting connectivity cost estimation and infrastructure planning. The paper closes with nationwide prediction maps, a Senegal case study, and a web tool for government-led validation.","feed_headline":"Satellite AI maps schools using only yes/no labels","feed_subtitle":"AUPRC above 0.96 across 10 African countries; Senegal case study finds thousands of unmapped schools for connectivity planning.","key_machinery":"The load-bearing mechanism is two-stage weak supervision. First, an image classifier ensemble, VSC-Ensemble, averages the softmax outputs of the best vision transformer, Swin transformer, and ConvNeXt models trained on 300-by-300-meter tiles with a fixed 1:2 positive-to-negative ratio and hard 300-meter sample spacing to prevent leakage. Second, weakly supervised localization uses class activation maps: the xy-coordinate of the highest-attention pixel is converted into a lat-lon coordinate, requiring only the classification label as supervision. Attribution quality is scored with the ROAD framework, which measures average confidence drop after perturbing the top 10% of pixels in most-relevant-first order, letting the authors pick the best CAM variant per country.","core_discovery":"The central claim is that image-level classification labels are sufficient to approximate the precise geographical coordinates of schools: a classifier that detects whether a satellite tile contains a school, combined with a class activation map, yields a point estimate of the school's location within the tile. Across ten pilot countries, the best ensemble of ViT, SwinV2, and ConvNext models (VSC-Ensemble) achieves AUPRC above 0.96, and GradCAM-family attribution methods give the most faithful importance maps by the ROAD confidence-drop measure. Comparing model predictions with official Senegalese records at a probability threshold of 0.5 and a matching distance of 250 meters, the pipeline matched 6,983 of 9,033 government-registered schools and surfaced 5,369 unmatched predictions for field validation. The authors conclude that governments can use these validation-ready predictions to correct incomplete registries and estimate the cost of connecting every school.","pith_inferences":["A natural extension the paper does not run is to validate CAM-derived coordinates against a held-out set of true school-building points; the Senegal match statistics only measure agreement with registry points, not geographic accuracy.","Because the noise analysis shows many government coordinates sit hundreds of meters off the actual building, a practical follow-up would use model predictions to correct existing registry coordinates, not only to discover unmapped schools.","The same classification-plus-CAM recipe could transfer to other dispersed infrastructure with distinctive overhead signatures, such as health clinics, water points, or markets, where connectivity planning faces the same data gaps.","The cross-country generalization pattern suggests a geography-aware training schedule could outperform pooling all countries equally, since neighboring countries generalize well to each other while small-sample countries do not."],"forward_implications":["Governments can generate or update nationwide school maps from cheap image-level labels instead of expensive bounding-box or pixel-level annotation campaigns.","The Senegal comparison suggests the pipeline can surface thousands of unmapped candidate schools, giving field teams a shortlist ordered by confidence.","Regional models trained on multiple countries perform about as well as country-specific models, so one model can serve countries with scarce local training data.","The threshold-based web tool lets stakeholders trade precision against recall, adapting the map to how much field validation effort they can afford.","Connectivity cost estimates can be built directly from the predicted school coordinates rather than from incomplete national registries."],"supporting_citations":[{"why":"Establishes the weakly supervised XAI localization approach this paper adapts, turning image-level labels into object coordinates for brick kilns.","marker":"Lee et al. 2021"},{"why":"Prior demonstration that deep learning on aerial imagery can approximate lat-lon coordinates of dispersed objects, cited as the localization precedent.","marker":"Handan-Nader and Ho 2019"},{"why":"Provides the F2-threshold mapping strategy and the large-scale prediction-plus-validation workflow this paper follows for nationwide maps.","marker":"Robinson et al. 2022"},{"why":"Main prior CNN-based school mapping baseline; the paper builds on it while changing non-school sampling to built-up areas.","marker":"Maduako et al. 2022"},{"why":"Supplies the ROAD evaluation framework used to score CAM attribution methods by average confidence drop.","marker":"Rong et al. 2022"},{"why":"Introduces the vision transformer architecture used as one of the three model families in the VSC-Ensemble.","marker":"Kolesnikov et al. 2021"},{"why":"Introduces Swin Transformer V2, the second architecture family in the ensemble.","marker":"Liu et al. 2022a"},{"why":"Introduces ConvNeXt, the convolutional family that completes the VSC-Ensemble.","marker":"Liu et al. 2022b"}],"fun_headline_variants":["Yes/no labels enough to map African schools from space","Satellite AI finds thousands of unmapped schools in Senegal","Weak supervision maps schools across 10 African countries","From yes/no to coordinates: AI school mapping scaled"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The coordinate-level claim rests on the unvalidated assumption that the single most important pixel in a class activation map points at the actual school building rather than at a surrounding field, court, or nearby structure.","fun_headline_variants_meta":{"raw":{"variants":["Yes/no labels enough to map African schools from space","Satellite AI finds thousands of unmapped schools in Senegal","Weak supervision maps schools across 10 African countries","From yes/no to coordinates: AI school mapping scaled"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000447,"raw_usage":{"total_tokens":2252,"prompt_tokens":932,"completion_tokens":1320,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":548,"completion_tokens_details":{"reasoning_tokens":1257}},"tokens_in":548,"tokens_out":1320,"duration_ms":9024,"temperature":1.0,"reasoning_tokens":1257,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T11:50:12.394471+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Hold out a set of tiles whose true school-building coordinates are known, compute the CAM argmax for each, convert it to lat-lon, and measure the geodesic distance to the true coordinate; if most predictions miss by more than roughly 100 meters, the GPS-level localization claim fails even if tile-level AUPRC stays above 0.96.","supporting_citations":[{"cited_title":"R.; Tajwar, F.; Burke, M.; Ermon, S.; Lobell, D","cited_arxiv_id":null,"evidence_quote":"Establishes the weakly supervised XAI localization approach this paper adapts, turning image-level labels into object coordinates for brick kilns."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Prior demonstration that deep learning on aerial imagery can approximate lat-lon coordinates of dispersed objects, cited as the localization precedent."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Provides the F2-threshold mapping strategy and the large-scale prediction-plus-validation workflow this paper follows for nationwide maps."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Main prior CNN-based school mapping baseline; the paper builds on it while changing non-school sampling to built-up areas."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ROAD evaluation framework used to score CAM attribution methods by average confidence drop."},{"cited_title":"International Conference on Learning Representations","cited_arxiv_id":null,"evidence_quote":"Introduces the vision transformer architecture used as one of the three model families in the VSC-Ensemble."}],"review_version":1}