{"id":"5ec668fd-e758-40d6-a6e0-dff6c9456ca2","arxiv_id":"2505.01242","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A dataset comprising 348 pre and 348 post-disaster satellite images of Blantyre with manually labeled building damage levels is released on Zenodo for machine learning research.","lead":"This paper introduces a public dataset of satellite images and building damage labels from Cyclone Freddy's 2023 floods in Blantyre, Malawi. It aims to help train machine learning models for flood damage detection in African cities.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The dataset's utility rests on unvalidated, manually assigned damage labels, and the authors themselves concede that the pre/post imagery may not faithfully reflect the flood damage.","rationale":"The reader's weakest assumption correctly identifies label accuracy and reproducibility as the load-bearing premise. My stress-test confirms this and finds additional support in the manuscript itself: the authors explicitly state that the pre/post images may not faithfully reflect the damage, which is an in-text admission that the label target is uncertain. The seven-month pre-event to two-month post-event interval makes non-flood changes plausible, and no validation, inter-annotator agreement, or baseline experiment is provided. This is not a disagreement with consensus; it is a correctness risk in the dataset's core annotation. The recommended CONDITIONAL verdict remains appropriate: the dataset may still be useful, but acceptance should require independent label validation and a basic ML benchmark. I do not see a reason to move the verdict to REJECT or UNVERDICTED, because the resource itself is real, publicly hosted, and potentially valuable; the concern is about the strength of the evidence for its central claim, not about the existence of the dataset.","tokens_in":8123,"tokens_out":3441,"duration_ms":40075,"concrete_test":"Select a random stratified sample of 100 paired pre/post tiles. Have two external annotators, blind to the authors' labels, independently label every building using the same xBD rubric; compute Fleiss' kappa between the authors and the external annotators and Cohen's kappa between the two external annotators. Additionally, for 20 of those tiles, compare the author labels against very-high-resolution post-cyclone imagery or field/ground-survey records, treating disagreements as label errors. If Fleiss' kappa is below 0.6 or more than 20% of building-level labels disagree with the reference, the damage labels are not reliable enough to support the dataset's central use case.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that mwBTFreddy is a useful localized dataset for building-damage classification. That usefulness is carried entirely by the building-level damage labels. The annotation process (Collection Process section) is described as visual assessment by the dataset creators following the xBD rubric, but no inter-annotator agreement, ground-truth comparison, or error analysis is reported. More importantly, the datasheet itself contains an explicit limitation under 'Is any information missing from individual instances?': 'We relied on the availability of the data around Cyclone Freddy from Google Earth Pro, hence the post and pre images may not be a faithful reflection of the damage.' Pre-disaster images are from August 2022 and post-disaster images from May 2023, while Cyclone Freddy struck in March 2023. This seven-month pre-event window and two-month post-event window mean that changes labeled as flood damage could instead reflect construction, demolition, seasonal vegetation, or post-cyclone cleanup. The labels therefore conflate the flood signal with unrelated temporal change, and the self-admitted 'not a faithful reflection' admission directly undermines the target variable of the dataset. Without independent validation of the labels, neither training nor evaluation on this dataset can be trusted to measure flood damage. The lack of recommended splits and the absence of any baseline experiment further weaken the claim of utility for machine learning.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces mwBTFreddy, a dataset of paired pre- and post-disaster satellite images of urban areas in Blantyre, Malawi, affected by Cyclone Freddy in March 2023. The dataset contains 696 annotated images (348 pre-disaster and 348 post-disaster) with JSON annotation files providing building polygons, geographic coordinates, and damage labels following the xBD four-class scale (no damage, minor, major, destroyed). The images were exported from Google Earth Pro, georeferenced in QGIS, and manually annotated by members of the Kuyesera AI Lab. The paper is structured as a datasheet following the standard template, with sections on motivation, composition, collection process, preprocessing, uses, distribution, and maintenance. The authors claim the dataset fills a gap in localized disaster-response resources for African urban contexts and is suitable for building detection and damage classification tasks. The dataset is publicly available on Zenodo and has been used in a Zindi competition.","tokens_in":8355,"tokens_out":4018,"duration_ms":42178,"significance":"If the dataset's labels are reliable, mwBTFreddy addresses a genuine gap: there are few publicly available, localized satellite datasets of flood damage in African informal urban settlements, and the paper openly documents many sources of noise and limitations. The detailed datasheet format, the explicit discussion of exclusion criteria, and the public Zenodo release are strengths, as is the fact that the dataset has already been exercised in a competitive machine-learning setting. However, the central claim that the dataset can support trustworthy damage classification rests on the accuracy and reproducibility of the manually assigned labels, and that premise is not demonstrated. The authors themselves note that the pre/post imagery may not faithfully reflect the damage, and no inter-annotator agreement, ground-truth comparison, or baseline experiment is provided. The significance of the contribution is therefore conditional on label validation, which is the main load-bearing issue in the manuscript.","major_comments":[{"comment":"The central claim that mwBTFreddy is suitable for training and evaluating machine-learning models for damage classification is not yet supported because the target labels are unvalidated. The Collection Process section describes manual visual assessment by the authors following the xBD scale, but no inter-annotator agreement, independent expert review, or ground-truth comparison is reported. This concern is amplified by the datasheet's own admission in Composition that 'the post and pre images may not be a faithfull reflection of the damage,' and by the temporal gaps: pre-disaster images are from August 2022, the flood event occurred in March 2023, and post-disaster images are from May 2023. Changes in that window could reflect construction, demolition, vegetation change, or post-cyclone cleanup rather than flood damage. Please add an inter-annotator agreement study or external validation (e.g., against official damage assessments or field reports), and quantify how the temporal window may affect label reliability.","section":"Composition, 'Is any information missing from individual instances?' and Collection Process"},{"comment":"The paper asserts that the dataset supports building detection and damage classification, but it provides no dataset statistics, no class distribution, no recommended train/validation/test splits, and no baseline experiments. The datasheet explicitly answers 'No' to the question of recommended splits, and no evaluation is reported anywhere in the manuscript. Without a baseline model run or at least a label-distribution analysis, a prospective user cannot assess whether the dataset is usable for the claimed tasks. Please report per-class building counts, geographic distribution of labels, and a simple baseline experiment (e.g., a standard object-detection or classification model) to demonstrate practical utility.","section":"Uses and Distribution; datasheet answer to 'Are there recommended data splits?'"},{"comment":"The image-count accounting is internally inconsistent and will confuse users. Table I reports 1,026 images for the three areas (Chilobwe 1,000, Ndirande 20, Chirimba 6), Table V reports 696 annotated images and 330 unannotated images, and Table III reports 348 pre-disaster images and 348 post-disaster images as the total annotated instances. If 696 is the total number of annotated images, then the 348/348 split means 348 paired locations, not 696 independent tiles; if the original 1,026 images include both pre and post tiles, the relationship between tiles, annotated images, and pre/post pairs should be stated explicitly. Please clarify the exact counting scheme and reconcile the tables.","section":"Table I, Table III, and Table V; Composition section"}],"minor_comments":[{"comment":"There are numerous typos and formatting errors, including 'anotated' for 'annotated', 'faithfull' for 'faithful', 'od March' for 'of March', and the consistently broken 'T able' table headers. A careful proofreading pass is needed.","section":"Throughout"},{"comment":"The answer 'The dataset contains all possible instances' is contradicted by the later statement that 330 unannotated images were retained in a separate folder and by the exclusion criteria for buildings. Please reword to distinguish the annotated core dataset from the full set of captured tiles.","section":"Composition, 'Does the dataset contain all possible instances?'"},{"comment":"The text says 'No pre-processing of images was done' but then states that JPEG images downloaded from Google Earth Pro were converted to TIFF after geo-tagging and annotation. Conversion, cropping, and resizing described in the Collection Process are preprocessing steps; please make the description internally consistent.","section":"Preprocessing/cleaning/labeling"},{"comment":"The license information is insufficient: 'Refer to Zenodo's terms of use' does not specify which license applies to the dataset. The authors should state an explicit license (e.g., CC-BY 4.0) and confirm that the Google Earth Pro imagery can be redistributed under that license.","section":"Distribution"},{"comment":"The annotation and preprocessing scripts are only 'available upon reasonable request,' which weakens reproducibility. Since the dataset is archived on Zenodo, the scripts and a description of the annotation environment should be included in the same archive.","section":"Collection Process / Maintenance"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is an honest and well-structured dataset description, but its central contribution—a usable, trustworthy damage-classification dataset—depends on label reliability that is currently unverified. The authors' own admission that pre/post images may not faithfully reflect the damage makes this a correctness-risk issue, not a minor gap. I would ask for inter-annotator agreement or external validation, explicit dataset statistics, and a baseline experiment. If validation is infeasible, the authors should substantially soften the claims and clearly frame the dataset as exploratory with unverified labels."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Start with the bottom line: the dataset is a real contribution to a real gap. mwBTFreddy gives the community 696 annotated pre/post satellite image pairs for three flood-affected areas in Blantyre, which no existing dataset covers. The datasheet is unusually candid, listing exclusion criteria, noise sources, and the fact that no funding was involved. Raw images are on Zenodo, and the dataset has already been used in the Zindi AI for Equity challenge, which suggests at least some practical take-up.\n\nThe main weakness is the thing the dataset is built for: the building-level damage labels. The authors describe visual annotation in QGIS following the xBD rubric, but report no inter-annotator agreement, no ground-truth comparison, no error analysis. More fundamental, the temporal window is off. Pre images are August 2022, post images are May 2023, and Cyclone Freddy hit in March 2023. That leaves seven months pre and two months post. Construction, demolition, seasonal vegetation, and clean-up all show up in that interval. The datasheet itself says the pre/post images 'may not be a faithful reflection of the damage.' That is a load-bearing admission because the labels are the target variable. Without independent validation, a model trained on these labels may be learning temporal change in general, not flood damage.\n\nAdditional soft spots are minor except in combination: no recommended train/test splits, no baseline experiments, and a licensing section that says only 'refer to Zenodo's terms of use.' For a dataset meant to be reused, that is thin.\n\nNone of this kills the dataset. It is fixable. The authors could add inter-annotator agreement on a sample, hand-verify a subset against any available high-resolution imagery or field reports, and be explicit about which instances should not be used for damage classification. A simple baseline—even a per-image classifier—would show what the labels are good for. A clear license is not hard.\n\nBottom line: this deserves peer review. I would accept with major revisions, not desk reject. The geographic gap is real, the data is released, and the authors have shown they can write honestly about limitations. With label validation and baselines, mwBTFreddy becomes a resource people will actually trust.","headline":"A genuinely useful new dataset for a data-scarce region, but the damage labels are unvalidated and the pre/post imagery dates the authors admit are not a faithful reflection of the flood event—worth a serious referee, not a desk reject.","tokens_in":8792,"tokens_out":2872,"would_cite":true,"duration_ms":28466,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The mwBTFreddy dataset pairs pre- and post-Cyclone Freddy satellite images of Blantyre with building-level damage labels to give machine learning a local benchmark.","keywords":["mwBTFreddy dataset","satellite imagery","flood damage assessment","building damage classification","Cyclone Freddy","Malawi","xBD damage scale","georeferenced annotations"],"falsifier":"Select a random sample of annotated buildings and have independent annotators re-label them from the same images, or survey the buildings on the ground; if agreement falls well short of the level needed for reliable training labels, or if field checks contradict the JSON damage levels, the dataset's central value is not established.","tokens_in":7968,"feed_emoji":"🛰️","tokens_out":6419,"duration_ms":58163,"temperature":0.7,"pith_summary":"The paper introduces mwBTFreddy, a dataset of 696 georeferenced satellite image tiles—348 pre- and 348 post-Cyclone Freddy—covering three flood-hit urban areas of Blantyre, Malawi. Each image comes with a JSON file of building polygons and a damage level (no, minor, major, or destroyed) assigned by visual inspection following the xBD damage scale. The authors' claim is that this localized resource fills a gap: existing disaster datasets are built on imagery and building styles from other regions, so models trained on them may not transfer to Malawian urban settings. If the dataset is fit for purpose, it gives researchers a benchmark for building detection and damage classification in an African urban context and supports spatial analysis for relocation, drainage, and emergency-response planning.","feed_headline":"Blantyre flood damage mapped in 696 annotated satellite images","feed_subtitle":"Pre/post Cyclone Freddy images with building-level damage labels give African urban machine learning a local benchmark.","key_machinery":"The load-bearing mechanism is the paired-image grid cell: each tile is captured before and after the cyclone, georeferenced from four corner coordinates, so the same building can be compared across time. Building polygons are drawn manually in GIS software and assigned one of four xBD damage levels (0 no damage, 1 minor, 2 major, 3 destroyed); the JSON annotation ties each polygon to pixel and geographic coordinates. The pre/post pairing is what lets a model separate flood damage from ordinary building differences, and the xBD scale is what makes the labels comparable to existing disaster datasets.","core_discovery":"The central discovery is that a compact, locally sourced dataset can capture building-level flood damage in a data-scarce African city. The dataset consists of 696 annotated instances drawn from 1,026 image tiles over Chilobwe, Ndirande, and Chirimba, with each instance pairing a pre-disaster (August 2022) and post-disaster (May 2023) image of the same grid cell; buildings too blurred, too obscured by trees, or split across tiles were excluded. Damage labels follow the four-level xBD scale and are attached to manually drawn building polygons with both pixel and geographic coordinates. The paper argues this resource enables building detection, damage classification, and flood-damage visualization specifically tuned to region-specific construction styles, and notes the dataset was already used in a disaster-damage challenge.","pith_inferences":["A dataset of 696 annotated images from three neighborhoods is small for training deep models from scratch, so its realistic role is likely as a benchmark or fine-tuning set rather than a standalone training corpus.","The labels encode the annotators' visual judgments following the xBD rubric; models trained on it will inherit those judgments, so adding inter-annotator agreement statistics or field-validated subsets would materially strengthen downstream conclusions.","Comparing mwBTFreddy against xBD or other flood datasets could quantify domain shift between Malawian urban building styles and the regions represented in existing benchmarks, which is a direct way to test the paper's motivation.","The dataset could be extended to other Malawian cities or to rural and peri-urban areas the cyclone also hit, where building types and damage patterns likely differ."],"forward_implications":["Machine-learning models for building detection and damage classification can be trained or evaluated on imagery and building styles from a Malawian urban setting rather than only on datasets from other countries.","Because pre- and post-disaster images are paired by grid cell, models can learn change signals that isolate flood effects from background variation.","The JSON annotations support spatial analysis and visualization, so planners can map damage concentrations around the affected mountains and target relocation or drainage decisions.","The 330 unannotated tiles retained separately allow users to reconstruct the broader landscape or study why some areas were excluded.","The dataset's prior use in a public disaster-damage challenge provides an external benchmark for comparing detection and classification approaches on this data."],"supporting_citations":[{"why":"Supplies the xBD damage scale and visual-assessment methodology used to label each building.","marker":"[Gupta et al., 2019]"},{"why":"Provides the public data archive where the dataset and raw images are released.","marker":"[KuyeseraAI, 2024]"},{"why":"Documents the disaster-damage challenge where the dataset was used, serving as an external use case.","marker":"[Zindi, 2025]"},{"why":"Establishes the flood-hazard context in Malawian cities and motivates the need for data-driven risk reduction.","marker":"[Gumindoga et al., 2024]"},{"why":"Supports the premise that satellite imagery can assess flood and earthquake building damage, the foundation of the dataset's purpose.","marker":"[Ma et al., 2022]"},{"why":"Documents Cyclone Freddy's impact on Malawi, establishing the disaster event the dataset targets.","marker":"[Lubanga et al., 2023]"}],"fun_headline_variants":["Cyclone Freddy's flood damage mapped in 696 images","696 annotated satellite images map Malawi flood damage","Pre and post Cyclone Freddy images with damage labels","696 annotated damage labels for Cyclone Freddy in Malawi","Urban Malawi gets 696 building-level flood damage labels"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manually assigned damage labels are accurate enough to be trusted; the annotations were produced by visual assessment by the authors with no reported inter-annotator agreement, independent ground-truth validation, or error analysis, so if those labels are unreliable, models trained or evaluated on the dataset would not measure true flood damage.","fun_headline_variants_meta":{"raw":{"variants":["Cyclone Freddy's flood damage mapped in 696 images","696 annotated satellite images map Malawi flood damage","Pre and post Cyclone Freddy images with damage labels","696 annotated damage labels for Cyclone Freddy in Malawi","Urban Malawi gets 696 building-level flood damage labels"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001734,"raw_usage":{"total_tokens":6798,"prompt_tokens":834,"completion_tokens":5964,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":450,"completion_tokens_details":{"reasoning_tokens":5900}},"tokens_in":450,"tokens_out":5964,"duration_ms":38438,"temperature":1.0,"reasoning_tokens":5900,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T04:21:55.700141+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Select a random sample of annotated buildings and have independent annotators re-label them from the same images, or survey the buildings on the ground; if agreement falls well short of the level needed for reliable training labels, or if field checks contradict the JSON damage levels, the dataset's central value is not established.","supporting_citations":[{"cited_title":"mw BTF reddy (1.0) [data set]","cited_arxiv_id":null,"evidence_quote":"Provides the public data archive where the dataset and raw images are released."},{"cited_title":"K uyeseraai disaster damage and displacement challenge","cited_arxiv_id":null,"evidence_quote":"Documents the disaster-damage challenge where the dataset was used, serving as an external use case."},{"cited_title":"T., Kowe, P., Maviza, A., Magidi, J., Chikwiramakomo, L., Mavaringana, M","cited_arxiv_id":null,"evidence_quote":"Establishes the flood-hazard context in Malawian cities and motivates the need for data-driven risk reduction."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Documents Cyclone Freddy's impact on Malawi, establishing the disaster event the dataset targets."}],"review_version":1}