{"id":"a34f0098-d720-4a15-b272-8f8bee1a7b88","arxiv_id":"2507.19818","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":6,"one_line_summary":"A hierarchical U-Net plus expert model plus Bayesian smoothing pipeline improves flood mapping on PlanetScope imagery for the Dubai 2024 event, with reported F1 gains up to 29%.","lead":"FM-LC is a three-stage pipeline that maps urban floods from 3-meter PlanetScope satellite imagery by first segmenting land cover, then refining the most confused class with a specialized expert model, and finally smoothing boundaries with Bayesian inference. It reports F1-score gains of up to 29% on the April 2024 Dubai flood, but its evaluation is weakened by using the test set to choose which class to refine.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Test-set leakage via expert-class selection invalidates reported F1 improvements; re-evaluation with selection on a validation split is required.","rationale":"The reader's stated weakest assumption (label accuracy at 3m from 10m Sentinel-2) is real but secondary. The paper's own experimental description reveals a more direct threat to the central claim: the test set is used to decide which class receives the expert model. Section IV.A explicitly says the vulnerable class is identified from the sampled test-set, and the expert is then trained accordingly. Evaluating on the same test set after making that choice constitutes test-set leakage, biasing the reported F1 gains—especially the large Vegetation improvement, which is exactly the class that would be flagged. Even if the labels were perfect, the comparison would not support the claimed advantage. The reader did identify the test-set leakage in their rationale, but did not make it the primary weakest assumption, hence 'partial' agreement. I also verified the headline numbers against Table I: the average absolute F1 gain is about 6 percentage points, not 12%; 'up to 29%' is a per-class relative gain. The proposed framework could be rehabilitated with a nested validation protocol and honest reporting, but as written the central claim is not reliably supported, so the verdict should remain REJECT.","tokens_in":7650,"tokens_out":4012,"duration_ms":45628,"concrete_test":"Repeat the evaluation with a strict split: train 70%, validation 10%, test 20%. Choose the expert class using only validation-set confusion or F1; train the expert; apply Bayesian smoothing; report metrics once on the untouched test set. Compare with Table I. Additionally, re-run the pipeline with the expert class fixed to each of the four classes and report all four results; if the Vegetation F1 gain is not reproduced when selection is blind, the original 'up to 29%' claim is an artifact of test-set selection. As a desk check, recompute the average F1 improvement from Table I (absolute and relative) to verify the 'over 12%' claim; if it is ~6 pp / ~8%, the abstract's number needs correction.","verdict_should_be":"REJECT","load_bearing_attack":"The load-bearing weakness is in Section IV.A: 'the vulnerable class can be identified by the sampled test-set, based on this, the class-specific expert can be trained.' This means the expert model's target class is selected using the same test set that later produces the reported metrics. Because the expert is deliberately trained to fix the class the baseline misclassifies most on that exact test set, the test-set F1 for that class—Vegetation in Table I, which accounts for the +0.215 gain—is optimistically biased. This is test-set-driven model selection, not an honest held-out comparison. The hierarchical design may well help, but the experiment as described cannot distinguish genuine improvement from selection favoring the most error-prone class. The claim 'over 12% average F1 improvement' is also not supported by Table I: absolute average F1 improvement over UNet(m) is about 0.06 (6 percentage points) and relative average improvement about 8%; 'up to 29%' refers to a single-class relative gain. Finally, the 10m Sentinel-2 labels manually corrected on 3m PlanetScope, without independent verification, leave ground-truth uncertainty, but the test-set leakage alone is sufficient to invalidate the central quantitative claim.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript proposes FM-LC, a three-stage hierarchical framework for urban flood mapping from eight-band PlanetScope imagery. The first stage is a multi-class U-Net producing water, vegetation, built area, and bare ground labels; the second trains a binary 'expert' U-Net to separate the most confused class from its partner; the third applies a localized Bayesian smoothing to the logits. The method is evaluated on the April 2024 Dubai flood using pre- and post-event PlanetScope composites over a 1,535 km2 area, with 1,188 image patches. The authors report average F1-score improvements of up to 29% and conclude 'over 12% average F1 improvement', supported by a confusion matrix (overall accuracy 96.09%, Kappa 0.939) and qualitative maps. The central quantitative claim is compromised by using the held-out test set to decide which class receives the expert model, and the reported aggregate improvements are not reproducible from Table I as stated.","tokens_in":7955,"tokens_out":5282,"duration_ms":55391,"significance":"If the reported gains were obtained from a clean held-out protocol, the work would be practically significant: it targets a real disaster event, uses publicly available high-resolution imagery, and proposes a modular way to sharpen confused classes that is easy to extend. The explicit reporting of per-class dice/precision/F1 and the discussion of why overall accuracy is insensitive to rare-class errors are strengths. However, the lack of a validation-based model-selection protocol, the absence of error bars, and the mismatch between the abstract/conclusion numbers and Table I currently prevent the paper from supporting its central claim. The hierarchical idea itself is worth pursuing, but the evidence as presented is not yet reliable.","major_comments":[{"comment":"The reported F1 improvements are not out-of-sample because the same 30% test split is used to select the expert class. The text states 'the vulnerable class can be identified by the sampled test-set, based on this, the class-specific ‘expert’ can be trained', and Table I is then computed on the same held-out set. This is equivalent to tuning a hyperparameter on the test set: the large Vegetation F1 gain (+0.235 in Table I) is expected by construction even if the expert provides no genuine improvement. Please select the vulnerable class on a validation split and evaluate only on a test set that is never inspected during design; if the sample is too small, use nested cross-validation or report the selection rule a priori.","section":"Section IV.A"},{"comment":"The quantitative claims do not match Table I. The abstract claims 'average F1-score improvements of up to 29% across all land-cover classes' and the conclusion says 'over 12% average F1 improvement'. From Table I, the macro-average F1 is 0.844 for UNet(m) and 0.904 for the hierarchical model, an absolute gain of 0.06 (6 percentage points) and a relative gain of roughly 7%; 'up to 29%' is only the single-class relative improvement for Vegetation (0.913 vs. 0.698). Please state absolute and relative gains per class and for the macro-average, and revise the abstract and conclusion to match the table.","section":"Abstract and Section V"},{"comment":"No measure of uncertainty is reported, and only one baseline (multi-class U-Net) is compared. The framework has several free hyperparameters (W, alpha, sigma^2, tau_k, lambda in Section III), and the expert-class choice is data-dependent; without repeated runs or ablations, it is impossible to tell whether the Vegetation gain is robust or an artifact of the particular split and selection. Please report mean +/- standard deviation over at least three random seeds, and include ablations (e.g., multi-class U-Net with Bayesian smoothing alone, expert without Bayesian smoothing, expert on a non-selected class) to isolate each stage's contribution.","section":"Section IV.A and Table I"},{"comment":"The label quality is not established. Labels were 'derived from 2023 open-access Sentinel-2 data at 10 m resolution and manually corrected via the 14 April PlanetScope mosaic'; these labels are used as ground truth for training and for the reported metrics at 3 m. Because the main improvement is precisely in the water/vegetation distinction, systematic label errors in ambiguous pixels could bias the confusion matrix and F1 numbers. Please report a label-quality assessment, such as an independent review of a random pixel sample or comparison with an independent high-resolution flood map.","section":"Section II"}],"minor_comments":[{"comment":"The heading 'The Backbond U-net Model' contains a typo; it should read 'The Backbone U-Net Model'.","section":"Section III.A"},{"comment":"The footnote 'Clarify: Here i and j are the pixel id' should be integrated into the main text rather than left as a footnote.","section":"Section III.A"},{"comment":"Reference [12] is listed as 'forthcoming manuscript'; this is not a verifiable citation and should be replaced with a published work or removed.","section":"References"},{"comment":"Figure 4 shows Recall, Kappa, and per-class accuracy without numerical values or error bars; please add the underlying numbers to the figure or provide a supplementary table.","section":"Section IV.A and Figure 4"},{"comment":"Figure 3 shows only FM-LC segmentation, not the baseline U-Net output; adding side-by-side baseline maps would support the claim of 'sharper flood delineations'.","section":"Section IV.B and Figure 3"},{"comment":"The Hierarchical column mixes metric values with deltas (e.g., '0.921+0.002'); use separate columns for the value and the change, and define the baseline for each delta.","section":"Table I"}],"recommendation":"major_revision","confidential_remarks":"The test-set leakage is serious and the aggregate numbers are overstated, but the core idea and dataset are relevant for TGRS. I would encourage a proper validation-based re-run and metric clarification rather than a desk rejection; if the authors cannot provide an honest held-out evaluation, the paper should not be published."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Read the FM-LC paper. Bottom line: the engineering is sensible, but the evaluation is compromised by test-set leakage, so the headline numbers don't stand.\n\nWhat's actually new: three known pieces—multi-class U-Net, a binary expert for the class the baseline confuses most, and Bayesian smoothing—assembled into a pipeline for PlanetScope flood mapping in the Dubai 2024 event. That combination is new for this application. The paper describes the method clearly, the qualitative figures look plausible, and the confusion matrix is internally consistent. The Bayesian smoothing is a reasonable way to avoid boundary artifacts.\n\nThe problem is Section IV.A: 'the vulnerable class can be identified by the sampled test-set, based on this, the class-specific expert can be trained.' That is classic test-set-driven model selection. The expert is hand-picked to fix the class that the baseline misclassifies most on the very test set used to report F1. So the vegetation-class gain of +0.235 in Table I is not an independent measurement. The claim of 'over 12% average F1 improvement' is also not supported by the table: the average absolute gain over UNet(m) is about 6 points, and the 'up to 29%' is a single-class relative number. The ground truth, from 10m Sentinel-2 manually corrected on 3m PlanetScope, is another source of uncertainty, though I'd call that secondary. No error bars, no code, and only one baseline round out the reproducibility problems.\n\nThe core idea is not bad. A hierarchical refinement where a specialist model handles the most confused class is a reasonable strategy. The flaw is in the evaluation protocol, not in the method itself. With a proper validation split for choosing the expert class and a clearer reporting of the aggregate gain, this could be a solid applied paper.\n\nWho gets value: remote sensing practitioners working with PlanetScope or high-res imagery in arid zones, and anyone who wants a clean example of how test-set leakage can sneak into a seemingly reasonable evaluation.\n\nMy recommendation: send it to peer review, but only with the understanding that the evaluation must be redone. A good referee could help the authors fix this.","headline":"Sensible pipeline, invalidated evaluation: test-set leakage in expert-class selection means the headline F1 gains are not credible; the paper needs a redo before it can be trusted.","tokens_in":8405,"tokens_out":2550,"would_cite":false,"duration_ms":26255,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A three-stage hierarchical framework, FM-LC, claims to improve urban flood mapping on PlanetScope imagery by adding a class-specific expert model and Bayesian smoothing to a multi-class U-Net.","keywords":["urban flood mapping","PlanetScope imagery","semantic segmentation","U-Net","Bayesian smoothing","land cover classification","Dubai flood 2024","remote sensing"],"falsifier":"Compare FM-LC's flood maps against independently digitized water boundaries from sub-meter imagery or field surveys for the same Dubai dates. If the expert stage aligns with the Sentinel-2-derived label corrections rather than those independent boundaries, the reported 12–29 percentage-point gains would not reflect true flood-detection skill.","tokens_in":7467,"feed_emoji":"🌊","tokens_out":5755,"duration_ms":65234,"temperature":0.7,"pith_summary":"This paper claims that a three-stage hierarchical framework, FM-LC, improves fine-scale urban flood mapping from PlanetScope imagery by first segmenting land cover with a multi-class U-Net, then using a lightweight binary expert network to correct the most confused class, and finally applying Bayesian smoothing to clean boundaries. It tests this on the April 2024 Dubai storm, where water and moist vegetation are spectrally hard to separate. The authors report average F1-score gains of over 12 percentage points, with the largest single-class gain exceeding 29 percentage points on the Dice metric, and much sharper flood outlines compared with a single-stage U-Net baseline. If true, this would make daily, high-resolution flood tracking more reliable in arid cities and support post-event recovery monitoring.","feed_headline":"Three-stage AI sharpens Dubai flood maps by up to 29 percent","feed_subtitle":"If it holds, cities can track flood extent and recovery daily from 3 m satellite imagery.","key_machinery":"The load-bearing mechanism is the hierarchy itself: a ResNet-34-encoded U-Net adapted to eight-band input produces per-pixel class probabilities; an expert binary U-Net, trained only on pixels whose coarse labels fall in the confused pair {k, k'}, outputs a mask that overwrites the coarse labels; and a Bayesian smoothing module computes local top-k logit means and variances inside a sliding window, then forms a Gaussian prior to re-estimate each logit via a posterior-mode update. The final hard map comes from softmax over the smoothed logits. This combination lets the expert specialize without forgetting global context, and the Bayesian step suppresses boundary artifacts introduced by overwriting.","core_discovery":"FM-LC is built on the observation that a single multi-class U-Net systematically confuses water with vegetation in arid urban scenes. The paper's central claim is that this confusion can be reduced by a dedicated expert binary U-Net trained only on the confused class pair, whose output overrides the coarse labels, followed by a local Bayesian logit-smoothing step that blends each pixel's own confidence with neighborhood statistics. On the held-out Dubai test set, the framework lifts vegetation F1 from 0.698 to 0.913 and vegetation Dice from 0.641 to 0.896, with smaller gains in built area and bare ground, and produces visually crisper inundation boundaries. The authors present this as evidence that hybrid multi-stage pipelines are a practical route to operational flood mapping at 3 m resolution.","pith_inferences":["The paper's evidence is confined to one storm event; a stronger test would apply FM-LC to a second flood with independently produced labels, ideally at PlanetScope resolution rather than corrected Sentinel-2 labels.","The 10 m Sentinel-2 origin of the labels means some improvements may reflect the expert model learning the labelers' corrections rather than true surface water; comparing against hand-digitized high-resolution water lines would settle this.","The per-class accuracy metric is dominated by large classes, so the authors' own explanation implies that small but important classes such as vegetation deserve class-weighted reporting.","A practical extension is to make the vulnerable-class selection automatic during inference, since the current pipeline relies on identifying the confused class from a sampled test set before training the expert."],"forward_implications":["If the reported gains hold, emergency managers can use daily PlanetScope imagery to track flood extent and recovery with per-pixel land-cover labels rather than coarse binary water masks.","The expert-model stage isolates the specific spectral confusion that limits single-stage networks, so the framework can be repurposed for other confused class pairs by retraining only the expert.","The Bayesian smoothing step makes final maps spatially coherent enough for direct use as GeoTIFF inputs to hydrological or hydrodynamic models.","Because the framework is modular, adding a temporal branch or domain adaptation for other arid cities should not require redesigning the whole pipeline."],"supporting_citations":[{"why":"Establishes that PlanetScope outperforms coarser sensors for flood damage mapping, motivating the choice of data source.","marker":"[5]"},{"why":"Shows high-resolution PlanetScope improves land-use/land-cover mapping and hydrological modeling in an arid region, supporting the approach.","marker":"[6]"},{"why":"Supplies the previously validated U-Net model that FM-LC extends with expert and smoothing stages.","marker":"[12]"},{"why":"Defines the eight-band, 3 m PlanetScope product used as model input.","marker":"[14]"},{"why":"Source of the 10 m Sentinel-2 data from which training labels were derived and manually corrected.","marker":"[15]"},{"why":"Provides the Bayesian post-processing rationale used for the smoothing step.","marker":"[16]"},{"why":"Defines the segmentation evaluation metrics, including F1, precision, and recall, used for comparison.","marker":"[17]"},{"why":"Provides the Kappa statistic reported alongside overall accuracy.","marker":"[18]"}],"fun_headline_variants":["Hierarchical AI cuts urban flood map errors by 29%","Two-step AI fixes water-vegetation confusion in flood maps","Staged neural net sharpens Dubai flood mapping by 29%","Expert AI + Bayesian smoothing boost flood map accuracy","Tiered model improves arid flood mapping by 29%"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the manually corrected Sentinel-2 labels at 10 m resolution are accurate enough at 3 m PlanetScope scale to serve as ground truth for both training and evaluation, especially in pixels where water and vegetation are ambiguous.","fun_headline_variants_meta":{"raw":{"variants":["Hierarchical AI cuts urban flood map errors by 29%","Two-step AI fixes water-vegetation confusion in flood maps","Staged neural net sharpens Dubai flood mapping by 29%","Expert AI + Bayesian smoothing boost flood map accuracy","Tiered model improves arid flood mapping by 29%"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1581,"prompt_tokens":965,"completion_tokens":616,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":581,"completion_tokens_details":{"reasoning_tokens":532}},"tokens_in":581,"tokens_out":616,"duration_ms":7421,"temperature":1.0,"reasoning_tokens":532,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:58:57.576415+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compare FM-LC's flood maps against independently digitized water boundaries from sub-meter imagery or field surveys for the same Dubai dates. If the expert stage aligns with the Sentinel-2-derived label corrections rather than those independent boundaries, the reported 12–29 percentage-point gains would not reflect true flood-detection skill.","supporting_citations":[{"cited_title":"Evaluating flood damage to paddy rice fields using planetscope and sentinel-1 data in north-western nigeria: towards potential climate adaptation strategies,","cited_arxiv_id":null,"evidence_quote":"Establishes that PlanetScope outperforms coarser sensors for flood damage mapping, motivating the choice of data source."},{"cited_title":"Impact of utilizing high-resolution planetscope imagery on the accuracy of lulc mapping and hydrological modeling in an arid region,","cited_arxiv_id":null,"evidence_quote":"Shows high-resolution PlanetScope improves land-use/land-cover mapping and hydrological modeling in an arid region, supporting the approach."},{"cited_title":"Previously validated u-net model for urban flood mapping,","cited_arxiv_id":null,"evidence_quote":"Supplies the previously validated U-Net model that FM-LC extends with expert and smoothing stages."},{"cited_title":"Planetscope imagery product specifications,","cited_arxiv_id":null,"evidence_quote":"Defines the eight-band, 3 m PlanetScope product used as model input."},{"cited_title":"Sentinel-2 user handbook,","cited_arxiv_id":null,"evidence_quote":"Source of the 10 m Sentinel-2 data from which training labels were derived and manually corrected."},{"cited_title":"Bayesian post-processing methods for jitter mitigation in sampling,","cited_arxiv_id":null,"evidence_quote":"Provides the Bayesian post-processing rationale used for the smoothing step."},{"cited_title":"Image segmentation evaluation: a survey of methods,","cited_arxiv_id":null,"evidence_quote":"Defines the segmentation evaluation metrics, including F1, precision, and recall, used for comparison."},{"cited_title":"Coefficient kappa: Some uses, misuses, and alternatives,","cited_arxiv_id":null,"evidence_quote":"Provides the Kappa statistic reported alongside overall accuracy."}],"review_version":1}