{"id":"94e8bbbc-4f47-45e5-9965-a8f3fd199674","arxiv_id":"2507.11143","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":4,"one_line_summary":"A U-Net variant with residual blocks, multihead attention, and 23-band inputs reports F1 98.23 on Landslide4Sense detection, F1 93.83 on Bijie detection, and mIoU 76.88 on Nepal segmentation.","lead":"The authors modified a standard U-Net image-segmentation model with residual connections, attention layers, extra spectral bands, and a combined loss, then report high accuracy on three landslide benchmark datasets. General readers may find it useful as a case study of how benchmark tuning choices, not just architecture, drive reported performance.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table XI's Landslide4Sense segmentation F1 (76.90) is not derivable from the paper's own ablation (Table X final: 74.63) and the abstract reports a different mIoU (63.74 vs 65.97), so the central claim is unsupported without a re-run or code.","rationale":"I considered the reader's concern about the 80:20 split and agree it is a valid methodological weakness. However, the more immediate and objective failure is that the paper's own tables contradict each other. Table X is explicitly the step-by-step construction of RMAU-NET, and its final row should be the model reported in Table XI. The +2.27 F1 gap cannot be attributed to the threshold (already included) or to any documented change. The abstract's mIoU mismatch (63.74 vs 65.97) reinforces that the numbers are unreliable. This is a correctness risk independent of evaluation protocol. I therefore recommend keeping the reader's REJECT verdict, but for the more concrete reason of internal inconsistency. The proposed test—re-running the documented configuration—would settle whether the discrepancy is a typo or a substantive overstatement. If the authors can reproduce 76.90 and explain the delta, the split concern would still need to be addressed; if not, the claim is refuted.","tokens_in":13739,"tokens_out":7385,"duration_ms":85741,"concrete_test":"Obtain the authors' training code and logs. Re-run the exact final configuration described by Table X's last row on the same 80:20 Landslide4Sense split—combined Focal+IoU loss, 23 bands, three resolution heads, Res-Conv with multihead attention, and threshold 0.95—and check whether the test F1 is 74.63 (as in Table X) or 76.90 (as in Table XI). If it is 74.63, Table XI overstates performance; if it is 76.90, the authors must provide the precise change (e.g., multi-task training) that produced the gain and share a reproducible script.","verdict_should_be":"UNCHANGED","load_bearing_attack":"RMAU-NET's reported performance rests entirely on Table XI. The paper's own development path (Table X) ends with all the claimed components—combined Focal+IoU loss, 23 input bands, three heads, Res-Conv and multihead attention, and the 0.95 threshold—at Landslide4Sense F1=74.63, mIoU=65.97. Table XI then lists the same task at F1=76.90, a +2.27 increase with no stated change in training or architecture. Section V only adds a detection head; the paper does not say the segmentation loss is multi-task or that the model is retrained differently. The abstract separately reports mIoU=63.74 for the same dataset, while Table XI gives 65.97. These discrepancies mean the advertised numbers cannot be reproduced from the experimental record. A properly held-out split would not fix this: even if the 80:20 split were perfectly unbiased, the final F1 is not derived from the documented experiments, so the central claim lacks support.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes RMAU-NET, a U-Net variant combining residual-convolutional blocks, a multihead attention layer, multi-resolution segmentation heads, 23 input bands, a combined Focal+IoU loss, and threshold-based post-processing, for landslide detection and segmentation from remote sensing images. The authors evaluate the model on three public datasets (Landslide4Sense, Bijie, Nepal) and report F1=98.23 for Landslide4Sense detection, F1=93.83 for Bijie detection, mIoU=65.97 for Landslide4Sense segmentation, and mIoU=76.88 for Nepal segmentation, claiming state-of-the-art performance. The evaluation is developed through an ablation study on an 80:20 split of the Landslide4Sense training set, with the post-processing threshold selected on that same split.","tokens_in":13943,"tokens_out":4525,"duration_ms":53308,"significance":"If the reported numbers were reproducible, RMAU-NET would be a practically valuable contribution to automated landslide mapping, since it addresses both detection and segmentation on publicly available benchmarks and the ablation study systematically isolates the contribution of each component. The use of three public datasets and the clear ablation progression from Table III to Table X are strengths that would help practitioners. However, the central empirical claim is compromised by evaluation on the development split and by internal inconsistencies between tables and the abstract; without code or a corrected evaluation protocol, the reported performance is not verifiable.","major_comments":[{"comment":"The Landslide4Sense evaluation uses a single 80:20 split of the training set, and the same split is used both to select every component (including the 0.95 threshold in Table IX) and to produce the final numbers in Table XI; because the threshold was chosen by maximizing F1/mIoU on this split, the reported F1=98.23 detection and F1=76.90/mIoU=65.97 segmentation numbers are not unbiased estimates of performance on unseen data. A separate validation split, or nested cross-validation, and a final evaluation on the official Landslide4Sense test set are needed before the headline results can be taken as evidence of generalization.","section":"Section II-C and Table IX/Table XI"},{"comment":"The final row of Table X lists the complete RMAU-NET configuration on Landslide4Sense segmentation as F1=74.63, mIoU=65.97, while Table XI reports F1=76.90 for the same task with no described change other than the addition of a detection head; the paper does not state that the model is retrained with a multi-task loss, so the +2.27 F1 increase is unexplained. The abstract also reports mIoU=63.74 for the same dataset, contradicting Table XI's 65.97. These discrepancies mean the central claim in Section V is not reproducible from the experimental record.","section":"Table X vs Table XI"},{"comment":"The paper claims state-of-the-art results, but Table XI contains no comparison on Landslide4Sense; only the proposed RMAU-NET is listed for that dataset, and the comparison rows cover only Nepal and Bijie. Without a comparison to published methods on the same Landslide4Sense split, or on the official test set, the state-of-the-art claim is unsupported.","section":"Section V and Table XI"},{"comment":"The threshold of 0.95 is applied to Bijie and Nepal after being selected on the Landslide4Sense 80:20 split, with no sensitivity analysis or validation on those datasets; since threshold is a free parameter, the cross-dataset results in Table XI are not independent evidence of robustness. The authors should report thresholds chosen on a validation set per dataset, or show that the chosen threshold transfers with bounded performance loss.","section":"Section IV-D and Table IX"}],"minor_comments":[{"comment":"The text states the best F1 score is 74.463, but Table X and Table IX report 74.63; the text and tables should be reconciled.","section":"Section IV-D"},{"comment":"The text refers to the proposed architecture as shown in Fig. 8, but Fig. 7 contains the RMAU-NET architecture and Fig. 8 shows segmentation results; the figure references should be corrected.","section":"Section V"},{"comment":"The sentence 'remaining the 70:30 ratio for the train and test sets' appears to contain a typo; it should likely read 'retaining the 70:30 ratio'.","section":"Section II-C"},{"comment":"There are typographical errors such as 'RBG' and 'landsline'; a copyedit pass is needed.","section":"Section II-A"},{"comment":"The loss mixing coefficient alpha is fixed at 0.5 without reporting sensitivity to its value; since alpha is a free parameter, a brief study or robustness check would strengthen the loss-function analysis.","section":"Equation (2)"},{"comment":"No code or complete training details (batch size, learning rate, number of parameters, inference time, and the exact multi-task training protocol for RMAU-NET) are provided, which limits reproducibility.","section":"Experimental settings"}],"recommendation":"reject","confidential_remarks":"The manuscript's central claim is not supported by the experimental record as written: the internal discrepancy between Table X and Table XI, together with the selection of the post-processing threshold on the same split used for final evaluation, are load-bearing issues that cannot be fixed by copyediting. Unless the authors can supply code and rerun experiments with a proper held-out split, I do not see a path to acceptance in this venue."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's my read of arXiv:2507.11143. The headline: the paper has a genuinely thorough ablation study, but the reported headline numbers don't survive contact with the paper's own tables. The threshold and most architecture choices are selected on the test split, and then the final Table XI numbers don't match the ablation's final step.\n\nWhat's actually new: the specific RMAU-NET combination—residual conv blocks, a multi-head attention from the authors' IGARSS paper, multi-resolution heads, 23 bands, and a combined Focal+IoU loss. Each component is known, but the systematic comparison across loss functions, band engineering, and attention variants on a public benchmark is useful. The ablation path in Table X is clear and honestly labels each addition.\n\nThe soft spots are not minor. First, the evaluation protocol: Table IX sweeps the post-processing threshold on the test split (Landslide4Sense 80:20 split) and picks 0.95. The architecture choices are also made using that same split. That's textbook test-set overfitting; the reported F1=74.63 at the end of the ablation is likely optimistic. Second, and worse: Table XI reports F1=76.90 for Landslide4Sense segmentation, but Table X ends at 74.63 with the same components. There is no explanation for the +2.27 jump. The abstract also says mIoU=63.74 for that dataset, while Table XI gives 65.97. These internal contradictions mean the central claim—that RMAU-NET achieves F1=98.23 detection and mIoU=65.97 segmentation—is not supported by the paper's own evidence. No code, seeds, or error bars either.\n\nThe state-of-the-art comparisons are thin: Nepal only against ResNet and U-Net from the original study, Bijie only against DDTL variants. That weakens the \"state-of-the-art\" language.\n\nIf the evaluation were fixed, the engineering contribution would be a plausible incremental advance. But as written, the numbers cannot be reproduced from the experimental record. A re-run with proper held-out validation and a clarified relationship between Table X and Table XI would make this a reasonable applied paper.\n\nWho should read it: people in the remote-sensing landslide community, mostly as a cautionary example of evaluation pitfalls. It deserves a serious referee—the ablation is substantial and the problem is important—but the referee should demand corrected evaluation, code, and a resolution of the internal inconsistencies.\n\nRecommendation: send to peer review, but expect heavy revision. My own verdict, if asked, is reject under the current reporting.","headline":"A thorough ablation study undermined by test-set selection of the threshold and architecture, and by internal contradictions between the ablation table and the final results table.","tokens_in":14515,"tokens_out":2350,"would_cite":false,"duration_ms":25591,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A residual multihead-attention U-Net reports top landslide detection and segmentation accuracy on three public remote-sensing benchmarks.","keywords":["landslide detection","landslide segmentation","remote sensing","U-Net","multihead attention","residual learning","Landslide4Sense","Sentinel-2"],"falsifier":"Run RMAU-NET on a landslide dataset from a region and time period completely absent from training, or on the official 800-image hidden test set of Landslide4Sense, and check whether the F1 and mIoU values stay near 98.23 and 65.97; a large drop would show the reported gains depend on seeing the test images during development.","tokens_in":13500,"feed_emoji":"🛰️","tokens_out":6708,"duration_ms":67470,"temperature":0.7,"pith_summary":"The paper tries to establish that a single deep-learning architecture, RMAU-NET, can automatically detect and segment landslides from free satellite imagery with accuracy high enough to be useful in real landslide observation systems. It reports F1 scores of 98.23 on Landslide4Sense detection and 93.83 on Bijie detection, and mIoU scores of 65.97 on Landslide4Sense segmentation and 76.88 on Nepal segmentation. The authors build the architecture by starting from a U-Net baseline and adding, one at a time, generated spectral bands, a combined Focal and IoU loss, multiple prediction heads at different resolutions, residual-convolutional blocks, a multihead attention layer, and a final threshold. Each addition is shown to improve the scores, which is the evidence that the full design, not any single trick, is what delivers the result.","feed_headline":"Attention U-Net tops landslide benchmarks on free satellite data","feed_subtitle":"Detection F1 reaches 98.23 and segmentation mIoU 65.97 on the Landslide4Sense benchmark.","key_machinery":"The load-bearing object is the RMAU-NET architecture itself. It is a U-Net whose double-convolution blocks are replaced by residual-convolutional layers, with a multihead attention layer inserted after every convolutional layer; the attention layer pools the feature map along each of its width, height, and channel dimensions, applies standard multihead attention to the pooled maps, and multiplies the result back onto the original feature map. Around this backbone the model adds 9 generated bands to the 14 original satellite bands, averages three prediction heads at 64x64, 128x128, and 256x256 resolution, trains with a combined Focal and IoU loss, and applies a 0.95 threshold to the output mask. The argument proceeds by ablation: each component raises F1 and mIoU on Landslide4Sense, from 67.83 and 60.01 for the plain U-Net baseline to 74.63 and 65.97 for the complete RMAU-NET.","core_discovery":"The central claim is that RMAU-NET, an encoder-decoder with residual skip blocks and a three-dimensional multihead attention mechanism, outperforms previous landslide mapping models on all three benchmark datasets it is tested on. On Landslide4Sense the model reaches 98.23 F1 for image-level detection and 65.97 mIoU for pixel-level segmentation; on Bijie it reaches 93.83 F1 for detection; on Nepal it reaches 76.88 mIoU for segmentation. The authors interpret these numbers as showing the architecture is robust across different sensors, regions, and landslide ratios, and therefore suitable for integration into operational landslide observation systems.","pith_inferences":["Because the Landslide4Sense test split was cut from the same training set and the 0.95 threshold was chosen on that split, the reported numbers likely overstate performance on truly unseen terrain; a geographically or temporally disjoint evaluation would be a stricter test.","The Nepal result (mIoU 76.88 with F1 69.43) suggests the model favors precise boundaries over recall, which may be suitable for inventory maps but could miss smaller landslides in emergency response.","The three-dimensional multihead attention design is dataset-agnostic and could be tested on other sparse-target segmentation problems, such as building damage or wildfire scar mapping.","A simple extension would be to learn the post-processing threshold per dataset instead of fixing it, which would remove a source of test-set leakage."],"forward_implications":["The same architecture handles both image-level detection and pixel-level segmentation, so a single model can be deployed for landslide early-warning and inventory mapping.","Generated bands (normalization, NDVI/NDMI/NBR, filters, gradients, edges) contribute beyond the original 14 bands, suggesting spectral and textural indices are worth adding for other remote-sensing tasks.","Multiple resolution heads and residual-plus-attention blocks give a concrete recipe for improving U-Net-style models on imbalanced segmentation problems.","The reported scores on Landslide4Sense, Bijie, and Nepal position RMAU-NET as a strong candidate baseline for future landslide mapping work."],"supporting_citations":[{"why":"Provides the Landslide4Sense benchmark: 14-band images, masks, and the competition setting the ablation study is run on.","marker":"[19]"},{"why":"Provides the Bijie dataset and the DDTL family of models that RMAU-NET is compared against for detection.","marker":"[37]"},{"why":"Provides the Nepal dataset and the ResNet and U-Net segmentation baselines that RMAU-NET is compared against.","marker":"[38]"},{"why":"Introduces the multihead attention layer used inside RMAU-NET.","marker":"[61]"},{"why":"CutMix augmentation, used for online data augmentation in the baseline and final model.","marker":"[45]"},{"why":"Focal loss, one of the two components of the combined loss.","marker":"[48]"},{"why":"IoU loss, the other component of the combined loss.","marker":"[50]"},{"why":"Multihead attention from the transformer literature, used inside the proposed attention layer.","marker":"[62]"}],"fun_headline_variants":["RMAU-NET: residual multihead attention U-Net tops landslide benchmarks","Attention U-Net hits 98.23 F1 and 65.97 mIoU on landslides","Residual attention U-Net sets landslide benchmarks on free satellite data","98.23 F1 landslide detection with residual attention U-Net","Multihead attention U-Net maps landslides at 98.23 F1, 65.97 mIoU"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The evaluation assumes that carving an 80:20 split from the Landslide4Sense training set gives a valid held-out test, and that tuning the final threshold on that same split does not inflate the reported scores.","fun_headline_variants_meta":{"raw":{"variants":["RMAU-NET: residual multihead attention U-Net tops landslide benchmarks","Attention U-Net hits 98.23 F1 and 65.97 mIoU on landslides","Residual attention U-Net sets landslide benchmarks on free satellite data","98.23 F1 landslide detection with residual attention U-Net","Multihead attention U-Net maps landslides at 98.23 F1, 65.97 mIoU"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001354,"raw_usage":{"total_tokens":5474,"prompt_tokens":903,"completion_tokens":4571,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":519,"completion_tokens_details":{"reasoning_tokens":4459}},"tokens_in":519,"tokens_out":4571,"duration_ms":35567,"temperature":1.0,"reasoning_tokens":4459,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T17:15:16.755188+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run RMAU-NET on a landslide dataset from a region and time period completely absent from training, or on the official 800-image hidden test set of Landslide4Sense, and check whether the F1 and mIoU values stay near 98.23 and 65.97; a large drop would show the reported gains depend on seeing the test images during development.","supporting_citations":[{"cited_title":"The outcome of the 2022 landslide4sense competi- tion: Advanced landslide detection from multisource satellite imagery,","cited_arxiv_id":null,"evidence_quote":"Provides the Landslide4Sense benchmark: 14-band images, masks, and the competition setting the ablation study is run on."},{"cited_title":"Landslide detection from an open satellite imagery and digital elevation model dataset using attention boosted convolutional neural networks,","cited_arxiv_id":null,"evidence_quote":"Provides the Bijie dataset and the DDTL family of models that RMAU-NET is compared against for detection."},{"cited_title":"Convolutional neural networks applied to semantic segmentation of landslide scars,","cited_arxiv_id":null,"evidence_quote":"Provides the Nepal dataset and the ResNet and U-Net segmentation baselines that RMAU-NET is compared against."},{"cited_title":"Landslide detection and segmentation using remote sensing images and deep neural networks,","cited_arxiv_id":null,"evidence_quote":"Introduces the multihead attention layer used inside RMAU-NET."},{"cited_title":"Cutmix: Regularization strategy to train strong classifiers with localizable features,","cited_arxiv_id":null,"evidence_quote":"CutMix augmentation, used for online data augmentation in the baseline and final model."},{"cited_title":"Focal loss for dense object detection,","cited_arxiv_id":null,"evidence_quote":"Focal loss, one of the two components of the combined loss."},{"cited_title":"Generalized intersection over union: A metric and a loss for bounding box regression,","cited_arxiv_id":null,"evidence_quote":"IoU loss, the other component of the combined loss."}],"review_version":1}