{"id":"314ceedd-5ee5-463c-b35f-781c50ea6da2","arxiv_id":"2504.13776","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"On the Pereira Landsat-8 wildfire dataset, a Swin-Unet achieves 89.93% IoU (0.93% above the best published CNN baseline), while a custom CNN U-Net achieves 93.58% IoU.","lead":"This paper applies two Vision Transformer architectures to satellite wildfire detection, finding that a well-tuned CNN still performs best with 93.58% IoU, while the best ViT slightly improves over published CNN baselines. A generalist might read it to see whether transformers are ready to replace CNNs for remote sensing segmentation; they are not yet.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Table 1's own IoU values are internally inconsistent with the reported precision/recall (Swin-Unet: P=88.28, R=92.30 imply IoU≈82.2, not 89.93), so the headline ViT outperformance may compare a differently defined IoU to the Pereira baselines.","rationale":"The reader identified a real weakness in comparing published baseline numbers to freshly computed metrics, but the more load-bearing problem is internal to the paper's own table. For binary segmentation, precision, recall, and F-score uniquely determine the foreground Jaccard IoU when all are computed from the same global confusion matrix. The reported F-scores are consistent with the reported precision and recall, yet the reported IoU values for all three in-house models violate the exact relation, e.g., Swin-Unet's numbers imply an IoU near 82%, not 89.93%. Because the baseline rows from Pereira are consistent with the standard Jaccard relation, the table appears to mix two different IoU definitions. If the authors' IoU is a class-averaged mean IoU, then it is not comparable to the foreground IoU quoted for Pereira's models, and the central claim that a ViT outperforms the CNN baseline is not established. This is a concrete, fixable issue rather than an integrity problem; the paper provides public code and a public dataset, so an independent rerun can settle it. I therefore keep the verdict conditional, but the revision should require clarification and recomputation of the IoU metric, not merely a discussion of center-cropping and statistical significance.","tokens_in":7786,"tokens_out":13721,"duration_ms":128011,"concrete_test":"Inspect the evaluation code in the public repository (https://github.com/amanbasu/wildfire-detection), rerun the saved Swin-Unet model on the 9044-image manually annotated test set, and recompute IoU from the raw per-pixel confusion matrix using only the fire class: IoU = TP/(TP+FP+FN), and separately as a class-averaged (fire+background)/2 IoU. If the fire-class IoU is ≈82.2%, then the Table 1 value of 89.93 is not the standard foreground IoU and cannot be compared with the Pereira baseline's 89.00; if the fire-class IoU is 89.93, then the precision and recall values reported in Table 1 must be re-derived.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that Swin-Unet outperforms the best published CNN baseline rests on the IoU comparison in Table 1: 89.93% versus 89.00%. That number is numerically incompatible with the same row's precision and recall under the standard binary foreground Jaccard formula IoU = PR/(P+R-PR). For Swin-Unet, P=88.28, R=92.30 give IoU≈82.2%, not 89.93%. TransUNet (P=88.46, R=86.88) gives IoU≈78.0%, not 87.49%, and the authors' own UNet (P=93.37, R=93.96) gives IoU≈88.1%, not 93.58%. The reported F-scores match 2PR/(P+R) closely, showing that precision, recall, and F-score are mutually consistent; only the IoU column is out of line for all in-house models, while the Pereira baseline rows are consistent with foreground Jaccard. This strongly suggests the in-house IoU was computed as a class-averaged (fire and background) or otherwise differently averaged quantity, while the baseline IoU values are foreground Jaccard. If so, the headline 'one ViT outperforms the baseline CNN by 0.93%' compares two different metrics and the conclusion is unsupported. The paper does not define the IoU computation, and the abstract's 0.92% versus the results' 0.93% further signals that the quantitative claim is not carefully controlled. This internal metric inconsistency is more decisive than, and independent of, the cross-protocol comparability concern about center-cropping.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper applies three semantic segmentation models—a custom U-Net, TransUNet, and Swin-Unet—to the Pereira et al. Landsat-8 wildfire detection dataset, training on algorithmically labeled data and testing on the manually annotated test set. It reports precision, recall, F-score, and IoU for each model and claims that Swin-Unet exceeds the best published CNN baseline by 0.93 percentage points, while the authors' own U-Net achieves the best overall performance. The manuscript includes a code repository and explicitly acknowledges that the Mask2Former experiment failed. The central comparative claim, however, is undermined by an internal inconsistency in the reported IoU values and by the lack of a same-protocol baseline comparison.","tokens_in":8132,"tokens_out":7355,"duration_ms":63375,"significance":"If the reported comparison were valid, the contribution would be a modest but useful empirical study: two ViT-based segmenters applied to a public wildfire benchmark, with the honest and non-obvious finding that a well-tuned CNN remains the best model. The paper ships a code link and builds on a previously published dataset, which are strengths. The conclusion that ViTs are competitive with CNNs for this task is plausible. However, the headline quantitative claim currently rests on an internally inconsistent IoU metric applied to the in-house models and on published baseline numbers obtained under a different protocol. The significance of the paper is therefore contingent on a corrected evaluation.","major_comments":[{"comment":"The IoU column for the three in-house models is not the foreground Jaccard index implied by the same rows' precision and recall values. Under the standard binary Jaccard formula IoU = PR/(P+R−PR), Swin-Unet's P=88.28 and R=92.30 yield approximately 82.2% (reported 89.93%), TransUNet's P=88.46 and R=86.88 yield approximately 78.0% (reported 87.49%), and the authors' U-Net's P=93.37 and R=93.96 yield approximately 88.1% (reported 93.58%). The F-scores satisfy F=2PR/(P+R), so precision, recall, and F-score are mutually consistent; only the IoU column is anomalous for all in-house models, while the Pereira baseline rows satisfy the foreground Jaccard relation. This indicates that the in-house IoU was computed with a different aggregation, likely class-averaged foreground/background IoU, making the headline comparison of Swin-Unet 89.93% against the baseline 89.00% a comparison of two different metrics. Please define the IoU computation precisely, report foreground Jaccard index for all models, and confirm whether the Pereira baseline values are foreground Jaccard.","section":"§4, Table 1"},{"comment":"The claim that Swin-Unet outperforms the Pereira et al. baselines assumes that the published metrics are directly comparable to the authors' evaluation pipeline. Section 3.3 describes training on 224×224 random crops, validation on a 224×224 center crop, AdamW, Dice loss, batch size 64, and 30–50 epochs; the Pereira baselines were not retrained or re-evaluated under this protocol. Differences in preprocessing, crop location, resolution, loss function, optimizer, and epoch count can therefore confound the reported 0.93 percentage-point advantage. To support the comparative claim, the baseline models should be re-evaluated under exactly the same preprocessing and evaluation protocol, or the claims should be restricted to comparisons among the in-house models.","section":"§3.3 and §4"},{"comment":"No repeated runs or error bars are reported. With a claimed advantage of only 0.93 percentage points, the difference is within the range of typical seed-to-seed and data-augmentation variability for segmentation models. Please report mean and standard deviation over at least three random seeds and, ideally, a paired comparison over the same test images, so the reader can judge whether the ViT-versus-CNN difference is statistically meaningful.","section":"§4"}],"minor_comments":[{"comment":"The abstract reports that the ViT outperforms the baseline by 0.92%, while §4 reports 0.93%; these numbers should be reconciled.","section":"Abstract and §4"},{"comment":"The paper says models were trained for '30-50 epochs' but does not give the exact epoch count, final learning rate, or number of training images per model; include these details for reproducibility.","section":"§3.3"},{"comment":"IoU is defined only as 'checks for the overlap between predicted and ground truth masks'; the exact formula (foreground Jaccard versus mean IoU) and any binarization threshold must be stated.","section":"§4"},{"comment":"The statement that the outputs are visually 'almost indistinguishable' is qualitative; if kept, support it with per-image quantitative IoU or error maps.","section":"Figure 5"},{"comment":"The claim that ViTs 'require substantially less computational resources' is not evaluated in this paper and should be qualified or removed to avoid overstating the motivation.","section":"§2"},{"comment":"Reference formatting is inconsistent, including 'UN2 [2022]' and 'Dewagan et al.'; these should be cleaned up.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The internal IoU inconsistency in Table 1 is decisive and should be the first issue the authors address. The comparison of in-house IoU values with the Pereira baseline IoU values appears to compare two different metrics, so the headline claim is currently unsupported. I recommend major revision rather than rejection because the dataset and code are public and the issues are correctable by recomputing all metrics under a single definition and by re-evaluating baselines under the same protocol."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nHere's the quick take: the paper is an honest, straightforward benchmark of two ViT segmentation models on the Pereira Landsat-8 wildfire dataset, but the headline claim—that Swin-Unet beats the published CNN baseline by 0.93% IoU—is not supported by the paper's own table. The reported IoUs for the in-house models are numerically incompatible with the same rows' precision and recall, while the Pereira rows check out. They appear to be mixing two different IoU definitions in one comparison.\n\nWhat's genuinely new: actual training and evaluation of Swin-Unet and TransUNet on this public dataset, with P/R/F1 numbers, open code, and an appendix that candidly reports a Mask2Former failure. The writing is clear and they don't oversell ViTs—they explicitly say their own U-Net is best. That transparency is worth something.\n\nThe soft spots, in order of severity. First and decisive: for Swin-Unet, P=88.28, R=92.30 imply a foreground Jaccard IoU around 82.2%, not 89.93%. Same pattern for TransUNet (78.0% implied vs. 87.49% reported) and their own UNet (88.1% vs. 93.58%). The F-scores match P/R, so only the IoU column is off. Meanwhile, the Pereira baselines' P/R/IoU are perfectly consistent with foreground Jaccard. So the 0.93% advantage is likely an artifact of a class-averaged or otherwise different IoU. The paper never defines the IoU computation. Second, they compare against published baseline numbers instead of retraining under the same protocol; the 224x224 center crop could interact with that. Third, there are no error bars, so a 0.93% difference would need to be assessed for variance anyway. Minor point: the abstract says 0.92%, the body says 0.93%.\n\nWho benefits from this paper? Practitioners who want a very quick sense of how two stock ViT segmenters behave on satellite fire imagery. It's not a methodological advance. The metric inconsistency is fixable, though: define IoU, recompute all rows with the same definition, and ideally retrain the baselines or at least justify comparability. Without that, the central quantitative claim is unreliable.\n\nI'd still send it to review; the topic matters and the experiments are real. A good referee should catch the IoU issue and push for a fix. Worth engaging, not worth citing as-is.\n\nBest","headline":"A useful but flawed benchmark: the headline claim that a ViT beats the CNN baseline evaporates once you notice Table 1 mixes two different IoU definitions.","tokens_in":8678,"tokens_out":3658,"would_cite":false,"duration_ms":32045,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A Swin-Unet vision transformer detects wildfires from Landsat-8 imagery with an IoU of 89.93%, slightly beating the best published CNN baseline, while a custom CNN U-Net remains the top model at 93.58%.","keywords":["wildfire detection","vision transformers","semantic segmentation","Landsat-8","U-Net","remote sensing","Swin-Unet","deep learning"],"falsifier":"Re-run the best published CNN baseline and Swin-Unet on the same test images using one shared evaluation script with multiple random seeds and identical crops; if the IoU gap disappears or reverses across seeds, the claimed outperformance is not robust. Additionally, train Swin-Unet with spectral-imagery pre-training; if it still fails to exceed the custom U-Net's 93.58%, the paper's proposed remedy for the ViT gap is called into question.","tokens_in":7572,"feed_emoji":"🔥","tokens_out":10351,"duration_ms":83903,"temperature":0.7,"pith_summary":"The paper asks whether Vision Transformers—networks built on self-attention rather than convolution—can match or beat convolutional networks for wildfire segmentation in satellite imagery. It reports that one transformer variant, Swin-Unet, reaches an IoU (pixel-overlap score) of 89.93% on a manually annotated Landsat-8 test set, edging the best previously published CNN baseline at 89.00%. The authors' own CNN U-Net scores 93.58% IoU, the highest of any model, so the paper's conclusion is mixed: transformers are competitive and efficient to train, but a carefully tuned CNN is still the best detector on this data. This matters because early wildfire detection from satellites needs models that train quickly and use context beyond local image patches.","feed_headline":"Swin-Unet beats CNN baseline on wildfire detection","feed_subtitle":"Sharpest gain is small: 89.93% vs 89.00% IoU, and a retuned U-Net still leads at 93.58%.","key_machinery":"The comparison hinges on three segmentation models that share a U-Net-style encoder-decoder skeleton. The CNN U-Net uses strided convolutions for downsampling and dilated convolutions (rates 1, 1, 2, 3) to widen its receptive field. TransUNet inserts a transformer encoder between CNN layers, while Swin-Unet replaces the CNN core with shifted-window self-attention, which computes attention within local windows that shift between layers to capture both fine detail and long-range context. This attention mechanism is what gives the ViT models their global view, and the reported results show it transfers to fire detection about as well as convolution does, with Swin's hierarchical windows faring better than TransUNet's plain transformer.","core_discovery":"The central claim is an empirical ranking on one fixed dataset. On the manually annotated test images, Swin-Unet produces an IoU of 89.93%, which is 0.93 percentage points above the best baseline U-Net (89.00%); TransUNet does not surpass the baseline (87.49%). The paper's own U-Net reaches 93.58% IoU and also has the highest precision, leading the authors to state that well-tuned CNNs remain the best technique for wildfire detection even though ViTs are comparably capable. They attribute the ViT shortfall mainly to a mismatch between the everyday RGB images used for pre-training and the multispectral satellite images in the task.","pith_inferences":["Our inference: the 0.93 percentage-point advantage over the baseline is small enough that it could fall within run-to-run training variance, so the superiority of Swin-Unet over CNNs should be treated as provisional until replicated across multiple seeds and training runs.","Our inference: the more durable takeaway is operational, not architectural—if transformers really train faster, equal accuracy would translate into faster model updates as fresh satellite data arrives, which matters more for fire response than a fraction of an IoU point.","Our inference: the same comparison could be run on other multispectral fire datasets (for example, Sentinel-2 imagery) and on the intersection and voting masks from the same source; a consistent small Swin-Unet edge across sensors would argue for a real architectural effect rather than a quirk of one test set.","Our inference: the pre-training-mismatch explanation is directly testable by fine-tuning Swin-Unet from weights learned on aerial or multispectral imagery; if its IoU climbs past 93.58%, the paper's stated path to improvement gains support."],"forward_implications":["If the reported numbers hold, transformer-based segmentation is a credible alternative to CNNs for active wildfire detection, with accuracy close enough to be usable in practice.","The custom U-Net's 93.58% IoU indicates that convolutional architecture details—dilated and strided convolutions—still move performance more than the choice of attention versus convolution on this dataset.","Swin-Unet's margin over TransUNet suggests that hierarchical, shifted-window attention is better suited to satellite fire imagery than a single transformer encoder.","Because the paper describes ViTs as cheaper to train and easier to run in parallel, parity with CNNs would already make them attractive for near-real-time monitoring of large satellite archives.","The paper's own diagnosis points to a concrete next step: pre-training the transformer on spectral or remote-sensing imagery rather than natural RGB photos could close or reverse the gap to the custom U-Net."],"supporting_citations":[{"why":"Supplies the Landsat-8 wildfire dataset, the manually annotated test set, and the CNN baseline metrics that the paper directly compares against.","marker":"de Almeida Pereira et al. [2021]"},{"why":"Introduces the Vision Transformer patch-and-attention approach that the paper applies to wildfire segmentation.","marker":"Dosovitskiy et al. [2020]"},{"why":"Introduces the Swin transformer with shifted-window self-attention, the core mechanism behind the best-performing ViT in the paper.","marker":"Liu et al. [2021]"},{"why":"Adapts Swin into a U-Net-shaped segmentation model (Swin-Unet), the architecture that achieves the paper's best ViT IoU.","marker":"Cao et al. [2021]"},{"why":"Provides TransUNet, the CNN-transformer hybrid used as the second ViT-style comparison model.","marker":"Chen et al. [2021]"},{"why":"Defines the U-Net encoder-decoder-with-skip-connections architecture that both the custom CNN and the transformer hybrids are built on.","marker":"Ronneberger et al. [2015]"},{"why":"One of the three algorithms whose outputs generate the training masks for the dataset.","marker":"Schroeder et al. [2016]"},{"why":"One of the three algorithms whose outputs generate the training masks for the dataset.","marker":"Murphy et al. [2016]"},{"why":"One of the three algorithms whose outputs generate the training masks for the dataset.","marker":"Kumar and Roy [2018]"}],"fun_headline_variants":["Vision Transformers match CNNs on wildfire spotting, but tuned U-Net still wins","ViTs rival CNNs for wildfire detection, yet custom U-Net tops all","Swin-Unet edges baseline CNN, but retuned U-Net remains best for fires","Wildfire detection: ViTs comparable to CNNs, but U-Net still leads","Transformer vs CNN for wildfires: close, but CNN U-Net takes the crown"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the previously published CNN scores were obtained under the same evaluation protocol—same test split, same 224x224 center crop, same preprocessing—as the new models; if the baselines used different crops, resolutions, or splits, the reported improvement may be an artifact of evaluation rather than a real architectural gain.","fun_headline_variants_meta":{"raw":{"variants":["Vision Transformers match CNNs on wildfire spotting, but tuned U-Net still wins","ViTs rival CNNs for wildfire detection, yet custom U-Net tops all","Swin-Unet edges baseline CNN, but retuned U-Net remains best for fires","Wildfire detection: ViTs comparable to CNNs, but U-Net still leads","Transformer vs CNN for wildfires: close, but CNN U-Net takes the crown"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000502,"raw_usage":{"total_tokens":2441,"prompt_tokens":921,"completion_tokens":1520,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":537,"completion_tokens_details":{"reasoning_tokens":1410}},"tokens_in":537,"tokens_out":1520,"duration_ms":9739,"temperature":1.0,"reasoning_tokens":1410,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T11:59:36.462583+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the best published CNN baseline and Swin-Unet on the same test images using one shared evaluation script with multiple random seeds and identical crops; if the IoU gap disappears or reverses across seeds, the claimed outperformance is not robust. Additionally, train Swin-Unet with spectral-imagery pre-training; if it still fails to exceed the custom U-Net's 93.58%, the paper's proposed remedy for the ViT gap is called into question.","supporting_citations":[{"cited_title":"Active fire detection using landsat-8/oli data","cited_arxiv_id":null,"evidence_quote":"One of the three algorithms whose outputs generate the training masks for the dataset."},{"cited_title":"Hotmap: Global hot target detection at moderate spatial resolution","cited_arxiv_id":null,"evidence_quote":"One of the three algorithms whose outputs generate the training masks for the dataset."},{"cited_title":"Global operational land imager landsat-8 reflectance-based active fire detection algorithm","cited_arxiv_id":null,"evidence_quote":"One of the three algorithms whose outputs generate the training masks for the dataset."}],"review_version":1}