{"id":"4694d46f-7fb5-454a-8baf-c89395e213fc","arxiv_id":"2412.18147","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"A ResNet50 and YOLOv11 pipeline for tornado damage assessment reports 90.28% classification and 60.83% detection accuracy, but class imbalance and missing evaluation details weaken the result.","lead":"This paper trains two standard deep learning models, ResNet50 and YOLOv11, to classify and detect building damage in tornado photos. It reports 90.28% accuracy for damage classification and 60.83% for detection, but provides no code, data, or per-class metrics.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Reported 90.28% accuracy may reflect train/test leakage from same buildings in the Moore dataset; a building-disjoint split is needed to verify generalization.","rationale":"The reader's weakest assumption is that the train/validation/test split is independent and free of temporally correlated duplicates, citing the 360-degree vehicle video. That mechanism applies to the YOLOv11 detection dataset (2021 Mayfield), not to the ResNet50 classification dataset (2013 Moore), which is the source of the 90.28% headline accuracy. Nonetheless, the reader's general concern about leakage is the most load-bearing issue: the Moore handheld dataset likely contains multiple images of the same building, and a random split can place the same building in both training and test sets, inflating the reported accuracy. This is a concrete, testable threat to the central claim. I also note that the paper's own admission of class imbalance makes overall accuracy alone an insufficient metric. My proposed check directly tests whether the headline number survives a building-disjoint split with per-class metrics. The paper's contribution is a straightforward application of established models to a new domain; with appropriate verification it could be a useful application, so the reader's CONDITIONAL verdict remains appropriate. I therefore recommend no change to the verdict, but the condition should explicitly include the building-aware split and per-class reporting described above.","tokens_in":5207,"tokens_out":3279,"duration_ms":31339,"concrete_test":"Perform a building-aware split of the 2013 Moore dataset: group all images of the same building (via geotags, visual near-duplicate clustering, or manual review) into a single partition, retrain ResNet50 with identical hyperparameters, and report per-class precision/recall, macro-F1, and overall accuracy on the disjoint test set. If accuracy drops by more than a few points or minority-class recall is near zero, the original 90.28% figure is inflated by leakage or class imbalance; if the metrics hold, the claim is supported. Also release the split indices and code to make the check reproducible.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that a ResNet50 classifier reaches 90.28% accuracy on multiclass tornado-damage classification (Table 1, Section IV). This claim assumes the test set contains genuinely unseen buildings or damage instances. The paper describes only an 80:10:10 random split (Table 1) and never describes grouping images by building or removing near-duplicates. The 2013 Moore dataset was collected with handheld cameras (Section II.A), which typically yields multiple photos of the same building from different angles. If the same building appears in both training and test partitions, the model can memorize building-specific textures, inflating accuracy. In addition, the paper reports only overall accuracy and acknowledges class imbalance (Section V); if a few classes dominate, high accuracy can coexist with poor minority-class recall. The paper provides no per-class precision/recall, macro-F1, or confusion-matrix numbers in text, despite claiming these are shown in Figure 4. The leakage concern is more plausible for the ResNet50 result than the reader's suggested 360-degree video mechanism, because the 90.28% accuracy is on the 2013 Moore handheld dataset, not the Mayfield 360-degree video dataset (which is used for YOLOv11 detection). However, the underlying issue is the same: without a building-aware split, the headline number is not evidence of generalizable damage assessment.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a dual-model deep learning pipeline for post-tornado building damage assessment. A ResNet50 classifier with transfer learning is trained on 2,635 images from the 2013 Moore tornado to categorize damage into five classes, reporting 90.28% accuracy and a 1529 ms per-image inference time; a YOLOv11 detector is trained on 1,776 frames extracted from 360° vehicle video of the 2021 Mayfield tornado, reporting 60.83% accuracy and 3 ms inference time. The authors argue that this automated approach can accelerate manual damage assessment and improve disaster response.","tokens_in":5423,"tokens_out":5851,"duration_ms":52600,"significance":"If the reported results are valid, the paper offers a practical application of standard deep learning to tornado damage assessment, a comparatively underexplored disaster type in the computer-vision literature. The use of two complementary tasks—object detection of buildings and fine-grained damage classification—on real post-disaster data is a strength, as is the reporting of inference times on a modern GPU. However, the evidence is weakened by evaluation-methodology gaps: no building-disjoint or sequence-aware split, no per-class metrics for a heavily imbalanced classification problem, an unexplained and implausible inference-time figure for ResNet50, and an undefined 'accuracy' metric for the detector. These gaps are load-bearing because the headline claims (near-expert accuracy and near-real-time speed) depend on them. The central idea is plausible, but the paper needs stronger validation before the claims can be accepted.","major_comments":[{"comment":"The 90.28% accuracy claim for the ResNet50 classifier is not supported as a generalization estimate because the train/validation/test split is performed at the image level, not grouped by building or de-duplicated. The Moore dataset (Section II.A) was collected with handheld cameras, which typically produce multiple images of the same building from different angles. If the same building appears in both training and test partitions, the model can memorize building-specific textures and inflate accuracy. The manuscript describes no building-level or sequence-aware splitting and no analysis of near-duplicate frames. Please evaluate with a building-disjoint split (e.g., grouping images by building or by geographic proximity) and report the resulting accuracy, or provide evidence that no building appears in more than one partition.","section":"§IV, Table 1"},{"comment":"For the heavily imbalanced Moore classification task, overall accuracy alone is insufficient. The bottom panel of Figure 2 indicates that the 'undamaged building' class dominates, and the paper acknowledges class imbalance in Section V. A model can achieve high overall accuracy by predicting the majority class for most inputs. The manuscript claims per-class metrics are shown in Figure 4 but does not report any numeric per-class precision, recall, or macro-F1 in the text. Please include a confusion matrix with class-wise precision/recall and a macro-averaged F1 score to substantiate the claim of near-expert performance.","section":"§IV, Table 1, Fig. 2"},{"comment":"The reported inference time of 1529 ms per image for ResNet50 on an NVIDIA GeForce RTX 4090 is conspicuously high; a standard ResNet50 forward pass on this GPU is expected to take only a few milliseconds. If this number includes image loading, preprocessing, and Python overhead, that must be stated explicitly and separated from the model's forward-pass time. As written, the 'near real time analysis' claim in Section V is not quantitatively supported. Please clarify the measurement protocol or correct the value.","section":"Abstract, Table 1"},{"comment":"The 'accuracy' metric for the YOLOv11 detection model is ambiguous. For an object detector, accuracy is not a standard metric; the paper reports mAP@0.5 ≈ 0.45 and mAP@0.5:0.95 ≈ 0.27, then states an overall accuracy of 60.83% without defining what counts as a correct prediction (per-image, per-bounding-box, or at what IoU threshold). Please define the accuracy computation and reconcile it with the mAP values, or replace it with standard detection metrics such as mAP and precision/recall.","section":"§IV, Table 1"}],"minor_comments":[{"comment":"There are several typos: 'a ny newly created pixels' should be 'any newly created pixels', 'Data Proprocessing' should be 'Data Preprocessing', and 'Training Paramaters' should be 'Training Parameters'.","section":"§II.B.1, Table 1"},{"comment":"The cross-entropy equation has rendering artifacts in the submitted PDF (garbled subscripts and formatting). Please ensure the equation is typeset correctly.","section":"§III.B"},{"comment":"The text states that all evaluation metrics are summarized in Figures 3 and 4, but it does not refer to specific panels when discussing each metric. Please reference the relevant panels in the body and ensure the figures are legible in the final version.","section":"§IV, Figures 3 and 4"},{"comment":"No data or code availability statement is provided. To support reproducibility, the authors should state whether the datasets or the trained models will be released, or at least describe the data-collection protocol in sufficient detail to allow re-collection.","section":"General"},{"comment":"The YOLOv11 model is attributed to Redmon et al. (the original YOLO paper), but YOLOv11 is a different implementation with its own publication. Please cite the correct YOLOv11 source or, at minimum, a repository or technical report that defines the architecture used.","section":"§I, References"}],"recommendation":"major_revision","confidential_remarks":"The paper is a straightforward application of standard deep learning models to a disaster-assessment problem. It addresses an important use case but is methodologically incremental relative to existing damage-assessment literature. The primary barrier is the evaluation methodology; if the authors provide a building-disjoint split, per-class metrics, and a clarified inference-time measurement, the paper could be acceptable for an applied venue. Additionally, the reference to YOLOv11 should be corrected to the appropriate source."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"First thing to know: this is an applied-transfer paper, not a methods paper. The new content is two tornado damage datasets (Moore 2013, Mayfield 2021) and a pair of off-the-shelf models run on them. The architecture, training, and evaluation are all standard. That is fine as a contribution, but the headline accuracy needs to be read with caution.\n\nThe paper does several things right. It describes data curation and preprocessing in enough detail to reproduce the pipeline. The two tasks—building detection and damage classification—are genuinely different and each uses a sensible model. The authors are open about class imbalance and future work. The citation pattern is normal; they cite the prior data-collection papers and the YOLO/ResNet sources.\n\nThe soft spots are in the evaluation. The 90.28% accuracy comes from a random 80:10:10 split of images. The Moore data was gathered with handheld cameras, so the same building almost certainly appears in multiple photos. If those photos land in both train and test, the model can memorize building appearance and overstate accuracy. The paper never mentions grouping by building, sequence, or near-duplicate removal. That is the main threat to the central claim. Also missing: per-class precision/recall (the text says they're in Figure 4 but doesn't give numbers), error bars or repeated runs, and any baseline comparison (even a simple ResNet without augmentation would help). The 1529ms per-image inference time is oddly high for ResNet50 on an RTX 4090—typical is tens of milliseconds—and is not explained. The YOLOv11 result (60.83% accuracy, mAP@0.5≈0.45) is modest; calling it 'rapid detection' is fair, but it does not support near-expert triage.\n\nThese are not fatal flaws. The paper is a legitimate application study and the domain gap for tornado damage is real. The right fix is a building-aware split and more careful reporting. That is a major revision, not a desk reject. I would send it to a knowledgeable reviewer and ask for those changes, plus code/data release so others can build on the datasets.","headline":"Applied tornado-damage classification with standard models; the numbers are plausible but the evaluation needs a building-aware split before they can be trusted.","tokens_in":5980,"tokens_out":2690,"would_cite":false,"duration_ms":24790,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A dual deep-learning pipeline automates post-tornado damage triage, with ResNet50 at 90.28% accuracy and YOLOv11 detecting buildings in 3 ms per frame.","keywords":["damage assessment","computer vision","resilience","post-tornado disaster","deep learning","YOLOv11","ResNet50","transfer learning"],"falsifier":"Re-run both evaluations on a test set constructed by grouping all frames from the same building or the same 360° video sequence into one side of the split, so no near-duplicate frames cross the train/test boundary; if classification accuracy falls materially below 90.28% or detection accuracy below 60.83%, the reported numbers are inflated by temporal correlation.","tokens_in":4966,"feed_emoji":"🌪️","tokens_out":8793,"duration_ms":67483,"temperature":0.7,"pith_summary":"This paper claims that a two-model computer vision system can automate post-tornado building damage assessment, replacing weeks of manual inspection with near-real-time analysis. On image sets from the 2013 Moore, Oklahoma and 2021 Mayfield, Kentucky tornadoes, the authors report that a ResNet50 classifier reaches 90.28% accuracy in assigning one of five damage states, while a YOLOv11 detector locates buildings at 3 ms per frame with 60.83% accuracy. If correct, this would give disaster responders a fast, consistent, and scalable triage tool for a disaster type that prior AI work has largely skipped. The figures are claimed for the specific datasets and splits described in the paper, not for all tornado imagery.","feed_headline":"Two-model AI system rates tornado damage with 90.28% accuracy","feed_subtitle":"ResNet50 sorts five damage states while YOLOv11 finds buildings in 3 ms, shrinking week-long surveys toward real time.","key_machinery":"The machinery is a two-part architecture. For damage classification, a ResNet50 backbone pretrained on ImageNet is stripped of its top layers and fitted with a global average pooling 2D layer, a dense layer of 1024 units with ReLU activation, and a softmax output; categorical cross-entropy loss drives the training. For building detection, YOLOv11 runs the entire image through a single forward pass to output bounding boxes, class labels, and confidence scores. Supporting this are a data pipeline that auto-orients and resizes frames, augmentation with rotations, shifts, shear, zoom, and flips, and two labeled datasets: hand-held camera images of the 2013 Moore tornado and 360° vehicle-video frames of the 2021 Mayfield tornado. The ResNet50 transfer-learning setup carries the classification accuracy, while the YOLOv11 single-pass design carries the real-time detection speed.","core_discovery":"The authors claim that a dual-model deep learning system—ResNet50 for fine-grained damage classification and YOLOv11 for building detection and localization—can perform post-tornado damage assessment at a level suitable for practical response workflows. On 2,635 images from the 2013 Moore tornado, the ResNet50 model distinguishes undamaged buildings, roof damage, wall collapse, wall-and-roof damage, and non-buildings with 90.28% accuracy at 1,529 ms per image. On 1,776 frames extracted from 360° vehicle video of the 2021 Mayfield tornado, YOLOv11 detects buildings with 60.83% accuracy, a mean average precision of 0.45 at 50% IoU, and a 3.0 ms inference time. The authors present this as a scalable, objective alternative to manual post-disaster assessment, and as the first tornado-specific application in a literature dominated by earthquake, flood, and hurricane damage.","pith_inferences":["If the reported accuracy holds on a split that is de-duplicated at the building or video-sequence level, the approach would likely transfer to other windstorm events; a direct test would be training on one tornado and evaluating on a later, unseen tornado.","The large gap between the 90.28% classification accuracy and the 60.83% detection accuracy suggests that building localization, not damage-state classification, is the current bottleneck, so improving the detector may yield larger system gains than further classifier tuning.","The 360° video source almost certainly produces temporally correlated frames, so a sequence-aware train/test split could materially change the reported numbers; reporting such a split would be the most direct stress test of the 90.28% figure."],"forward_implications":["Post-tornado damage assessment could drop from weeks of manual inspection to near-real-time analysis of drive-by video and still images.","The same dual-model pipeline could be retrained on damage datasets from other hazard types, since the underlying architectures are generic.","YOLOv11's 3 ms per frame makes it feasible to run building detection live during vehicle-mounted surveys, flagging damaged structures as they are filmed.","Automated damage-state labeling would give authorities a consistent, repeatable damage assessment that is not subject to human evaluator bias."],"supporting_citations":[{"why":"It supplies the 360° vehicle-video data collection and processing method used for the 2021 Mayfield tornado detection dataset.","marker":"[2]"},{"why":"It supplies the 2013 Moore, Oklahoma tornado images used for damage classification.","marker":"[10]"},{"why":"It defines the YOLO single-pass object detection approach that YOLOv11 builds on.","marker":"[7]"},{"why":"It provides the deep residual network architecture and ImageNet pretraining used as the ResNet50 base model.","marker":"[8]"},{"why":"It contributes the rapid data dissemination and deep learning viewer pipeline used to extract and package video frames.","marker":"[9]"},{"why":"It provides the INCORE damage state taxonomy (undamaged, slight, moderate, extensive, complete) used to label the Mayfield detection data.","marker":"[1]"},{"why":"It documents the gap in tornado-specific AI damage assessment that the paper positions itself against.","marker":"[5]"}],"fun_headline_variants":["90.28% tornado damage rating: ResNet50 + YOLOv11 combo","Dual AI model rates tornado damage at 90.28% accuracy","3ms building detection, 90% damage rating: tornado AI","AI assesses tornado damage in real time: 90.28% accurate","YOLOv11 and ResNet50 team up for fast tornado assessment"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The reported accuracies depend on the test images being genuinely new to the models, meaning no near-duplicate frames from the 360° video and no second images of the same building appear in both the training and test sets.","fun_headline_variants_meta":{"raw":{"variants":["90.28% tornado damage rating: ResNet50 + YOLOv11 combo","Dual AI model rates tornado damage at 90.28% accuracy","3ms building detection, 90% damage rating: tornado AI","AI assesses tornado damage in real time: 90.28% accurate","YOLOv11 and ResNet50 team up for fast tornado assessment"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000629,"raw_usage":{"total_tokens":2878,"prompt_tokens":887,"completion_tokens":1991,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":503,"completion_tokens_details":{"reasoning_tokens":1893}},"tokens_in":503,"tokens_out":1991,"duration_ms":13615,"temperature":1.0,"reasoning_tokens":1893,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T04:59:10.656884+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run both evaluations on a test set constructed by grouping all frames from the same building or the same 360° video sequence into one side of the split, so no near-duplicate frames cross the train/test boundary; if classification accuracy falls materially below 90.28% or detection accuracy below 60.83%, the reported numbers are inflated by temporal correlation.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It supplies the 360° vehicle-video data collection and processing method used for the 2021 Mayfield tornado detection dataset."},{"cited_title":"Turning Disaster into Knowledge in Geotechnical Earthquake Engineering,","cited_arxiv_id":null,"evidence_quote":"It defines the YOLO single-pass object detection approach that YOLOv11 builds on."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"It provides the INCORE damage state taxonomy (undamaged, slight, moderate, extensive, complete) used to label the Mayfield detection data."},{"cited_title":"Lisa” Wang et al., “Application of Multidisciplinary Community Resilience Modeling to Reduce Disaster Risk: Building Back Better,","cited_arxiv_id":null,"evidence_quote":"It documents the gap in tornado-specific AI damage assessment that the paper positions itself against."}],"review_version":1}