{"id":"bad39fde-6401-47ce-91ca-3c6497a77721","arxiv_id":"2501.12087","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A quantized Swin Transformer runs real-time disaster classification on drone hardware, with the new DisasterEye dataset introduced as a benchmark.","lead":"This paper shows that a Swin Transformer image classifier, shrunk with TensorRT quantization, can classify disasters from drone images in real time on small hardware including the Jetson Nano. The authors also release a new web-sourced dataset, DisasterEye, for testing such models.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"DisasterEye's image-level split of scraped YouTube/Google frames risks train/test leakage; without a source-grouped or deduplicated split, the reported 'almost similar' quantization accuracy is not evidence for real-world generalization.","rationale":"The reader's weakest assumption already identified this as the main risk. After reading the manuscript, it is the most load-bearing because the accuracy numbers are the only quantitative evidence that quantization preserves accuracy; the speed and memory gains are not in dispute. The manuscript itself provides the supporting fact: DFAN is described as sourced from videos with duplicates, and DisasterEye follows the same collection-and-split recipe. A uniform image-level random split of video frames is a textbook leakage route. The concern is not that the authors are deceptive; it is that the default split described cannot rule out memorization of near-duplicates. A secondary inconsistency (the text claims '3x to 5x faster' while Table I shows 16-26x speedups) is harmless because it understates rather than overstates the result and does not affect the accuracy argument. The proposed test is feasible: the dataset and code are publicly linked, and perceptual hashing is cheap. If the re-evaluation shows that quantization loss remains small on deduplicated/source-grouped splits, the paper's central claim stands and the condition is satisfied. If the loss grows sharply, the deployment claim would need to be revised. This matches the reader's CONDITIONAL verdict, so no change is proposed.","tokens_in":10261,"tokens_out":7980,"duration_ms":83976,"concrete_test":"Use the linked repository to obtain DisasterEye, compute pHash (or SSIM) between every test image and every training image, and count test images with a near-duplicate in training (e.g., pHash distance <= 4 or SSIM > 0.85). Then re-evaluate the original Swin-T and the TensorRT INT8/FP16 engines on (a) the deduplicated test subset and (b) a source-grouped split, where frames from the same YouTube video or web source are kept in one partition. Report the accuracy drop on the clean subsets; if the INT8 drop exceeds roughly 3 percentage points while the reported <1% drop does not reproduce, the accuracy-preservation claim fails for novel disaster scenes. For DFAN, repeat using the original video-level grouping from [38] if available.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim (quantized Swin Transformer preserves accuracy while meeting real-time constraints) is only as strong as the test sets used to measure accuracy. Section IV-A states DisasterEye is built from Google Images and YouTube and then 'uniformly divided' 70/20/10 at the image level, 'similarly to the DFAN dataset' — the same paragraph notes DFAN was sourced from videos and contains duplicate images. YouTube sources yield correlated video frames; near-duplicate web images can also appear in both training and test. This violates the requirement that test frames represent unseen disaster scenes. It can inflate absolute accuracy, and it can also mask quantization-induced degradation: a quantized model may retain accuracy on memorized near-duplicates while losing more on genuinely novel scenes. The reported 0.7% INT8 accuracy drop on DisasterEye (Table I) is therefore not credible evidence for deployment until splits are deduplicated or grouped by source. DFAN is explicitly acknowledged to contain duplicates, so the same caveat applies to the DFAN row and to the Table II comparison. No error bars or confidence intervals are provided, so the 'almost similar' claim rests entirely on these possibly-leaky splits.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes a UAV-assisted edge framework for real-time disaster image classification. The authors fine-tune a Swin Transformer on three datasets (AIDER, DFAN, and their own DisasterEye), then apply post-training quantization using TensorRT (FP16 and INT8) and compare it with several other quantization methods (MinMax, EMA, OMSE, Percentile, FQ-ViT). They report that TensorRT quantization yields large latency reductions (up to hundreds of FPS on an NVIDIA T4) and smaller model sizes, with only small accuracy drops, and that the INT8 model achieves around 45 FPS on a Jetson Nano. They also introduce a new dataset, DisasterEye, containing 2,751 images in eight classes, and position it as a benchmark for real-world disaster classification.","tokens_in":10489,"tokens_out":3761,"duration_ms":38779,"significance":"If the reported results hold, the paper offers a practical demonstration that transformer-based models can be quantized for real-time onboard UAV inference, and the DisasterEye dataset could be a useful resource for the community. The paper deserves credit for testing on two external benchmarks (AIDER, DFAN) in addition to the new dataset, for comparing several quantization methods, and for releasing the code and dataset. However, the significance is conditional on resolving the data-split leakage risk in DisasterEye and DFAN and on a more rigorous latency-measurement protocol; the current evaluation makes the magnitude and even the direction of some performance comparisons unreliable.","major_comments":[{"comment":"The DisasterEye dataset is built from Google Images and YouTube and then 'uniformly divided' at the image level into 70/20/10 train/validation/test sets. Since YouTube sources yield correlated frames and near-duplicate web images can appear in both training and testing, this split can leak source content into the test set. The paper itself notes in the same section that DFAN contains duplicate images from video sources, and applies the same splitting strategy to DFAN. This leakage undermines the reported absolute accuracy numbers and, more importantly, the claim of 'almost similar' accuracy after quantization: a quantized model can retain accuracy on memorized near-duplicates while degrading on genuinely novel scenes. The authors should provide a source-grouped or deduplicated split (e.g., grouping frames by video source) and report accuracy, precision, recall, and F1 on such a split for DisasterEye and DFAN.","section":"Section IV-A, Figure 3"},{"comment":"The latency and throughput measurements are not adequately specified. The paper does not state the batch size, input resolution after preprocessing, number of repeated runs, warm-up procedure, or whether the reported latencies include I/O and preprocessing. No error bars, standard deviations, or confidence intervals are given, so the numbers in Table I (e.g., 2.97 ms vs. 48.87 ms on AIDER) are presented as exact point estimates from single or unreported runs. The authors should describe the measurement protocol in detail and report mean and standard deviation over multiple trials for both the desktop/T4 and Jetson Nano evaluations.","section":"Section IV-D, Table I"},{"comment":"The statement that the proposed model 'surpasses all lighter models that contain fewer parameters and smaller model sizes' in terms of FPS is not supported by the table: MobileNet Compressed on Jetson Nano achieves 71 FPS, while the proposed INT8 model on Jetson Nano achieves 45.22 FPS. In addition, the FPS numbers listed in Table II are obtained on different hardware (T4 GPU, Jetson Nano, RTX 3090, etc.), so cross-row comparisons are not valid. The comparison should be restricted to matched hardware or clearly accompanied by a caveat that raw FPS values on different devices are not directly comparable.","section":"Section IV-G, Table II"},{"comment":"The claimed speedup of '3x to 5x faster than the original' does not match the numbers in Table I. On AIDER, for example, the original latency is 48.87 ms and TensorRT FP16 is 2.97 ms, which is roughly a 16x speedup; similar ratios appear for the other datasets. Either the original latency was measured on different hardware (e.g., RTX 2080 vs. T4), which should be stated explicitly, or the '3x to 5x' claim is inaccurate. Please reconcile the stated speedup with the reported experimental numbers.","section":"Section I and Table I"}],"minor_comments":[{"comment":"There are numerous typographical issues, including 'UA V' and 'UA Vs' spacing in the abstract and introduction, 'TensorTR' instead of 'TensorRT' in Section IV-D, and 'prepossessing' instead of 'preprocessing' in Section IV-C. A careful proofread is needed.","section":"Throughout"},{"comment":"The FQ-ViT row for AIDER reports an accuracy of 0.9839, which is higher than the original model's 0.9825. This is an interesting result that is not discussed; please add a brief explanation of why this particular quantization method can improve accuracy over the full-precision model.","section":"Table I"},{"comment":"The paper says 'All models are fine-tuned until convergence' but does not report the number of training epochs, batch size, or data augmentation details beyond resizing, cropping, and normalization. Including these training hyperparameters would improve reproducibility.","section":"Section IV-C"},{"comment":"The description of TensorRT states that it uses 'five different optimizations,' including precision calibration, layer fusion, kernel auto-tuning, multi-stream executions, and dynamic tensor memory. The paper does not discuss which of these optimizations is responsible for the observed latency reductions, so it is unclear whether the gains come primarily from reduced precision or from the other TensorRT optimizations. A sentence clarifying this would be helpful.","section":"Section III-C"},{"comment":"Reference [27] (Swin Transformer) is given as an arXiv preprint, but it has been published at ICCV 2021; the published version should be cited for completeness.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The paper has a potentially useful empirical result and a new dataset, but the validity of the central 'almost similar accuracy' claim depends on the data split issue, and the speed comparisons need to be made on a controlled basis. The authors should be asked to rerun or at least re-report the DisasterEye and DFAN evaluations with a leakage-free split, and to provide a rigorous latency methodology. The FPS comparison contradiction with MobileNet Compressed should be fixed. These are fixable within revision, so major revision rather than rejection seems appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe short version: this is a straightforward engineering paper—fine-tune a Swin Transformer on three disaster image datasets, quantize it with TensorRT (FP16 and INT8), and measure accuracy, latency, FPS, and model size. The one genuinely new artifact is DisasterEye, a 2,751-image, 8-class dataset scraped from Google Images and YouTube. The main result, that TensorRT quantization gives large speedups with modest accuracy loss, is consistent with how TensorRT behaves and is probably reproducible.\n\nWhat the paper does well: it benchmarks on three datasets and compares several post-training quantization methods (MinMax, EMA, OMSE, Percentile, FQ-ViT, TensorRT) in a single table. It validates on two external benchmarks (AIDER, DFAN) in addition to its own dataset. The code and dataset are promised in a public repo. That is a reasonable empirical contribution to the UAV disaster-detection subfield.\n\nThe soft spots are real and some are sized like load-bearing walls. First, the headline speedup claim is contradicted by the paper's own Table I. The abstract and conclusion say \"3x to 5x faster than the original,\" but Table I shows 11x to 26x latency reduction depending on dataset and precision. That is an overclaim in the wrong direction—the numbers are better than the claim, which suggests sloppy writing or a changed baseline, but it still needs fixing. Second, the comparison to prior work is unfair. In Table II, MobileNet Compressed on Jetson Nano runs 71 FPS while their INT8 model on the same device runs about 45 FPS, yet the text says their model \"surpasses all lighter models\" in FPS. It does not. And FPS is compared across different GPUs (T4 vs RTX 2070/3090) without any standardization, so those rows are not apples-to-apples.\n\nThird, the data-leakage concern is legitimate. DisasterEye is scraped from Google Images and YouTube and split uniformly at the image level. The paper even notes that DFAN, built from videos, contains duplicate images, and DisasterEye's split is described as \"similarly\" done. Frames from the same YouTube video or near-duplicate web images can appear in both training and test, inflating accuracy and masking quantization-induced degradation. Neither dataset is deduplicated or split by source, and there are no error bars or confidence intervals anywhere. That weakens the \"almost similar accuracy\" claim.\n\nWho this is for: people working on edge deployment of vision transformers or practical UAV disaster response. It is not a methodological advance, but the dataset and the benchmark numbers could be useful once the leakage problem is addressed.\n\nRecommendation: send it to peer review, but with major revisions. The empirical backbone is plausible; the claims are not. Fix the speedup numbers, add a proper latency methodology, rerun or qualify the comparisons, and re-split DisasterEye by source or demonstrate that duplicates do not cross the train/test boundary. With those changes it could be a solid benchmark paper.","headline":"A useful, plausible deployment benchmark for quantized Swin on UAV images, but the paper oversells its speedups and its new dataset has a real train/test leakage risk.","tokens_in":10995,"tokens_out":2633,"would_cite":false,"duration_ms":27600,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-tuned Swin Transformer quantized with TensorRT post-training quantization detects disasters in aerial images at 3–5x the original speed, with near-original accuracy, on both server GPUs and an NVIDIA Jetson Nano.","keywords":["UAV disaster detection","post-training quantization","Swin Transformer","TensorRT","edge inference","DisasterEye dataset","real-time classification","Jetson Nano"],"falsifier":"Re-split DisasterEye by source video (keeping frames from the same clip in one split) and remove near-duplicate web images using perceptual hashing; if accuracy on that cleaned test set falls by more than a few points from the reported values, the generalization claim is partly an artifact of data leakage.","tokens_in":10091,"feed_emoji":"🚁","tokens_out":8468,"duration_ms":75384,"temperature":0.7,"pith_summary":"The paper argues that a full transformer model, not just a small CNN, can be used for real-time disaster detection aboard a drone if it is quantized after training. The authors fine-tune a Swin Transformer on aerial images of fires, floods, earthquakes, landslides, and other disasters, then apply TensorRT FP16 and INT8 post-training quantization. They report 3–5x faster inference, a model-size reduction of roughly 50% (FP16) and 70% (INT8), and accuracy close to the original FP32 model, including real-time speeds on an NVIDIA Jetson Nano. To support real-world use, they release a new eight-class dataset, DisasterEye. If the claims hold, drones could classify disaster scenes onboard without cloud latency, preserving privacy and connectivity independence where links are unreliable.","feed_headline":"Quantized transformer spots disasters 3-5x faster on drones","feed_subtitle":"FP16 and INT8 quantization keeps near-original accuracy while cutting model size up to 70% for onboard UAV use.","key_machinery":"The central mechanism is post-training quantization with TensorRT, which converts the fine-tuned Swin Transformer's weights, activations, and attention maps from 32-bit floating point to 16-bit or 8-bit precision, then applies layer fusion, kernel auto-tuning, multi-stream execution, and dynamic tensor memory. The Swin Transformer's shifted-window hierarchical attention is the model being compressed, and the quantization must handle non-linear layers like Softmax, GELU, and LayerNorm that earlier CNN-focused methods leave in floating point. TensorRT's precision calibration is what preserves accuracy while cutting latency and memory, and the DisasterEye dataset is what tests the approach on a wider mix of real-world disaster scenes.","core_discovery":"Post-training quantization with TensorRT, applied to a fine-tuned Swin Transformer, delivers real-time disaster classification on UAV-class hardware while preserving accuracy close to the original FP32 model. FP16 quantization achieves 336.92 frames per second on an NVIDIA T4 GPU with an F1 score of 0.980 on AIDER and a model size of 58.89 MB (down from 107 MB); INT8 achieves 236.41 FPS, F1 0.977, and 36.76 MB. On a Jetson Nano, the INT8 model reaches 45.22 FPS. The paper argues that this makes a full transformer practical for onboard aerial disaster detection through post-training quantization.","pith_inferences":["Because DisasterEye is split at the image level without deduplication or grouping by source video, the reported accuracy on that dataset likely overstates generalization to unseen disaster footage.","The T4 GPU frame rates are not representative of drone hardware; the Jetson Nano numbers are closer to deployment, but the paper does not report power consumption or thermal throttling, which are the next limiting factors.","The same quantization recipe could be extended from scene classification to object detection or segmentation, which would locate victims or damage rather than only labeling the scene type.","A fairer comparison with the MobileNet baselines would run all models on the same GPU and input resolution; the current tables mix hardware and frameworks."],"forward_implications":["A drone can classify disaster scenes onboard at over 200 FPS on a server GPU and over 45 FPS on a Jetson Nano, making round-trip cloud analysis unnecessary for detection.","INT8 quantization reduces the model from 107 MB to roughly 34 MB, freeing memory for other onboard tasks on small UAV platforms.","The same TensorRT quantization pipeline is not tied to specific disaster classes, so it can be applied to other transformer backbones for edge classification.","On datasets with enough samples per class, accuracy stays within about one to two F1 points of the original model, so the speedup does not force a large accuracy tradeoff.","The DisasterEye dataset provides an eight-class aerial-disaster benchmark that includes conflict and post-earthquake scenes, extending beyond the fire-only focus of earlier benchmarks."],"supporting_citations":[{"why":"Provides the UAV-assisted edge framework and MobileNet baselines on Jetson Nano that the paper compares against for real-time disaster management.","marker":"[8]"},{"why":"Supplies EmergencyNet, an efficient aerial image classifier baseline that motivates the need for lightweight but sufficiently complex architectures.","marker":"[9]"},{"why":"Supplies the AIDER dataset of five disaster and normal classes used to evaluate the proposed model.","marker":"[10]"},{"why":"Introduces the Swin Transformer backbone that the paper fine-tunes and then quantizes.","marker":"[27]"},{"why":"FQ-ViT is the transformer-specific post-training quantization method used as a key baseline in the ablation study.","marker":"[28]"},{"why":"Provides the DFAN fire dataset and the DFAN baseline model that the paper compares against in Table II.","marker":"[38]"},{"why":"Establishes the integer-arithmetic-only quantization method underlying the MinMax and EMA-style baselines the paper tests.","marker":"[39]"},{"why":"TensorRT is the NVIDIA SDK whose FP16 and INT8 precision calibration and optimization passes produce the reported latency and memory reductions.","marker":"[42]"}],"fun_headline_variants":["Quantized transformer enables real-time drone disaster spotting","Drones get faster disaster detection with quantized transformer","Onboard transformer quantization speeds disaster detection","Swin transformer optimized for UAV real-time disaster classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the DisasterEye test set measures how well the model generalizes to new disaster scenes; because the images come from Google Images and YouTube and are split at the image level without removing near-duplicates or grouping frames from the same video, some of the reported accuracy may come from the model having already seen the same scene during training.","fun_headline_variants_meta":{"raw":{"variants":["Quantized transformer enables real-time drone disaster spotting","Drones get faster disaster detection with quantized transformer","Onboard transformer quantization speeds disaster detection","Swin transformer optimized for UAV real-time disaster classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000193,"raw_usage":{"total_tokens":1322,"prompt_tokens":890,"completion_tokens":432,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":506,"completion_tokens_details":{"reasoning_tokens":373}},"tokens_in":506,"tokens_out":432,"duration_ms":5717,"temperature":1.0,"reasoning_tokens":373,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T17:30:29.964355+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-split DisasterEye by source video (keeping frames from the same clip in one split) and remove near-duplicate web images using perceptual hashing; if accuracy on that cleaned test set falls by more than a few points from the reported values, the generalization claim is partly an artifact of data leakage.","supporting_citations":[{"cited_title":"and Theocharides, T","cited_arxiv_id":null,"evidence_quote":"Supplies EmergencyNet, an efficient aerial image classifier baseline that motivates the need for lightweight but sufficiently complex architectures."},{"cited_title":"(2020) ’AIDER (Aerial Image Dataset for Emergency Response Applications),’ Zenodo","cited_arxiv_id":null,"evidence_quote":"Supplies the AIDER dataset of five disaster and normal classes used to evaluate the proposed model."},{"cited_title":"A., Gupta, S","cited_arxiv_id":null,"evidence_quote":"Provides the DFAN fire dataset and the DFAN baseline model that the paper compares against in Table II."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Establishes the integer-arithmetic-only quantization method underlying the MinMax and EMA-style baselines the paper tests."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"TensorRT is the NVIDIA SDK whose FP16 and INT8 precision calibration and optimization passes produce the reported latency and memory reductions."}],"review_version":1}