{"id":"5fa73d57-af8e-4d6e-b8cd-6fbeeb678b95","arxiv_id":"2411.08171","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":2.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"Using pretrained VGG-16 and VGG-19 improved wildfire detection accuracy over small custom CNNs, but the study lacks error bars and released data.","lead":"This paper compares six deep learning models for spotting wildfires in images: three small custom-built networks and three large pretrained networks. It finds that the pretrained VGG-16 and VGG-19 achieve higher accuracy and fewer false alarms than the custom models on the authors' private dataset.","discovery_kind":"incremental","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The comparison confounds transfer learning with architecture depth and trainable capacity; no same-architecture ablation is run.","rationale":"I read the paper in good faith: the reported accuracy, precision, and recall numbers may be accurate for the exact models and split used. The reader's concern about dataset leakage and representativeness is plausible and worth checking, but even with a completely clean split the headline conclusion would still not follow from the current design, because transfer learning is never tested against from-scratch training of the same architecture. The paper explicitly avoids training VGG-16/19 from scratch and instead compares them to shallower custom models, so the empirical comparison conflates pretraining with model depth and parameter count. This is the most load-bearing weakness because it undermines the causal claim in the title and conclusion. A matched-architecture ablation would settle whether the advantage is due to transfer learning or to the architectures themselves. The verdict should remain conditional, with the condition sharpened to require this controlled comparison in addition to leakage checks.","tokens_in":9774,"tokens_out":5527,"duration_ms":65083,"concrete_test":"The decisive check is a matched-architecture ablation: take the custom VGG-7 architecture and train two copies on the same cleaned train/validation/test split, one from random initialization and one with the first convolutional layers initialized from ImageNet-trained VGG-16 weights and frozen while training the classifier head identically. If the transfer-initialized VGG-7 does not outperform the random-init VGG-7 by more than seed-level variance, the reported gains cannot be attributed to transfer learning.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that transfer learning outperforms custom-built models is not supported by the comparisons as designed. Section III.B states that VGG-16/19 were not trained from scratch because they 'proved to be ineffective', so the authors substituted shallower custom VGG-7/VGG-10 and a CNN-SVM. Section III.C then freezes all pretrained convolutional layers of VGG-16/19 and trains only 263,169 parameters, while VGG-7 and VGG-10 train 10.09M and 6.65M parameters from scratch. Thus the comparison varies at least two factors at once: architecture depth/capacity and initialization (pretrained versus random). Any performance gap, such as VGG-16's 99.45% test accuracy versus VGG-7's 96.54%, could be due to depth or capacity rather than to transfer learning itself. The conclusion in Section V that pretrained models are more generalized therefore overreaches the experimental design. This is an internal design limitation, not a disagreement with consensus, and it is repairable with a controlled ablation.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper compares three custom-built models (VGG-7, VGG-10, and CNN-SVM) trained from scratch with three pretrained models (VGG-16, VGG-19, and ResNet101) for binary wildfire detection on a custom dataset combining internet images and FLAME. On a 550-image test set, the pretrained VGG-16 achieves the highest accuracy (99.45%) with 2 false positives and 1 false negative, and the paper concludes that transfer learning yields more accurate and more generalizable wildfire detection than custom models. The manuscript includes architecture tables, training/validation curves, and confusion-matrix counts for all six models.","tokens_in":9989,"tokens_out":7577,"duration_ms":71495,"significance":"If the claimed comparison were properly controlled, this would be a useful applied study: it uses a realistic mixed-source dataset, reports full confusion-matrix statistics rather than only aggregate accuracy, and compares six architectures under a common wildfire-detection task. The paper's strengths are the explicit architecture descriptions, the test-set TP/FP/FN/TN counts, and the attention to false-alarm and missed-detection trade-offs. However, the central claim is not established by the experiments as designed because the comparison confounds transfer learning with architecture depth, trainable capacity, and model selection on the same data, and because all results come from single runs without leakage checks or statistical uncertainty. The result would be worth publishing only after a controlled ablation and a more rigorous evaluation protocol.","major_comments":[{"comment":"The headline comparison confounds transfer learning with architecture depth and trainable capacity. Custom VGG-7 and VGG-10 train all of their parameters (10,090,865 and 6,650,993) from scratch, while pretrained VGG-16 and VGG-19 freeze the convolutional base and train only 263,169 parameters (Table I). Section III.B further states that the VGG-16 architecture trained without pretrained weights 'proved to be ineffective,' so the authors substituted shallower custom models. Consequently, the observed advantage of VGG-16 (99.45% test accuracy) over VGG-7 (96.54%) could be due to depth, initialization, or trainable capacity rather than to transfer learning per se. A same-architecture ablation (e.g., VGG-16/19 trained from scratch on the same data, or ImageNet-initialized VGG-7/10) is needed to support the conclusion in Section V that pretrained models are more generalized.","section":"III.B, III.C, Table I"},{"comment":"The evaluation does not establish statistical reliability. Section II.A describes only training and validation splits (3,629 and 385 images), and the test set of 550 images first appears in Section IV.C with no description of how it was selected, whether it overlaps with training or validation, or whether any duplicate or near-duplicate check was performed. All results come from a single run, with no error bars, confidence intervals, or repeated-seed experiments, so the near-perfect accuracies and the small differences between VGG-16 and the custom models cannot be meaningfully assessed. The manuscript should report the split procedure, a leakage check, and repeated-run statistics.","section":"II.A, IV.C, Table III"},{"comment":"The custom models are tuned on the same dataset used for evaluation, which biases the reported test metrics. Section III.A says the CNN-SVM configuration 'was chosen due to its optimal results across test, train, and validation accuracies,' and Section III.B describes 'repetitive trials' for VGG-7 and VGG-10. Because the test set is part of the model-selection loop, the custom models' test numbers are optimistically selected, while the pretrained models are not selected on this criterion. A fixed test set must be held out before any architecture or hyperparameter choices are made.","section":"III.A"},{"comment":"The paper's description of the pretrained VGG models is internally inconsistent. Section III.C states the output activation was changed from softmax to linear for the binary task, but Table I lists softmax for both VGG-16 and VGG-19. In addition, Table I gives the VGG-16 conv5 filters as 1,024 and the VGG-19 conv2 filters as 64, whereas the non-trainable parameter counts (14,714,688 and 20,024,384) match the standard Keras VGG-16/VGG-19 feature extractors with 512 filters in the last block and 128 filters in VGG-19's second block. Please correct Table I and specify the loss function and thresholding used with the linear output.","section":"Table I, III.C"}],"minor_comments":[{"comment":"The dataset description should state the total number of unique images, the number of images per class in the test set, and the process used to create the test split; currently the only mention of a 550-image test set is in Section IV.C.","section":"II.A"},{"comment":"The text says 'as presented in Table I' when reporting ResNet101 validation results, but those results are in Table II; Table I is the architecture table.","section":"IV.B"},{"comment":"The sentence listing models in Table II omits VGG-19 ('VGG-16 and ResNet101'), although Table II includes VGG-19; please correct this for consistency.","section":"IV.B"},{"comment":"Several citations do not support their claims: [24] and [25] are not about diverse landscapes for wildfire detection, and [26] for VGG-16/VGG-19 should be Simonyan and Zisserman's paper rather than the Akhloufi et al. drone survey.","section":"References"},{"comment":"The phrase 'all layers, except the top layers (input and output), are set to be untrainable' should be reworded: in the experiments, the convolutional base is frozen and only the dense classifier is trained.","section":"III.C"},{"comment":"The subplot axes and legends are too small to read at print size; please enlarge them and add clear title and axis labels for each subplot.","section":"Figure 3"},{"comment":"The description of the ResNet modification is incomplete: ResNet101's original classification layer has 1,000 units, not 512; please specify which layers were frozen and which were replaced.","section":"III.D"}],"recommendation":"major_revision","confidential_remarks":"The central design flaw (confounded comparison) is repairable with a controlled same-architecture ablation and a proper hold-out protocol, but the current manuscript overclaims the transfer-learning advantage. I would encourage the editor to request a revision that adds these experiments and repeated-run statistics before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things before reading this paper: the numbers are probably fine, but the central conclusion is not supported by the experiment as designed. The paper compares three custom-built shallow CNNs (VGG-7, VGG-10, CNN-SVM) against three ImageNet-pretrained deep models (VGG-16, VGG-19, ResNet101) for wildfire detection and reports that the pretrained VGGs achieve ~99% test accuracy while the custom models sit around 96-97%. The topic is practically relevant, and the authors do a decent job of documenting the architectures, using a combined internet-plus-FLAME dataset, and reporting not just accuracy but precision, recall, false positives, and false negatives.\n\nWhat's actually new: not much. Comparing pretrained versus scratch CNNs on a small fire dataset is a routine exercise, and the paper doesn't introduce a new architecture, dataset, or theoretical angle. Its value is as an incremental data point.\n\nThe soft spots are real and one is load-bearing. The main issue is that the comparison confounds transfer learning with architecture depth and trainable capacity. The authors say VGG-16 from scratch 'proved to be ineffective,' so they substituted shallower VGG-7/VGG-10, while the pretrained models keep their deep frozen convolutional backbones and only train a small classifier head. Any performance gap could be due to depth or capacity, not transfer learning per se. A same-architecture ablation—train VGG-16 from scratch or fine-tune a shallow model—is missing. That's an internal design limitation, and it undercuts the conclusion in Section V that pretrained models are inherently more generalized. The other issues are softer: single runs with no error bars, near-perfect accuracies suggesting an easy or possibly leaky benchmark (no duplicate checks or split description), and a few internal inconsistencies (e.g., output activation listed as softmax for VGG-16/19 in Table I while the text says linear; ResNet101's validation accuracy of 89.98% versus its test accuracy of 92.73%).\n\nThe reader's conditional verdict matches my own. The paper is not a waste of time: it is clear, honestly reports its choices, and the raw numbers are useful to practitioners in drone-based wildfire detection who want a rough comparison. But the headline claim overreaches the experimental design.\n\nFor peer review: yes, a serious referee should engage, provided the authors are asked to fix the confound with a controlled ablation, run multiple seeds, and release code and data (or at least a cleaned, split-described dataset). As written, it would need major revision to fully support the conclusion.","headline":"A clear but routine comparison of pretrained vs scratch CNNs for wildfire detection; the numbers are plausible but the design confounds transfer learning with architecture depth, so the central claim overreaches.","tokens_in":10489,"tokens_out":2915,"would_cite":false,"duration_ms":28818,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Pretrained VGG-16 beats every from-scratch model in wildfire detection, with only three errors on 550 test images.","keywords":["wildfire detection","transfer learning","VGG-16","VGG-19","ResNet101","custom CNN","CNN-SVM","image classification"],"falsifier":"Run a perceptual-hash duplicate check between the 550 test images and the training set, then re-evaluate all six models on an independent public wildfire dataset; if VGG-16's false positives and false negatives rise toward the custom models' levels, or if a from-scratch model matches it, the claimed transfer-learning advantage is a property of the benchmark rather than the method.","tokens_in":9576,"feed_emoji":"🔥","tokens_out":9887,"duration_ms":91678,"temperature":0.7,"pith_summary":"The paper sets out to determine whether transfer learning outperforms training from scratch for binary wildfire detection. It compares three custom-built models (VGG-7, VGG-10, and CNN-SVM) with three ImageNet-pretrained models (VGG-16, VGG-19, and ResNet101) on a mixed internet-plus-FLAME dataset of roughly 4,500 images. Its central claim is that the transfer-learning models are more accurate and produce fewer false alarms and missed fires, with VGG-16 reaching 99.45% test accuracy, two false positives, and one false negative on a 550-image test set. The authors argue that pretrained models generalize better to real-world distractors, while custom models remain attractive mainly for their faster inference in real-time drone use.","feed_headline":"Pretrained VGG-16 hits 99.45% wildfire-detection accuracy","feed_subtitle":"On 550 held-out images, it made only 2 false alarms and 1 miss, beating every from-scratch model.","key_machinery":"The load-bearing mechanism is frozen-feature transfer learning. Lower convolutional layers of VGG-16 and VGG-19, pretrained on ImageNet's 1,000 classes, are kept untrainable and act as generic extractors of edges, textures, and scene structure; only the final dense layer is collapsed to a single linear node for binary fire/non-fire output. Against this, VGG-7, VGG-10, and CNN-SVM learn every weight from the custom dataset alone, with CNN-SVM using an SVM as the final classifier. The paper's argument is that the pretrained layers supply general visual knowledge that a small wildfire dataset cannot teach from scratch, which is why the VGG transfer models post lower false-positive and false-negative rates.","core_discovery":"On the paper's own numbers, the central discovery is that freezing an ImageNet-pretrained VGG-16 backbone and replacing its classifier with a single linear output node yields 99.45% test accuracy, 99.69% recall, and 99.38% precision, with two false positives and one false negative out of 550 test images. VGG-19 follows at 99.09% accuracy, and the best custom model, CNN-SVM, reaches 96.91% accuracy with 98.46% recall. ResNet101, despite being pretrained, falls to 92.73% accuracy, so the paper's conclusion is not that pretraining always helps, but that transfer learning with VGG features is particularly well suited to this fire/no-fire task. The authors further infer from the results that custom from-scratch models are more prone to false alarms from fire-colored objects and are better reserved for speed-critical real-time deployment.","pith_inferences":["The paper does not test its own explanation that pretrained models resist fire-colored distractors; a dedicated test set of autumn leaves, sunsets, and watchtowers would turn that explanation into a checkable claim.","Because the compared models differ in both pretraining and architecture depth, the accuracy gap is not a pure measure of transfer learning; pretraining the same VGG-7 and VGG-10 architectures on ImageNet would isolate the contribution of pretraining.","The near-ceiling accuracies on a small private dataset suggest the benchmark may be easy; evaluating all six models on an independent public wildfire dataset with identical preprocessing would show whether the transfer-learning advantage persists."],"forward_implications":["For wildfire detection with limited labeled data, pretrained VGG-16 or VGG-19 should be the default starting point over from-scratch CNNs on accuracy grounds.","A 0.31% false-negative rate on VGG-16 means roughly one missed fire per 300 fire images, a level the paper argues is suitable for early-warning systems.","Custom shallow models trade a few percentage points of accuracy for speed, making them the paper's recommended option for real-time drone processing.","ResNet101's weaker 92.73% accuracy shows that pretraining alone does not guarantee transfer success; architecture choice matters in this domain."],"supporting_citations":[{"why":"Supplies the data-augmentation recipe and the earlier SegNet wildfire model from which the custom training set and method are taken.","marker":"[1]"},{"why":"Provides ImageNet, the source dataset whose pretrained weights define the transfer-learning side of the comparison.","marker":"[3]"},{"why":"Cited to support the transferability of learned features across visual domains, the premise the pretrained models rely on.","marker":"[22]"},{"why":"In the paper's text this is the cited source for the pretrained VGG-16 and VGG-19 models.","marker":"[26]"},{"why":"In the paper's text this is the cited source for ResNet101, the third pretrained baseline.","marker":"[27]"}],"fun_headline_variants":["Transfer learning VGG-16 tops wildfire detection at 99.45%","Pretrained VGG-16 beats custom models for wildfire detection","Wildfire detection: pretrained VGG-16 hits 99.45% accuracy","Transfer learning shines: VGG-16 hits 99.45% on wildfire data","99.45% wildfire detection accuracy with pretrained VGG-16"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison depends on the custom fire/non-fire image set being a clean, representative sample with no duplicated or near-duplicated images across the training and test splits, so that the near-perfect accuracies reflect true generalization rather than an easy or contaminated benchmark.","fun_headline_variants_meta":{"raw":{"variants":["Transfer learning VGG-16 tops wildfire detection at 99.45%","Pretrained VGG-16 beats custom models for wildfire detection","Wildfire detection: pretrained VGG-16 hits 99.45% accuracy","Transfer learning shines: VGG-16 hits 99.45% on wildfire data","99.45% wildfire detection accuracy with pretrained VGG-16"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000675,"raw_usage":{"total_tokens":3085,"prompt_tokens":971,"completion_tokens":2114,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":587,"completion_tokens_details":{"reasoning_tokens":2013}},"tokens_in":587,"tokens_out":2114,"duration_ms":14145,"temperature":1.0,"reasoning_tokens":2013,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T21:53:12.360403+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a perceptual-hash duplicate check between the 550 test images and the training set, then re-evaluate all six models on an independent public wildfire dataset; if VGG-16's false positives and false negatives rise toward the custom models' levels, or if a from-scratch model matches it, the claimed transfer-learning advantage is a property of the benchmark rather than the method.","supporting_citations":[{"cited_title":"SegNet: A segmented deep learning based convolutional neural network approach for drones wildfire detection,","cited_arxiv_id":null,"evidence_quote":"Supplies the data-augmentation recipe and the earlier SegNet wildfire model from which the custom training set and method are taken."},{"cited_title":"Unmanned aerial vehi- cles for wildland fires: Sensing, perception, cooperation and assistance,","cited_arxiv_id":null,"evidence_quote":"In the paper's text this is the cited source for the pretrained VGG-16 and VGG-19 models."}],"review_version":1}