{"id":"0cb973f7-1412-49c4-97fd-e0bc0a2899b1","arxiv_id":"2412.02831","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"FLAME 3 provides the first aerial radiometric thermal wildfire image dataset with per-pixel temperature TIFFs and nadir thermal plots, plus a pipeline and benchmark.","lead":"This paper introduces FLAME 3, a dataset of paired visual and radiometric thermal drone imagery from six prescribed fires, along with a processing pipeline that turns raw thermal files into temperature-ready training data. It also reports image classification benchmarks suggesting the thermal temperature data helps fire detection models, though the evaluation has important caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The TIFF classification benchmark is circular: Fire/No-Fire labels are derived from temperature thresholds in the same TIFF inputs, so the reported 91.38% accuracy does not demonstrate generalization.","rationale":"The reader's weakest_assumption precisely identifies the circularity between threshold-generated labels and TIFF inputs. My independent reading of Sections III.A.1 and V.B confirms that the classification benchmark cannot distinguish genuine generalization from threshold memorization, because the same temperature values that define the labels are also the model input. This is the single most load-bearing concern because the paper's fourth stated contribution is the empirical demonstration that TIFF inputs improve performance. The dataset novelty claim (first aerial radiometric dual-spectrum UAV-collected prescribed fire imagery dataset) is plausible and not directly contradicted by the text, and the paper is transparent about alignment limitations and the preliminary nature of threshold labeling. However, the performance claims should not be accepted as evidence without addressing the circularity. The reader's verdict of CONDITIONAL remains appropriate: the dataset contribution can stand, but the paper should either add a human-expert-labeled held-out test set, report a temperature-threshold baseline, or explicitly reframe the classification results as a sanity check rather than a generalization result. My recommendation is therefore UNCHANGED relative to the reader's conditional verdict.","tokens_in":17301,"tokens_out":3236,"duration_ms":36979,"concrete_test":"Run a zero-parameter baseline on the Section V.B test set (Sycan Marsh) that predicts Fire iff the maximum TIFF pixel temperature exceeds 200°C and No-Fire iff it is below 80°C, handling the middle range with the same rule used to generate the dataset labels (or, failing that, by excluding those samples exactly as the pipeline did). Compare its accuracy to the reported 91.38% FLAME 3 TIFF-only CNN accuracy. If the baseline matches or exceeds 91.38%, the reported result is fully explained by the threshold labeling rule and does not support the generalization claim.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The strongest quantitative evidence that radiometric TIFFs improve wildfire classification comes from Section V.B, where FLAME 3 TIFF-only input reaches 91.38% test accuracy on Sycan Marsh. But the Fire/No-Fire ground truth for those test images was itself generated, at least in part, by the temperature-threshold rule described in Section III.A.1: images with maximum temperature below about 80°C are No-Fire, above about 200°C are Fire, with manual review only for intermediate cases. A model given the same TIFF as input can therefore achieve high accuracy by learning a simple temperature-threshold rule, without learning anything about fire semantics or generalizing across environments. The paper's claim that TIFF files 'contain rich contextual information' and allow models to 'generalize much better' (Section V.B) is confounded by this label--input overlap. This is not an internal inconsistency, because the paper is transparent that thresholding was used for preliminary labeling, but the evaluation design does not control for the resulting circularity. Consequently, the classification comparison in Table VIII cannot support the stated contribution of 'highlighting improved performance with the use of radiometric thermal TIFF data.' The dataset itself may still be valuable, but this particular empirical argument is not secure.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"This manuscript introduces FLAME 3, a UAV-collected dataset of side-by-side RGB and radiometric long-wave infrared imagery from six prescribed fires, together with thermal TIFF files and nadir thermal plot imagery. It also presents a semi-automated processing pipeline that pairs images, extracts radiometric TIFFs, and applies manual field-of-view corrections, and it reports classification experiments comparing FLAME 1, FLAME 2, and FLAME 3 under RGB-only, RGB-thermal JPEG, RGB-TIFF, and TIFF-only inputs. The authors report that FLAME 3 TIFF-only input achieves the highest test accuracy (91.38% on a Sycan Marsh test set) and claim this demonstrates that radiometric TIFF data provide richer contextual information and better generalization.","tokens_in":17600,"tokens_out":5488,"duration_ms":54960,"significance":"If the dataset is made publicly available as described, it would fill a genuine gap: publicly accessible aerial radiometric thermal wildfire imagery is currently almost nonexistent, and the per-pixel temperature information could support detection, segmentation, and fire-behavior modeling. The data collection guidelines and the processing pipeline are practical contributions, and the Kaggle subset provides a checkable artifact. However, the classification evaluation that supports the paper's headline claim about the value of TIFF data is confounded by the fact that the Fire/No-Fire labels are derived from temperature thresholds in the same TIFF modality, and the comparison lacks statistical rigor. These issues do not invalidate the dataset itself, but they require revision of the empirical claims before the paper can be accepted.","major_comments":[{"comment":"The Fire/No-Fire labels are generated by applying temperature thresholds to the radiometric TIFF data: images with maximum temperature below approximately 80°C are labeled No-Fire, those above approximately 200°C are labeled Fire, with manual review only for intermediate cases. The TIFF-only classifier takes exactly the same temperature values as input. An accuracy of 91.38% on the Sycan test set can therefore largely be achieved by learning the labeling rule itself, and it does not by itself demonstrate that TIFF files 'contain rich contextual information' or 'generalize much better' across environments. To support the central quantitative claim, please either evaluate TIFF inputs against labels generated independently of the temperature data (for example, manual annotation from RGB imagery or independent human review of the borderline cases), or report a simple baseline that classifies each image by its maximum temperature using the same thresholds and show that the learned model substantially exceeds that baseline. Without this control, Table VIII cannot support the claim that radiometric TIFF inputs improve wildfire classification.","section":"Section III.A.1 and Section V.B (Table VIII)"},{"comment":"The FLAME 3 training set contains imagery from Shoetank, and the FLAME 3 test set also contains imagery from Shoetank. The text states that the specific testing images from Shoetank were 'randomly sampled and removed from the training set,' but other Shoetank images remain in training. This same-burn overlap means that the reported FLAME 3 test accuracy may be optimistically biased and does not demonstrate generalization to unseen burn sites or fuel types. Please either exclude the entire Shoetank burn from training when evaluating on Shoetank, or report per-burn results so that readers can assess cross-burn performance. The exact construction of the train/test split should also be stated unambiguously.","section":"Section V.A (Table VII)"},{"comment":"All classification results are reported as single point estimates without error bars, confidence intervals, or the number of training runs per configuration. With 232 test images in Table VIII, the 0.43 percentage point difference between TIFF-only (91.38%) and RGB-TIFF (90.95%) is well within sampling noise, and the ranking of the top-performing inputs is not statistically supported. Please report means and standard deviations over multiple seeds, or at least binomial confidence intervals for the test accuracies, and moderate the comparative claims accordingly.","section":"Section V (Tables VII and VIII)"}],"minor_comments":[{"comment":"There is a typo in the text: 'hyperaparameter' should be 'hyperparameter'.","section":"Section V.A"},{"comment":"'Eucledian distance' should be 'Euclidean distance'.","section":"Section IV.B.1"},{"comment":"The novelty claim is phrased differently in the abstract ('first comprehensive collection') and in Section II.B.2 ('first ever aerial radiometric dual multi-spectral UAV-collected prescribed fire imagery dataset'); please align these wordings to avoid ambiguity.","section":"Abstract and Section II.B.2"},{"comment":"The caption of Figure 8 says 'from Figure 8' where it appears to mean the overlay shown in Figure 7; please correct the cross-reference.","section":"Figure 8"}],"recommendation":"major_revision","confidential_remarks":"The dataset and pipeline appear to be genuinely useful contributions, and the authors are transparent about their threshold-based labeling procedure. However, the circularity of the TIFF evaluation is a serious concern because it directly undercuts the paper's stated contribution about improved performance with radiometric TIFF data. The issue is fixable within the scope of the manuscript by adding a threshold baseline, using independent labels, or substantially moderating the claim. The same-burn overlap and lack of uncertainty estimates should also be addressed. I would be willing to review a revised version."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading for the data, not for the numbers. FLAME 3 is genuinely the first aerial dataset I know of that ships radiometric thermal TIFFs—per-pixel temperatures—alongside paired RGB from prescribed fires. That's a real gap, and the six-burn collection, the FOV-correction pipeline, the nadir thermal plots, and the public Kaggle subset make the resource usable. The authors document their collection procedures thoroughly, and Table I is a useful map of what's missing in existing aerial fire datasets. That's the kind of detail dataset papers usually skimp on.\n\nThe soft spots are in the evaluation. The Fire/No-Fire labels in Section III.A.1 are generated by a temperature-threshold rule (max temp below ~80°C No-Fire, above ~200°C Fire, with manual review only for intermediate cases). Then Section V.B feeds the same TIFFs into a classifier and reports 91.38% accuracy. That's largely a restatement of the labeling rule, not evidence that TIFFs 'generalize much better.' The model could simply learn to threshold the input. The paper is transparent about the thresholding, and I don't think the authors are hiding anything, but they didn't control for the circularity. Test sets are also small (116 per class in Table VIII, 348 per class in Table VII) and no error bars or significance tests are reported. In Table VII, Shoetank appears in both training and test—the authors say the specific test images were removed from training, which avoids image-level leakage, but the site-level overlap still inflates optimism. The full six-burn dataset is only 'available upon request,' not openly archived; the Kaggle subset helps, but it's a single burn.\n\nNone of this kills the dataset contribution. The central claim—that radiometric thermal aerial wildfire imagery is new and useful—stands. What doesn't stand is the specific performance comparison. That section needs rework: use human-verified labels for the test set (or at least a holdout set labeled independently of the temperature rule), report confidence intervals, and do a site-holdout evaluation. The authors should also deposit the full dataset with a DOI and versioned processing code.\n\nThe paper deserves a serious referee. It's a dataset paper, and the resource itself is valuable enough to publish after revision. My recommendation: send it out, but tell the authors the classification benchmark is circular and must be redone before acceptance.","headline":"Valuable new data resource with a circular benchmark that needs fixing before the performance claims can be taken seriously.","tokens_in":18122,"tokens_out":3423,"would_cite":true,"duration_ms":33041,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A new drone dataset pairs visible video with per-pixel temperature maps from prescribed fires, and models trained on the raw temperatures outperform RGB-only fire detectors.","keywords":["wildfire detection","radiometric thermal imagery","UAV","thermal TIFF","RGB-thermal pairs","prescribed fire","fire image dataset","image processing pipeline"],"falsifier":"Have expert annotators label a held-out set of FLAME 3 frames as Fire/No-Fire using visible evidence and fire perimeters, ignoring the TIFF temperature values, then retrain and test the TIFF-only model on those labels; if its accuracy drops to the level of RGB-only models, the claim that radiometric inputs 'generalize much better' is not supported.","tokens_in":17153,"feed_emoji":"🔥","tokens_out":4698,"duration_ms":45381,"temperature":0.7,"pith_summary":"This paper argues that the missing ingredient in AI wildfire management is radiometric thermal data: per-pixel temperature values rather than color-mapped infrared images. It presents FLAME 3, a dataset of synchronized visible and radiometric long-wave infrared imagery collected by drones at six prescribed burns, together with a processing pipeline that turns raw drone files into aligned RGB/thermal-TIFF pairs ready for machine learning. The paper claims this is the first comprehensive aerial dual-spectrum radiometric fire dataset, and reports that models trained on the temperature TIFFs reach about 91% Fire/No-Fire test accuracy, outperforming RGB-only and JPEG-thermal inputs and generalizing across burn sites. If correct, the dataset removes the main obstacle to temperature-aware wildfire detection, segmentation, and fire-behavior modeling.","feed_headline":"Per-pixel heat maps beat RGB for drone wildfire detection","feed_subtitle":"Six prescribed burns, side-by-side visible and per-pixel temperature imagery, plus a reusable collection pipeline.","key_machinery":"The load-bearing object is the radiometric thermal TIFF: a one-band raster in which every pixel stores a temperature estimate from the long-wave infrared camera. It carries the argument because temperature is the quantity that separates fire from non-fire directly, which lets the authors auto-generate Fire/No-Fire labels by thresholding, gives pixel-level temperature ground truth for possible segmentation and regression tasks, and provides the input modality that scores highest in their classification tests. The supporting machinery is the FOV-correction step that crops and scales RGB frames toward the thermal camera's 640x512 view so that paired data can later be fused by convolution.","core_discovery":"FLAME 3 provides paired RGB images and radiometric thermal TIFF files from six prescribed burns in pine, grass, and sagebrush fuels, where each TIFF pixel is a temperature reading rather than a color. The authors' extraction pipeline converts proprietary radiometric JPEG metadata into standard TIFFs, regenerates thermal JPEGs with a fixed color map, and crops/scales the RGB frames to approximate 1:1 pixel alignment with the thermal camera. Temperature thresholds automatically assign preliminary Fire/No-Fire labels (below about 80°C as No-Fire, above about 200°C as Fire), which human reviewers then clean. In their classification comparison, the FLAME 3 TIFF-only input achieved 91.38% test accuracy and RGB-TIFF 90.95%, above all FLAME 1/FLAME 2 combinations, which the authors attribute to the richer contextual information in radiometric data. The dataset also includes nadir thermal plots, repeat overhead radiometric sequences that can measure rate of spread and energy release.","pith_inferences":["Because the benchmark labels are created by temperature thresholds on the same TIFFs the top model sees, the 91% accuracy likely overstates real-world generalization; an independent label source, such as expert-annotated fire perimeters, would give a fairer comparison.","The fully aligned pixel pairs could enable a model to regress temperature from RGB alone, letting non-radiometric cameras act as cheap temperature proxies during future fires.","The six-burn diversity is also a stress test: if the dataset is used as a pretraining pool, held-out wildfires (not prescribed burns) should be used to see whether radiometric features transfer to more intense fire behavior.","Reducing the residual ~20-pixel alignment error with algorithmic distortion correction would unlock early convolutional fusion of RGB and thermal layers, which the authors expect to outperform late fusion."],"forward_implications":["Models trained on FLAME 3 can treat temperature as a first-class input, making fire visible through smoke and reducing false positives that plague RGB-only detectors.","The same TIFFs can auto-generate pixel-level Fire/No-Fire masks via Otsu or hysteresis thresholding, cutting the manual labeling burden for segmentation.","Nadir thermal plots provide repeat georeferenced temperature maps that can support rate-of-spread and energy-release measurements during prescribed burns.","The classification results imply that replacing JPEG thermal images with radiometric TIFFs improves accuracy even with fewer training samples.","The published collection and processing procedures give other groups a repeatable path to gather radiometric dual-spectrum fire data."],"supporting_citations":[{"why":"Supplies the FLAME 3 computer-vision subset of paired RGB and radiometric thermal imagery from Sycan Marsh.","marker":"[1]"},{"why":"Supplies the nadir thermal plot data from Hanna Hammock that supports the modeling subset and repeat-imagery claims.","marker":"[2]"},{"why":"FLAME 1 is the prior single-burn dataset that FLAME 3 extends and the RGB baseline in the classification comparison.","marker":"[21]"},{"why":"FLAME 2 is the prior RGB/thermal-pair dataset that FLAME 3 extends and the paired baseline in the classification comparison.","marker":"[22]"},{"why":"Provides the one-stream and two-stream CNN architectures used in the Fire/No-Fire classification evaluation.","marker":"[56]"},{"why":"Establishes the method for using UAS-derived infrared imagery to measure rate of spread and energy release from nadir plots.","marker":"[52]"},{"why":"The notable non-wildfire thermal TIFF dataset cited to support the claim that FLAME 3 is the first UAV radiometric fire dataset.","marker":"[50]"},{"why":"Example of an existing RGB-thermal wildfire dataset with unaligned pairs that motivates FLAME 3's FOV correction.","marker":"[6]"}],"fun_headline_variants":["FLAME 3: per-pixel temperature data for wildfire AI","UAV radiometric thermal dataset fuels wildfire models","Side-by-side RGB and thermal: FLAME 3 dataset","Temperature-sensing drones: new wildfire dataset","Six burns of radiometric thermal data for fire AI"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Fire/No-Fire labels used to judge the models come from temperature thresholds applied to the very same TIFF pixels the best model reads as input, so the headline accuracy partly measures the threshold rather than independent ground truth.","fun_headline_variants_meta":{"raw":{"variants":["FLAME 3: per-pixel temperature data for wildfire AI","UAV radiometric thermal dataset fuels wildfire models","Side-by-side RGB and thermal: FLAME 3 dataset","Temperature-sensing drones: new wildfire dataset","Six burns of radiometric thermal data for fire AI"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000186,"raw_usage":{"total_tokens":1357,"prompt_tokens":1007,"completion_tokens":350,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":623,"completion_tokens_details":{"reasoning_tokens":272}},"tokens_in":623,"tokens_out":350,"duration_ms":4375,"temperature":1.0,"reasoning_tokens":272,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T23:03:27.856794+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have expert annotators label a held-out set of FLAME 3 frames as Fire/No-Fire using visible evidence and fire perimeters, ignoring the TIFF temperature values, then retrain and test the TIFF-only model on those labels; if its accuracy drops to the level of RGB-only models, the claim that radiometric inputs 'generalize much better' is not supported.","supporting_citations":[{"cited_title":"Flame 3 computer vision subset (sycan marsh),","cited_arxiv_id":null,"evidence_quote":"Supplies the FLAME 3 computer-vision subset of paired RGB and radiometric thermal imagery from Sycan Marsh."},{"cited_title":"Flame 3 -nadir thermal fire image dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the nadir thermal plot data from Hanna Hammock that supports the modeling subset and repeat-imagery claims."},{"cited_title":"Aerial imagery pile burn detection using deep learning: The flame dataset,","cited_arxiv_id":null,"evidence_quote":"FLAME 1 is the prior single-burn dataset that FLAME 3 extends and the RGB baseline in the classification comparison."},{"cited_title":"Wildland fire detection and monitoring using a drone-collected RGB/IR image dataset,","cited_arxiv_id":null,"evidence_quote":"Provides the one-stream and two-stream CNN architectures used in the Fire/No-Fire classification evaluation."},{"cited_title":"Pixels to pyrometrics: Uas-derived infrared imagery to evaluate and monitor prescribed fire behavior and effects,","cited_arxiv_id":null,"evidence_quote":"Establishes the method for using UAS-derived infrared imagery to measure rate of spread and energy release from nadir plots."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"The notable non-wildfire thermal TIFF dataset cited to support the claim that FLAME 3 is the first UAV radiometric fire dataset."},{"cited_title":"RGB-thermal wildfire dataset,","cited_arxiv_id":null,"evidence_quote":"Example of an existing RGB-thermal wildfire dataset with unaligned pairs that motivates FLAME 3's FOV correction."}],"review_version":1}