{"id":"5a7c0f79-2b94-497f-bda1-e4f59efd8d78","arxiv_id":"2506.08690","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.5,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"Introduces a multi-modal 100m wildfire forecasting benchmark for Canada and shows deep learning models benefit from fusing Sentinel-2 imagery with environmental predictors.","lead":"CanadaFireSat is a new open dataset for predicting wildfire probability at 100-meter resolution across Canada, pairing satellite imagery with weather and vegetation data. The paper benchmarks deep learning models and reports that combining image and environmental inputs yields the best fire forecasts, a potentially useful tool for fire management teams.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Label construction in §2.1.1 lets already-burning fires count as positive labels, so the 60.3% F1 may measure detection of active fire or burn scars rather than 8-day forecasting skill.","rationale":"The reader's weakest assumption is the right target: the temporal aggregation in Section 2.1.1 does not enforce that positive labels correspond to new ignitions in the forecast window, and this is the condition that must hold for the headline 60.3% Test F1 to be a forecasting number rather than a detection number. I considered the alternative concern that Table 4 contradicts the abstract's 'across all metrics' claim, since the ViT ENV-Only model reaches 59.9 Test F1 versus 59.2 for the ViT Multi-Modal model. That is a real wording and claim-calibration issue, but it does not threaten the dataset or the average multi-modal conclusion, so it is less load-bearing. The label-overlap issue, by contrast, affects every headline number and the interpretation of the task itself. I would keep the reader's CONDITIONAL verdict: CanadaFireSat's public code, dataset, and Test Hard construction are genuine contributions, but the forecasting claim needs additional analysis that either filters to new ignitions or quantifies the overlap. The proposed concrete test does exactly this and is feasible with the public NBAC start dates and generation code. No part of this critique impugns the authors' intent; the issue appears to be an evaluation-construction gap rather than a deliberate misrepresentation.","tokens_in":27265,"tokens_out":8474,"duration_ms":91182,"concrete_test":"Using the public CanadaFireSat-Data generation code and NBAC start dates, compute for every positive sample the earliest fire start date relative to the 8-day window start t. Remove from Val and Test all positive samples whose fire started before t, keeping only labels with ignition in [t, t+7], and re-run Table 4 on this filtered subset. Separately, retrain the multi-modal ResNet-50 and the SITS-Only and ENV-Only variants on the corresponding filtered training labels. If the peak Test F1 drops well below 60.3, or if the multi-modal advantage over the best single-modality model shrinks or reverses, the leakage is confirmed; if the metrics are essentially unchanged, the forecasting claim survives.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 2.1.1 defines the task as predicting, for time step t, the probability of a fire in [t, t+7] from predictors in [t-64, t-1]. The label construction, however, first merges all fires overlapping a grid cell into one event spanning its first start to last end, and then marks every 8-day MODIS window overlapping that event as positive. A fire that ignited before t and is still burning during [t, t+7] is therefore a positive label at t even though no new fire starts in the forecast horizon. Because Sentinel-2 inputs end at t-1, those inputs may already contain the active fire or burn scar. The model can then solve the task partly as detection of ongoing burning rather than forecasting of new fires. The paper does not quantify the fraction of positive labels whose NBAC start date precedes t, and the Limitations section does not mention this leakage. Since the burn signal is strongest in the high-resolution optical modality, the absolute 60.3% Test F1 and the claimed multi-modal advantage could both be inflated, even though the Test Hard relative ordering may still hold.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces CanadaFireSat, a benchmark dataset and set of baseline models for 8-day wildfire forecasting at 100 m resolution over Canada, using multi-modal inputs: Sentinel-2 image time series, MODIS products, ERA5-Land reanalysis, and CEMS fire indices. The authors benchmark a ResNet-50-based CNN and a DINOv2-based ViT in three modality settings (satellite only, environmental only, multi-modal), compare against a knowledge-driven FWI baseline, and additionally construct an adversarially sampled Test Hard set. The central claim is that multi-modal temporal inputs outperform single-modal inputs across all metrics, with a peak test F1 of 60.3% on the 2023 season.","tokens_in":27443,"tokens_out":6091,"duration_ms":74358,"significance":"If the results are sound, the dataset is a valuable community resource: it raises the target resolution from the typical ~0.1 deg to 100 m, covers continental Canada, includes a temporal dimension for the high-resolution optical modality, and provides code and data publicly on GitHub and HuggingFace. The adversarial Test Hard set is a thoughtful contribution to studying the difficult ignition-dominated regime. The paper also honestly discusses limitations related to ignition modeling and wetland fires. However, the headline claims currently rest on a label-construction choice that may let models detect ongoing fires or burn scars rather than forecast new ignitions, and the abstract overstates the multi-modal advantage relative to the paper's own Table 4. These issues are fixable but require re-analysis of the benchmark construction and the reported conclusions.","major_comments":[{"comment":"The label construction appears to allow already-burning fires to be labeled as positive at time t, which creates a leakage path through the Sentinel-2 inputs. The temporal aggregation merges all fires overlapping a grid cell into a single event from first start to last end, and every 8-day MODIS window overlapping that event is labeled positive. Thus, for a fire whose start date s is before t and whose end date is after t, the window [t, t+7] is positive even though no new fire starts in that window. Because the model uses Sentinel-2 images from t-64 to t-1, those inputs may contain the active fire or burn scar, allowing the model to solve the task partly as detection rather than forecasting. The paper does not quantify the fraction of positive labels whose NBAC start date precedes t, does not filter such samples, and the Limitations section does not mention this issue. I request a re-analysis with positives defined by fire start dates inside [t, t+7] (or an explicit treatment of ongoing fires), and a comparison of the headline results with and without such samples.","section":"Section 2.1.1"},{"comment":"The abstract states that multi-modal temporal inputs outperform single-modal inputs 'across all metrics,' but Table 4 contradicts this. For ViT-S, the multi-modal model has lower Val PRAUC (43.9 vs 45.2), lower Val F1 (50.0 vs 50.6), and lower Test Hard PRAUC (25.1 vs 25.7) than the satellite-only model. The later text in Section 4.1 correctly phrases the comparison as 'on average,' but the abstract and the concluding sentence need to be revised so that the claim matches the reported per-split results, or explicit qualification should be added that the multi-modal advantage is an average effect and not uniform across metrics and splits.","section":"Abstract; Section 4.1, Table 4"},{"comment":"The multi-modal gains over satellite-only inputs are small (e.g., CNN average PRAUC +1.8 and average F1 +1.3; ViT average PRAUC +1.1 and average F1 +1.7), and all reported results appear to come from a single run per configuration. No standard deviations, repeated seeds, or significance tests are provided. Given that the ViT multi-modal model is worse than satellite-only on several individual metrics, the claimed multi-modal advantage is not robustly established without some measure of run-to-run variability. I request either multiple seeds with error bars or a statistical comparison, or a clearly softened conclusion.","section":"Section 4.1, Table 4"}],"minor_comments":[{"comment":"There is a typographical error in the affiliation: 'Earth Science Division„' should be 'Earth Science Division'.","section":"Title page"},{"comment":"The word 'distriminative' in 'the main distriminative factor' should be 'discriminative'.","section":"Section 2.1.2"},{"comment":"The sentence introducing Sentinel-2 processing says 'This represents 13 images' after describing the average revisit time; this appears to refer to the nominal number of images in a 64-day window, but the relationship could be stated more clearly, especially since later filtering allows variable temporal lengths.","section":"Section 2.2.1"},{"comment":"The layout of Table 3 is unclear: the two blocks of rows are not labeled with the modality setting they correspond to, making it hard to see which rows describe 'satellite images only' versus 'satellite and environmental data.'","section":"Table 3"},{"comment":"The weighted F1 definition in the text is introduced in the caption of Figure 9 rather than in the main text; moving this definition into the main text would improve readability.","section":"Section 4.1"}],"recommendation":"major_revision","confidential_remarks":"The label-construction leakage is the most consequential issue: it directly affects the benchmark's validity and the headline numbers. The authors should be given the opportunity to fix it by restricting positives to new ignitions or by quantifying and reporting the leaked fraction. The abstract overclaim is easy to correct but should not be left as-is. The single-run nature of the comparisons is a secondary concern that strengthens the need for more cautious claims."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Short version: this is a genuinely useful dataset contribution, but the headline forecasting numbers are not clean. The 60.3% F1 and the multi-modal advantage are both suspect because the positive labels include fires that are already burning before t, and the Sentinel-2 inputs end at t-1, so the model can see the burn scar or active fire.\n\nWhat is new: CanadaFireSat is the first 100m, Canada-wide wildfire forecasting benchmark combining Sentinel-2 time series with ERA5, MODIS, and CEMS predictors, covering 2016-2023, with public data and code. The Test Hard set is a good idea: sampling negatives to match the FWI, land cover, and month distribution of positives makes ignition the discriminating factor and gives a lower-bound evaluation. That adversarial set is the most useful part methodologically. The CNN/ViT baselines are reasonable, and they beat the FWI baseline especially at lower FWI values, which is the regime that matters operationally.\n\nSoft spots, in proportion. First and most important: label leakage. Section 2.1.1 merges temporally overlapping fires in a grid cell into one event spanning first start to last end, then labels every 8-day window intersecting that event as positive. A fire that ignited before t and is still burning in [t, t+7] is positive even though no new ignition occurs in the forecast horizon. Since the Sentinel-2 series covers t-64 to t-1, it can contain the active fire or burn scar. The paper never quantifies the fraction of positives whose NBAC start date precedes t, and the limitations section does not mention this. That likely inflates absolute performance and could inflate the multi-modal gain, because the burn signal is strongest in the optical modality. The Test Hard relative ordering may be more robust, but even there the optical leakage is present. Second, the abstract's claim that multi-modal inputs outperform single-modal inputs \"across all metrics\" is contradicted by Table 4: the ViT multi-modal model scores lower than SITS-only on validation F1 (50.0 vs 50.6) and on Test Hard PRAUC (25.1 vs 25.7). The average gains are real but small, around 1-2 points. Third, all results are single runs with no error bars. With differences that small, seed variance could change the ranking.\n\nWho this is for: researchers building wildfire forecasting benchmarks and anyone comparing fusion strategies at high resolution. It deserves a serious referee, but the forecasting claims need to be reworked. Filter positives to new ignitions or quantify the leakage, add multiple seeds or uncertainty estimates, and soften the abstract. The dataset itself is worth having even after those fixes.","headline":"Valuable dataset with real leakage in label construction; the headline forecasting numbers should be read as partly detection until positives are filtered to new ignitions.","tokens_in":28057,"tokens_out":2117,"would_cite":true,"duration_ms":25242,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that fusing high-resolution Sentinel-2 time series with coarse environmental drivers yields the most accurate 100-meter wildfire probability forecasts across Canada, reaching a peak F1 of 60.3% on the 2023 test season.","keywords":["wildfire forecasting","benchmark dataset","multi-modal learning","Sentinel-2","100-meter resolution","boreal ecosystem","deep learning","Earth observation"],"falsifier":"Recompute F1 and PRAUC on the subset of test samples whose National Burned Area Composite fire start date falls inside the 8-day forecast window; if performance drops substantially relative to the full test set, part of the reported skill is detection of ongoing fires rather than prediction of new ignitions.","tokens_in":27000,"feed_emoji":"🔥","tokens_out":9589,"duration_ms":103187,"temperature":0.7,"pith_summary":"This paper introduces CanadaFireSat, a benchmark dataset for forecasting wildfires at 100-meter resolution across Canada on an 8-day horizon, and tries to show that deep models trained on multi-modal temporal inputs—Sentinel-2 multispectral time series plus coarse environmental drivers from ERA5, MODIS, and fire-weather indices—outperform models using either modality alone. On the 2023 test season, which was never seen in training, the best multi-modal CNN reaches 60.3% F1, exceeding both single-modality variants and a knowledge-driven Fire Weather Index baseline. The authors argue that high-resolution satellite imagery, used as proxies for fuel and moisture dynamics, and coarse hydrometeorological predictors carry complementary signal, and that fusing them makes continental-scale high-resolution wildfire probability maps feasible. If correct, this points toward operational tools that allocate fire-management resources at scales current ~0.1-degree products cannot support.","feed_headline":"Satellite plus weather data beats single-source fire forecasts","feed_subtitle":"A new Canadian benchmark hits 60.3% F1 on the 2023 season, beating the fire-weather index baseline.","key_machinery":"The load-bearing object is CanadaFireSat itself: a spatio-temporal benchmark built from National Burned Area Composite fire polygons sampled on a 2.8-kilometer grid, with 2.64-kilometer Sentinel-2 L1C tiles at 10-meter resolution, MODIS, ERA5-Land, and CEMS environmental predictors, an 8-day forecasting horizon, and two negative-sampling schemes including an adversarial Test Hard set. The modeling machinery is a factorized fusion design: each satellite time step is encoded independently by a pretrained ResNet-50 or DINOv2 ViT-S, temporally aggregated with ConvLSTM, while tabular environmental series are projected and encoded by an LSTM, then fused with the spatial features before patch classification at the 100-meter output resolution. This design allows coarse and high-resolution modalities to be combined while keeping the final probability map at the fine spatial scale.","core_discovery":"The central claim is that multi-modal temporal inputs—Sentinel-2 image time series combined with environmental predictors—give the most accurate 100-meter wildfire probability forecasts, beating both satellite-only and environmental-only settings across validation, test, and adversarial test sets, with a peak F1 of 60.3% and PRAUC of 57.0 on the 2023 test set. The paper also establishes that both a ResNet-50 and a ViT-S benefit from fusion, with average gains of +1.8% and +1.1% in PRAUC and +1.3% and +1.7% in F1 over single-modality models, and that both deep models outperform the FWI baseline by double-digit margins. The authors interpret the Test Hard set, sampled adversarially to balance FWI, as revealing a lower-bound performance in which ignition rather than flammability is the discriminating factor.","pith_inferences":["Beyond the paper, a clean stress test would split the test set by whether the fire started before or after the prediction time; the current 8-day aggregation labels any overlapping fire as positive, so this split would quantify how much of the 60.3% F1 is forecasting versus detecting an already-visible fire or burn scar.","Beyond the paper, the open-access input recipe suggests direct transfer to other fire-prone regions such as the pan-Arctic and tropics, with local burned-area composites and local fire-weather indices as the main adaptations.","Beyond the paper, the land-cover results imply that adding radar (Sentinel-1) or soil-moisture-specific predictors, rather than more optical bands, is the most promising next step for wetland and peatland fire forecasting; the paper names SAR as future work, and its own analysis of weak wetland performance supports this."],"forward_implications":["Multi-modal temporal inputs (Sentinel-2 plus environmental predictors) should be the default configuration for high-resolution wildfire forecasting on CanadaFireSat, since they outperform both single-modality settings in every metric on average.","Deep learning models beat the knowledge-driven Fire Weather Index baseline by double-digit margins on the 2023 test set, indicating that learned multi-modal models can serve as a stronger fire-danger signal than the standard index.","The adversarial Test Hard evaluation, where negative samples are matched to positives by FWI, yields much lower performance, so models trained and evaluated on standard negative sampling overestimate skill in extreme seasons like 2023.","Using the full Sentinel-2 time series rather than a single recent image improves validation performance for both architectures, so temporal dynamics carry predictive information beyond a static snapshot.","The land-cover breakdown shows weakest performance on wetland and cropland fires, so operational maps from this benchmark are most reliable for forested areas and should be supplemented by specialized models for peatland and agricultural ignitions."],"supporting_citations":[{"why":"It supplies the National Burned Area Composite polygons that define the positive fire labels in CanadaFireSat.","marker":"(Hall et al., 2020)"},{"why":"It is the prior 1-km US wildfire forecasting dataset whose resolution and sampling CanadaFireSat extends.","marker":"(Huot et al., 2020)"},{"why":"It is the prior 1-km Eastern Mediterranean wildfire danger dataset used as a resolution and method comparison point.","marker":"(Prapas et al., 2021)"},{"why":"It established convolutional LSTM spatio-temporal wildfire danger forecasting, the basis for the CNN baseline here.","marker":"(Kondylatos et al., 2022)"},{"why":"It provides the ResNet-50 backbone used to encode each Sentinel-2 time step.","marker":"(He et al., 2016)"},{"why":"It provides the vision transformer architecture benchmarked as the alternative encoder.","marker":"(Dosovitskiy, 2020)"},{"why":"It provides the DINOv2 ViT-S weights used for transformer-based image encoding.","marker":"(Oquab et al., 2023)"},{"why":"It motivates the per-feature MLP projection used to encode tabular environmental predictors.","marker":"(Gorishniy et al., 2022)"},{"why":"It establishes the 8-day temporal aggregation and global wildfire forecasting setup that CanadaFireSat reuses.","marker":"(Prapas et al., 2022)"},{"why":"It supports using multi-spectral satellite time series as proxies for fire predictors in Canadian peatlands.","marker":"(Pelletier et al., 2023)"}],"fun_headline_variants":["Multi-modal fusion lifts wildfire forecast F1 to 60.3%","High-resolution wildfire forecasting gets multi-modal boost","CanadaFireSat: multi-modal 100m wildfire probability benchmark","Fusing satellite and weather data improves fire forecasts","Multi-modal inputs beat single-source for 100m fire risk"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that a label at time t means a new fire starts in the next eight days, but the dataset actually labels any fire overlapping that window, so a fire already burning in the input imagery still counts as a positive.","fun_headline_variants_meta":{"raw":{"variants":["Multi-modal fusion lifts wildfire forecast F1 to 60.3%","High-resolution wildfire forecasting gets multi-modal boost","CanadaFireSat: multi-modal 100m wildfire probability benchmark","Fusing satellite and weather data improves fire forecasts","Multi-modal inputs beat single-source for 100m fire risk"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000258,"raw_usage":{"total_tokens":1620,"prompt_tokens":1019,"completion_tokens":601,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":635,"completion_tokens_details":{"reasoning_tokens":520}},"tokens_in":635,"tokens_out":601,"duration_ms":6384,"temperature":1.0,"reasoning_tokens":520,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T05:04:39.077960+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Recompute F1 and PRAUC on the subset of test samples whose National Burned Area Composite fire start date falls inside the 8-day forecast window; if performance drops substantially relative to the full test set, part of the reported skill is detection of ongoing fires rather than prediction of new ignitions.","supporting_citations":[],"review_version":1}