{"id":"bc0ad943-d097-4f40-a119-687146ab5624","arxiv_id":"2606.27128","paper_version":1,"verdict":"UNVERDICTED","confidence":"LOW","novelty_score":6.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"FlameVQA is a new VQA benchmark with 34 questions per image across six operational groups for UAV wildfire intelligence, using RGB-thermal pairs and providing MLLM baselines that highlight failures in smoke detection and coverage estimation.","lead":"FlameVQA introduces a multiple-choice visual question answering benchmark for UAV wildfire monitoring that pairs RGB imagery with radiometric thermal TIFFs from the FLAME 3 dataset. A smart generalist might read it to see how current AI models handle safety-critical reasoning tasks like fire detection through smoke.","discovery_kind":"new_application","skeptic_critique":{"model":"grok-4.3","headline":"Ground-truth label reliability for the 34 questions per image depends on MLLM-assisted annotation whose error characteristics remain unquantified.","rationale":"The reader's weakest_assumption directly identifies the load-bearing point for any new VQA benchmark. The full text expands the method description but does not add the quantitative safeguards needed to close the concern; therefore the UNVERDICTED verdict and low confidence are unchanged.","tokens_in":1714,"tokens_out":355,"duration_ms":15849,"concrete_test":"On a random 10% subset of images, obtain independent human-only annotations (no MLLM assistance) for the six operational groups; compute per-question agreement (Cohen's kappa) between the original pipeline labels and the new human labels. If kappa < 0.7 on smoke-related presence or coverage questions, the benchmark's reliability claim is weakened.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central claim—that current MLLMs require domain-specific adaptation—rests on observed performance gaps (strong with cross-modal cues, failures on smoke presence detection and coverage estimation). These gaps are only interpretable if the ground-truth answers are accurate. The annotation pipeline (MLLM-assisted labeling + deterministic thermal rules on radiometric TIFFs + consistency checks + human auditing) is described at a high level, but the full text does not report quantitative validation such as inter-annotator agreement, error rates on held-out expert labels, or ablation of the MLLM component. If the assisting MLLM shares failure modes with the evaluated models (e.g., smoke occlusion), or if thermal rules are incomplete for distribution/coverage questions, systematic label bias could artifactually create the reported gaps.","agreement_with_reader":"agree"},"referee_report":{"model":"grok-4.3","summary":"The paper introduces FlameVQA, a multiple-choice VQA benchmark for UAV wildfire monitoring built on the FLAME 3 dataset. It pairs RGB imagery with radiometric thermal TIFFs to support 34 questions per image across six operational groups (detection, localization, distribution/coverage estimation, cross-modal reasoning, and flight planning). Labels are produced by an MLLM-assisted annotation pipeline that incorporates deterministic thermal rules on radiometric data, cross-question consistency checks, and human auditing. Baseline evaluations of representative MLLMs show strong results when explicit cross-modal cues are supplied but notable failures on presence detection under heavy smoke and on coverage estimation tasks. The authors conclude that current MLLMs require domain-specific adaptation for disaster monitoring and release the dataset and benchmark code at github.com/mobiiin/WildFire_VQA.","tokens_in":1877,"tokens_out":588,"duration_ms":28241,"significance":"If the ground-truth labels are shown to be reliable, the benchmark would offer a valuable, physically grounded resource for evaluating MLLMs on safety-critical aerial reasoning tasks that RGB-only models struggle with. The explicit use of radiometric thermal supervision and the open release of the dataset plus evaluation code are concrete strengths that support reproducibility and follow-on work.","major_comments":[{"comment":"Annotation Pipeline (described in the abstract and § on dataset construction): no quantitative validation of label quality is reported, such as inter-annotator agreement, error rates against held-out expert labels, or an ablation measuring the contribution of the assisting MLLM. This is load-bearing for the central claim, because the reported performance gaps (strong with cross-modal cues, failures on smoke presence and coverage) are only interpretable if the 34-question ground truth is accurate; systematic bias from the annotation process could artifactually produce those gaps.","section":"Annotation Pipeline"},{"comment":"Evaluation section: the paper states that MLLMs exhibit 'notable failures' on presence detection under heavy smoke and coverage estimation, yet provides no breakdown of the number or distribution of such cases, no statistical tests on the performance differences, and no controls confirming that the evaluated models were given identical input formats and prompting as the annotation MLLM. Without these details the strength of the adaptation recommendation cannot be assessed.","section":"Evaluation Results"}],"minor_comments":[{"comment":"The abstract lists six capability groups but does not enumerate the exact 34 questions or their distribution across groups; a compact table in the main text would improve readability.","section":"Abstract"},{"comment":"Figure captions and table headers should explicitly state whether thermal TIFFs are used only for annotation or also supplied to the evaluated MLLMs.","section":null}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":{"model":"grok-4.3","summary":"We thank the referee for the constructive and detailed feedback. The comments correctly identify areas where additional validation and analysis would strengthen the manuscript's claims regarding label reliability and evaluation robustness. We address each major comment below and commit to revisions that directly incorporate the suggested improvements.","responses":[{"response":"We agree that the absence of quantitative validation metrics for the annotation pipeline is a limitation that affects the interpretability of the results. The current manuscript relies on the combination of deterministic thermal rules, cross-question consistency checks, and human auditing to promote label quality, but does not report inter-annotator agreement, error rates against expert labels, or an ablation of the MLLM's role. In the revised version, we will add these elements: agreement statistics from the human auditing phase, error analysis on a held-out subset, and an ablation study quantifying the MLLM's contribution to the final labels. This will directly address the concern and support the central claims.","revision_made":"yes","referee_comment":"[Annotation Pipeline] Annotation Pipeline (described in the abstract and § on dataset construction): no quantitative validation of label quality is reported, such as inter-annotator agreement, error rates against held-out expert labels, or an ablation measuring the contribution of the assisting MLLM. This is load-bearing for the central claim, because the reported performance gaps (strong with cross-modal cues, failures on smoke presence and coverage) are only interpretable if the 34-question ground truth is accurate; systematic bias from the annotation process could artifactually produce those gaps."},{"response":"We concur that the evaluation section would benefit from greater quantitative detail and transparency to substantiate the observed failures and the recommendation for domain-specific adaptation. The manuscript currently describes the failures qualitatively without case breakdowns, statistical tests, or explicit confirmation of input/prompting equivalence. In the revision, we will include: (1) a breakdown of failure cases by subcategory (e.g., counts and distributions for heavy-smoke detection and coverage tasks), (2) statistical significance tests comparing model performances, and (3) clarification of the input formats and prompting used for baselines versus the annotation MLLM, noting any controls or differences. These additions will allow readers to better assess the findings.","revision_made":"yes","referee_comment":"[Evaluation Results] Evaluation section: the paper states that MLLMs exhibit 'notable failures' on presence detection under heavy smoke and coverage estimation, yet provides no breakdown of the number or distribution of such cases, no statistical tests on the performance differences, and no controls confirming that the evaluated models were given identical input formats and prompting as the annotation MLLM. Without these details the strength of the adaptation recommendation cannot be assessed."}],"tokens_in":1485,"tokens_out":581,"duration_ms":25256,"standing_objections":[]},"desk_editor":{"model":"grok-4.3","letter":"The main takeaway is that FlameVQA creates a new VQA benchmark on FLAME 3 UAV imagery by pairing RGB with radiometric thermal TIFFs and defining 34 questions per image across six operational groups. This is a concrete addition for the wildfire monitoring niche.\n\nThe work does a few things right. The questions are tied to actual tasks like presence detection under smoke, localization, coverage estimation, and flight planning. Using deterministic thermal rules on the temperature data for part of the labeling gives the answers a physical basis that pure RGB benchmarks lack. Releasing the dataset and code openly lets others test and extend it. The reported MLLM baselines show a clear pattern: models improve with cross-modal cues but drop on heavy smoke and coverage tasks, which matches what one would expect in this domain.\n\nThe soft spot is the annotation process. The abstract describes MLLM-assisted labeling combined with thermal rules, consistency checks, and human auditing, but it gives no numbers on inter-annotator agreement, error rates against expert labels, or how much the MLLM step was validated. If the assisting model shares failure modes with the evaluated ones, the performance gaps could be partly artifactual. That concern is central because the claim about needing domain-specific adaptation depends on the labels being trustworthy.\n\nThis paper is for people working on multi-modal models for remote sensing or disaster response. It will interest readers who need testbeds with operational questions and thermal supervision. It shows clear thinking about the application constraints and does not overclaim.\n\nI would send it to peer review. The new benchmark and open data are worth referee time, even if the authors need to add quantitative validation of the labels.","headline":"FlameVQA adds a targeted wildfire VQA benchmark with thermal grounding on real UAV data, but label reliability rests on an unquantified annotation pipeline.","tokens_in":2372,"tokens_out":418,"would_cite":false,"duration_ms":24221,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"grok-4.3","headline":"MLLMs succeed on wildfire VQA with explicit thermal cues but fail on smoke-obscured detection and coverage estimation.","keywords":["wildfire monitoring","visual question answering","UAV imagery","thermal imaging","multimodal large language models","disaster response","benchmark dataset","radiometric supervision"],"falsifier":"Independent expert re-labeling of a random subset of images that yields disagreement rates above 10 percent on presence detection or coverage questions.","tokens_in":2615,"feed_emoji":"🔥","tokens_out":686,"duration_ms":26077,"temperature":0.7,"pith_summary":"The paper presents FlameVQA, a benchmark built on paired RGB and radiometric thermal UAV images from wildfire scenes, with 34 multiple-choice questions per image across detection, localization, coverage, cross-modal reasoning, and flight planning. It evaluates representative multimodal large language models and reports strong results when thermal information is supplied directly, yet clear shortfalls in identifying fire presence through heavy smoke and in estimating burned area distribution. A sympathetic reader would care because UAVs already collect such paired data in disaster zones, so reliable automated reasoning could reduce pilot exposure and speed response where smoke blocks normal vision. The work supplies open data and baselines to test whether models can be adapted for these safety-critical tasks.","feed_headline":"Benchmark reveals MLLM failures on smoke-hidden fire detection","feed_subtitle":"Paired thermal data shows current models need adaptation for UAV wildfire coverage and presence tasks under occlusion.","key_machinery":"FlameVQA benchmark consisting of 34 questions per image spanning six operational groups, generated via MLLM-assisted annotation augmented by thermal rules and cross-question consistency checks.","core_discovery":"FlameVQA establishes a physically grounded VQA benchmark on the FLAME 3 dataset that pairs RGB imagery with radiometric thermal TIFFs to support temperature-verified reasoning over complex aerial wildfire scenes. Using MLLM-assisted annotation, deterministic thermal rules, consistency checks, and human auditing to create reliable labels, the benchmark reveals that current MLLMs achieve strong performance when given explicit cross-modal thermal cues yet exhibit notable failures on presence detection under heavy smoke and on coverage estimation tasks.","pith_inferences":["Similar paired thermal-RGB benchmarks could be created for flood or earthquake damage assessment using the same annotation pipeline.","Real-time UAV systems might incorporate the benchmark questions as an online evaluation loop to flag when model outputs become unreliable.","If thermal supervision proves sufficient, future models could be trained to request temperature data on demand rather than always receiving it."],"forward_implications":["MLLMs can already assist UAV wildfire tasks when thermal data is explicitly provided in the prompt.","Persistent failures on smoke and coverage indicate that standard training leaves gaps in handling occlusion and scale variation.","The open dataset supplies a concrete testbed for measuring whether domain-specific fine-tuning or architectural changes close those gaps.","Successful adaptation would directly improve automated support for flight planning and resource allocation during active fires."],"fun_headline_variants":["MLLMs struggle detecting fires under smoke in FlameVQA thermal tests","FlameVQA thermal benchmark shows MLLM failures in smoke and coverage","Current MLLMs fail coverage estimation and smoke-hidden detection tests","Thermal data benchmark shows MLLM gaps for UAV wildfire monitoring"],"cache_read_input_tokens":64,"weakest_assumption_plain":"The combination of MLLM-assisted annotation, deterministic thermal rules, consistency checks, and human auditing produces accurate ground-truth labels for every question.","fun_headline_variants_meta":{"raw":{"variants":["MLLMs struggle detecting fires under smoke in FlameVQA thermal tests","FlameVQA thermal benchmark shows MLLM failures in smoke and coverage","Current MLLMs fail coverage estimation and smoke-hidden detection tests","Thermal data benchmark shows MLLM gaps for UAV wildfire monitoring"]},"model":"grok-4.3","cost_usd":0.009772,"raw_usage":{"total_tokens":4352,"prompt_tokens":671,"num_sources_used":0,"completion_tokens":74,"cost_in_usd_ticks":97724500,"prompt_tokens_details":{"text_tokens":671,"audio_tokens":0,"image_tokens":0,"cached_tokens":256},"completion_tokens_details":{"audio_tokens":0,"reasoning_tokens":3607,"accepted_prediction_tokens":0,"rejected_prediction_tokens":0}},"tokens_in":671,"tokens_out":74,"duration_ms":25541,"temperature":1.0,"reasoning_tokens":3607,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-06-26T04:54:08.933399+00:00","model_set":{"reader":"grok-4.3"},"falsifier":"Independent expert re-labeling of a random subset of images that yields disagreement rates above 10 percent on presence detection or coverage questions.","supporting_citations":[],"review_version":1}