{"id":"1f4ae93f-ee65-4f0c-827a-d76defb077ab","arxiv_id":"2411.15770","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An optical-SAR visual question answering dataset with 6,008 image pairs and 1,036,694 questions is introduced, together with a text-guided fusion network that outperforms baselines on that dataset.","lead":"This paper introduces the OSVQA benchmark, a dataset pairing optical and SAR satellite images with 1,036,694 question-answer pairs, and proposes TGFNet, a network that fuses both image types for visual question answering. The model beats older baselines on its own benchmark, but the reported gains are small and the evaluation has several caveats.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Headline metrics exclude the 'fog-dark', 'quality', and 'match' categories, so the claimed robustness improvement under cloud/low-light conditions is never actually measured on the categories designed to test it.","rationale":"The paper's central contribution is a benchmark for optical-SAR RSVQA under cloud/low-light conditions plus a fusion model that significantly improves robustness. For that claim to hold, the evaluation must measure performance on the scenarios and categories the benchmark is designed to stress. Section 5.1 excludes three categories from the reported OA/AA: 'quality', 'fog-dark', and 'match'. 'fog-dark' is the category most directly tied to the motivating conditions; 'quality' is the paper's unique modality-assessment category. Their exclusion means the reported 71.89/65.12 numbers do not quantify the paper's key claimed ability. The reader's dataset-validity concern is real but broader; the exclusion is a sharper, internal inconsistency that can be fixed by reporting per-category numbers. I do not regard this as grounds for rejection, since the authors may have excluded these categories to keep single-modality baselines meaningful, but the burden is on them to show the excluded categories do not overturn the conclusion. Single-run, small-margin results compound the issue. A full-category evaluation and multi-seed confidence intervals would settle it.","tokens_in":21478,"tokens_out":5829,"duration_ms":56170,"concrete_test":"Recompute headline OA/AA over the remaining categories plus 'fog-dark' separately, and report per-category accuracy on 'fog-dark', 'quality', and 'match' for all MUL methods (and for OPT methods on 'fog-dark'). If TGFNet's 'fog-dark' accuracy is not above the best optical-only baseline, or its 'quality' accuracy is near chance, the central robustness claim fails. Additionally, run five seeds with the same protocol and report mean +/- std; if the Bi-Modal MUL vs TGFNet AA gap confidence interval crosses zero, the word 'significantly' should be dropped.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 5.1 states that OA/AA exclude 'quality', 'fog-dark', and 'match' because these categories 'involve both optical and SAR images'. This exclusion removes exactly the categories that test the paper's central robustness claim. 'fog-dark' asks whether clouds or darkness are present, an optical-only model can answer it, so the stated fairness rationale does not justify dropping it. 'quality' is the paper's self-described unique modality-assessment category, yet no per-category accuracy for it appears anywhere in Table 3. Consequently, the headline improvements (AA 71.89 vs 69.97, OA 65.12 vs 64.15 over Bi-Modal MUL) are computed on the remaining 13 categories and provide no evidence that TGFNet is more robust in fog/dark scenes or that it performs the quality-aware reasoning the dataset was designed to elicit. The problem is compounded by single-run results without error bars or significance tests, and by the MUL baselines, which are constructed by a simple addition fusion of a second image encoder, so the comparison may partly measure fusion capacity rather than the proposed architecture. Table 4's ablation margins are also small (OA +1.51, AA +2.67 from Exp1 to Exp4) and unreplicated. Because the paper's strongest claim is about robustness under adverse conditions, omitting the categories that directly probe those conditions leaves the central claim unverified rather than merely under-powered.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes TGFNet, a text-guided coarse-to-fine fusion network for remote sensing visual question answering (RSVQA) that combines optical and synthetic aperture radar (SAR) imagery. The method consists of a Text-guided Coarse-to-Fine Attention Refinement (CFAR) module, which routes question-relevant image regions and enhances them through cross-attention, and an Adaptive Multi-Expert Fusion (AMEF) module, which combines predictions from optical, SAR, and fused-image experts via learned weights. The authors also introduce OSVQA, a new dataset of 6,008 optical-SAR image pairs and 1,036,694 question-answer pairs across 16 question types, including a \"quality\" category intended to assess which modality is more informative. Experiments on OSVQA compare TGFNet with six existing RSVQA methods and include ablations of the proposed modules and fusion strategies. The reported headline results are an average accuracy (AA) of 71.89% and overall accuracy (OA) of 65.12%, which the authors state outperform all baselines. The central claims are that text-guided optical-SAR fusion improves RSVQA under cloud-covered and low-light conditions, and that OSVQA is the first large-scale benchmark for this task.","tokens_in":21834,"tokens_out":5374,"duration_ms":50427,"significance":"If the claims are substantiated, the paper would make a useful contribution: OSVQA is, to my knowledge, the first large-scale optical-SAR RSVQA benchmark, and the proposed architecture is a reasonable instantiation of text-guided, multi-expert fusion. The authors have made the dataset publicly announced (though the URL in the paper is incomplete), and the method description is sufficiently detailed to be reimplemented. The CFAR module's coarse-to-fine routing and the AMEF module's adaptive expert weighting are sensible design choices, and the ablation structure in Table 4 gives some credit to each component. However, the experimental evidence as presented is under-powered: all numbers come from single runs without error bars or significance tests, the gains over the strongest baseline are modest (1.92% AA and 0.97% OA), and the reported metrics exclude three of the sixteen question types, including the categories most directly tied to the paper's stated novelty and robustness claims. The benchmark's validity also depends on the semi-automatic annotation pipeline and the artificial fog/darkness simulation, which are not yet described with enough detail or external validation.","major_comments":[{"comment":"The headline OA and AA are computed after excluding the 'quality', 'fog-dark', and 'match' question categories, with the stated rationale that these categories involve both optical and SAR images. This rationale does not apply to 'fog-dark', which asks about the presence of clouds or darkness in the optical image and can be answered by an optical-only model. More importantly, the excluded 'quality' category is the dataset's self-described unique modality-assessment category (Section 4.2), and 'fog-dark' directly probes the adverse conditions that motivate the paper. Since no per-category accuracy for any of the three excluded types is reported anywhere in Table 3 or the text, the claimed improvement in robustness under cloud-covered and low-light conditions is not actually measured by the reported numbers. Please report per-category accuracies for all 16 question types, and for 'match' either define a protocol that is fair to single-modality models or report that category separately for multi-modal models only.","section":"Section 5.1, Table 3"},{"comment":"All quantitative results appear to come from a single training run; no standard deviations, confidence intervals, or significance tests are reported. The headline gains over the strongest baseline are 1.92% AA and 0.97% OA, and the ablation margins in Table 4 range from 0.34% to 0.63% OA and 0.03% to 2.15% AA, which are small relative to typical seed variance in VQA training. Please rerun all models with at least three (preferably five) random seeds, report mean and standard deviation, and include a paired significance test (e.g., bootstrap or Wilcoxon signed-rank) for the main comparisons between TGFNet and the best baseline and between the ablation variants.","section":"Section 5.2, Tables 3-5"},{"comment":"OSVQA is constructed entirely by the authors, partly from their own earlier OGSOD-1.0 dataset, using manual attribute annotation followed by template-based question generation, with artificial fog and darkness added post-hoc. No inter-annotator agreement, external human evaluation, or quantitative description of the fog/dark simulation (e.g., degradation type, parameters, and whether answers were re-verified after modification) is provided. These details are needed to establish that the benchmark measures multi-modal reasoning rather than annotation artifacts or linguistic shortcuts in the templates. Please report the simulation protocol, the annotation quality controls, and at least a small-scale human evaluation or internal consistency check on a sample of questions.","section":"Section 4.2"},{"comment":"For the MUL setting, all single-modality baselines are adapted by adding a second identical image encoder and fusing optical and SAR features with a simple addition operation, while TGFNet uses the full AMEF machinery with three experts, a question-guided fusion module (RQAF), and adaptive weighting. This comparison conflates architectural capacity with the proposed fusion mechanism. Table 5 partially addresses this by comparing fusion strategies within a fixed backbone, but the main benchmark should also include a stronger fusion baseline (e.g., cross-attention or transformer-based fusion of optical/SAR features) so that the reported improvements can be attributed to the adaptive multi-expert design rather than to the absence of any learned fusion in the baseline.","section":"Section 5.1, Table 3"}],"minor_comments":[{"comment":"Equation (10) contains a typographical error: the term before lambda_4 reads '+ +' instead of a single '+'. Please correct it.","section":"Equation (10)"},{"comment":"The column header 'Precence' is misspelled and should be 'Presence'; the abbreviations 'Locat.' and 'Num.' should be defined in the table caption.","section":"Table 3"},{"comment":"The number of answer classes C in Equation (7) is not specified. Please state the value used for OSVQA (the paper reports 140 distinct answers in Section 4.3) and explain how the classification head handles the full answer vocabulary.","section":"Section 3.3.2, Equation (7)"},{"comment":"The notation in Equation (1) is ambiguous: the Mean is said to be taken along dimension 1, but the dimensions of W_T, W_I, and the resulting score tensor S are not made explicit. Please clarify the exact shapes and the role of the mean over the sequence dimension.","section":"Section 3.2.1, Equation (1)"},{"comment":"The dataset URL in the abstract is just 'https://github.com/mmic-lcl/' with no repository path. Also, the abstract describes the image pairs as 'well-aligned', while Section 4.2 says the selected pairs are 'approximately aligned'. Please provide the full URL and specify the alignment procedure.","section":"Abstract and Section 4.2"},{"comment":"The sentence 'the input questions are embedded into 71×512 vectors' is unclear: please specify how the maximum length of 71 is determined, how padding/truncation is handled, and which tokenizer is used for the CLIP text encoder.","section":"Section 5.1"},{"comment":"The caption of Figure 4 contains untranslated Chinese text ('MCA 和IE的⽹络结构。...'). This should be translated or removed for publication.","section":"Figure 4"},{"comment":"The loss weights lambda_1 through lambda_4 are described as 'regularization parameters', which is imprecise; they are weighting coefficients for the four cross-entropy losses. In addition, no sensitivity analysis is given for the chosen value of 0.5.","section":"Section 3.4"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is a new benchmark plus a model, so the editor may want to verify that the dataset is actually released and that the benchmark can be independently used. The authors rely heavily on their own earlier OGSOD-1.0 dataset, and the manuscript would be strengthened by an independent evaluation or at least by detailed dataset statistics and annotation-quality checks. I do not see evidence of intentional misreporting, but the exclusion of the 'fog-dark' and 'quality' categories from the headline metrics is a serious evaluation-protocol issue that must be fixed in revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe one thing to know: this paper gives the field a genuinely new resource — OSVQA, the first optical-SAR benchmark for RSVQA, with 6,008 aligned image pairs and over a million QA pairs across 16 question types, including a unique 'quality' type that asks which modality is more informative. The TGFNet architecture is a reasonable recombination of known ideas — text-guided region routing, cross-attention, multi-expert fusion, adaptive weighting — and it is described clearly with proper ablations. That part is solid.\n\nThe soft spot is in the evaluation, and it is load-bearing. The headline OA/AA numbers exclude three question types: 'quality', 'fog-dark', and 'match', with the stated rationale that they involve both optical and SAR images. But 'fog-dark' simply asks whether the scene is cloudy or dark; an optical-only model can answer that. And 'quality' is the paper's own unique modality-assessment category, yet no per-category accuracy for it appears anywhere in the results. So the claimed robustness improvement under cloud and low-light conditions is never actually measured on the categories designed to test it. The reported gains are also modest — 1.92% AA and 0.97% OA over the best baseline — and they come from single runs without error bars or significance tests. The ablation margins are similarly small. None of this makes the dataset worthless; it just means the headline claim is unverified, not merely under-powered.\n\nThe dataset is self-created, partly from the authors' own OGSOD-1.0, with template-generated questions and simulated fog/darkness. The split is clean and the annotation process is transparent, so this is not fatal, but independent validation would help. Also, the provided dataset link is only a group GitHub page, not a direct release.\n\nIf you work on RSVQA or multi-source fusion, read this for the dataset and the discussion of modality complementarity. I would send it to peer review, but the authors must report all 16 categories, add error bars, and ideally break down performance by image condition. As it stands, the central robustness claim does not hold up until they do.\n\nRecommendation: serious referee, major revision.","headline":"Useful new optical-SAR RSVQA benchmark and a reasonable fusion net, but the evaluation hides the categories that would actually test the robustness claim.","tokens_in":22318,"tokens_out":4391,"would_cite":true,"duration_ms":36131,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Question-guided fusion of optical and radar satellite images keeps remote-sensing visual question answering accurate in cloud-covered and low-light scenes, with the best results on a new benchmark.","keywords":["remote sensing visual question answering","SAR-optical fusion","multi-source image fusion","text-guided attention","coarse-to-fine attention","multi-expert fusion","OSVQA dataset","adverse imaging conditions"],"falsifier":"Train a text-only model that sees the question but no images on OSVQA; if it reaches a large share of TGFNet's reported accuracy, then the fusion advantage is substantially explained by answer priors in the template questions rather than by optical-SAR reasoning.","tokens_in":21293,"feed_emoji":"🛰️","tokens_out":9120,"duration_ms":76817,"temperature":0.7,"pith_summary":"This paper tries to show that visual question answering on remote sensing imagery can be made robust to cloud cover and low light by fusing optical images with synthetic aperture radar (SAR) images, using the question text to steer the fusion. To test this, the authors build OSVQA, described as the first large-scale optical-SAR benchmark, with 6,008 aligned image pairs and 1,036,694 question-answer pairs spanning 16 question types, and propose TGFNet, a network that first routes attention from broad regions to fine question-relevant details and then adaptively combines predictions from optical, SAR, and fused-image experts. On this benchmark TGFNet reports the best accuracy, with overall accuracy of 71.89% and average accuracy of 65.12%, surpassing all compared methods. A reader should care because real-world satellite question answering must work when optical images fail.","feed_headline":"Radar + optical fusion pushes satellite VQA past clouds and darkness","feed_subtitle":"A new benchmark plus a question-guided radar/optical network keeps answers accurate under clouds and low light.","key_machinery":"The load-bearing machinery is a question-conditioned routing and fusion pipeline built on fine-tuned CLIP encoders. CFAR splits each image into regions, scores every region against the embedded question, selects the top-k regions, applies multi-head cross-attention between question tokens and those regions, and then enhances the full image features by similarity-weighted attention; this implements the 'coarse-to-fine' idea of first choosing broad areas and then refining detail. AMEF then combines three experts: an optical expert, a SAR expert, and a fusion expert whose input is produced by Regional Quality-Aware Fusion (RQAF), which for each spatial location selects the top patches from both modalities and uses question-guided softmax weights to blend them. A small adaptive-fusion network learns per-expert weights from their predictions, and the final answer is the softmax of that weighted sum. Each expert uses a two-layer transformer decoder plus a classification MLP, and all experts are trained jointly with cross-entropy losses while only the adaptively fused output is used at inference.","core_discovery":"On its own terms, the paper's central claim is that text-guided, coarse-to-fine, adaptive fusion of optical and SAR imagery substantially improves remote sensing visual question answering under adverse conditions. The TGFNet architecture is built around two mechanisms: the Text-guided Coarse-to-Fine Attention Refinement (CFAR) module, which scores image regions against the question, keeps the top-k regions, and refines them with cross-attention and similarity enhancement; and the Adaptive Multi-Expert Fusion (AMEF) module, which learns patch-level quality-aware fusion plus an adaptive weighted combination of an optical expert, a SAR expert, and a fusion expert. Compared with the strongest baseline, TGFNet raises overall accuracy from 69.97% to 71.89% and average accuracy from 64.15% to 65.12% on OSVQA. The paper also reports that each added module contributes: AMEF alone adds 0.34 overall accuracy and 2.15 average accuracy, CFAR adds 0.63 and 0.03, and RQAF adds 0.54 and 0.49. Because the evaluation excludes the three question types that explicitly ask about modality quality ('quality', 'fog-dark', 'match'), the reported gains describe fusion benefit apart from questions that literally name the data sources.","pith_inferences":["Editorial inference: because the benchmark's questions are template-generated and the optical degradations are partly artificial, the absolute accuracies may reflect the dataset's linguistic regularities; a text-only model trained on the same questions would reveal how much of the reported accuracy comes from language priors rather than image fusion.","Editorial inference: the excluded 'quality' question type is arguably the most direct test of whether the model has learned to trust the right sensor; including it in the headline metric, or using the model's quality answers to predict its fusion weights, would expose whether the adaptive fusion is genuinely semantic or just a learned re-weighting.","Editorial inference: the coarse-to-fine routing idea transfers naturally to other sensor-fusion tasks where one modality degrades, such as cloud-inpainting, change detection, or disaster mapping; the same question-conditioned routing could be applied there."],"forward_implications":["If TGFNet's results hold, question-guided optical-SAR fusion is a viable route to robust remote sensing VQA: it beats the best compared method on OSVQA and beats every single-modality input.","The ablations imply that each of the three design pieces pays off: adaptive multi-expert fusion, text-guided coarse-to-fine attention, and region-level quality-aware fusion each improve accuracy when added.","The OSVQA dataset gives the field a shared benchmark with 16 question types, over a million question-answer pairs, and aligned optical-SAR pairs, including modality-quality and relational-reasoning questions that previous RSVQA datasets lack.","Fusion generally helps, but the paper's comparisons show that naive addition of features is a weak fusion: on OSVQA, addition and concatenation lag transformer-based fusion, and TGFNet's adaptively weighted experts outperform all of them."],"supporting_citations":[{"why":"Supplies the frozen CLIP encoders that turn questions, optical images, and SAR images into the shared feature space on which all routing and fusion is built.","marker":"[33]"},{"why":"RSVQA is the foundational optical-only RSVQA method and dataset; it serves as a primary baseline in the comparison.","marker":"[6]"},{"why":"MAIN is a mutual-attention RSVQA baseline whose bilinear fusion approach is compared in the benchmark.","marker":"[7]"},{"why":"HRVQA is the high-resolution aerial-image VQA baseline used in the comparison.","marker":"[8]"},{"why":"Bi-Modal is the strongest competing method and supplies the CLIP-based transformer baseline for multi-modal input.","marker":"[11]"},{"why":"FETH provides the curriculum-learning RSVQA baseline that is compared in the benchmark.","marker":"[12]"},{"why":"TRAR supplies a dynamic attention-routing baseline for visual transformers in VQA.","marker":"[43]"},{"why":"RSAdapter supplies an adapter-based multimodal baseline for remote sensing VQA.","marker":"[44]"},{"why":"Provides the QXS-SAROPT aligned optical-SAR image pairs used to build part of OSVQA.","marker":"[42]"},{"why":"Provides the OGSOD-1.0 optical-SAR pairs used to build part of OSVQA.","marker":"[17]"}],"fun_headline_variants":["Question-guided optical-SAR fusion lifts satellite VQA past bad weather","Coarse-to-fine fusion network sharpens remote sensing VQA in cloudy scenes","New optical-SAR dataset plus adaptive fusion improves satellite question answering","Text-guided radar-optical fusion improves accuracy for all-weather satellite Q&A","Adaptive fusion of radar and optical images enhances satellite VQA in darkness"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the OSVQA dataset, with its template-generated questions, artificially degraded optical images, and author-provided annotations, captures genuine optical-SAR visual question answering ability rather than letting models exploit linguistic shortcuts or annotation biases.","fun_headline_variants_meta":{"raw":{"variants":["Question-guided optical-SAR fusion lifts satellite VQA past bad weather","Coarse-to-fine fusion network sharpens remote sensing VQA in cloudy scenes","New optical-SAR dataset plus adaptive fusion improves satellite question answering","Text-guided radar-optical fusion improves accuracy for all-weather satellite Q&A","Adaptive fusion of radar and optical images enhances satellite VQA in darkness"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.001052,"raw_usage":{"total_tokens":4516,"prompt_tokens":1142,"completion_tokens":3374,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":758,"completion_tokens_details":{"reasoning_tokens":3278}},"tokens_in":758,"tokens_out":3374,"duration_ms":23031,"temperature":1.0,"reasoning_tokens":3278,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T13:54:29.036138+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train a text-only model that sees the question but no images on OSVQA; if it reaches a large share of TGFNet's reported accuracy, then the fusion advantage is substantially explained by answer priors in the template questions rather than by optical-SAR reasoning.","supporting_citations":[{"cited_title":"Rsvqa: Visual question answering for remote sensing data.IEEE Transactions on Geoscience and Remote Sensing, 58:8555–8566, 2020","cited_arxiv_id":null,"evidence_quote":"RSVQA is the foundational optical-only RSVQA method and dataset; it serves as a primary baseline in the comparison."},{"cited_title":"Mu- tual attention inception network for remote sensing visual question an- swering","cited_arxiv_id":null,"evidence_quote":"MAIN is a mutual-attention RSVQA baseline whose bilinear fusion approach is compared in the benchmark."},{"cited_title":"Hrvqa: A visual questionansweringbenchmarkforhigh-resolutionaerialimages","cited_arxiv_id":null,"evidence_quote":"HRVQA is the high-resolution aerial-image VQA baseline used in the comparison."},{"cited_title":"Bi-modal transformer-based approach for visual question answering in remote sensing imagery","cited_arxiv_id":null,"evidence_quote":"Bi-Modal is the strongest competing method and supplies the CLIP-based transformer baseline for multi-modal input."},{"cited_title":"From easy to hard: Learning language-guided curriculum for visual question answering on remote sensing data.IEEE Transactions on Geoscience and Remote Sensing, 60:1–11, 2022","cited_arxiv_id":null,"evidence_quote":"FETH provides the curriculum-learning RSVQA baseline that is compared in the benchmark."},{"cited_title":"Trar: Routingtheatten- tion spans in transformer for visual question answering","cited_arxiv_id":null,"evidence_quote":"TRAR supplies a dynamic attention-routing baseline for visual transformers in VQA."},{"cited_title":"Rsadapter: Adapting multimodal modelsforremotesensingvisualquestionanswering","cited_arxiv_id":null,"evidence_quote":"RSAdapter supplies an adapter-based multimodal baseline for remote sensing VQA."},{"cited_title":"Category-oriented localization distillation for sar object detection and a unified benchmark","cited_arxiv_id":null,"evidence_quote":"Provides the OGSOD-1.0 optical-SAR pairs used to build part of OSVQA."}],"review_version":1}