{"id":"b4fada42-628e-4955-b6f9-717ef668f454","arxiv_id":"2501.08131","paper_version":2,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":1,"one_line_summary":"A two-stage 'prompt' pipeline that turns SAR image classifications into text outperforms an end-to-end model for remote sensing visual question answering, and late fusion with optical data gives the best overall accuracy.","lead":"The paper introduces a dataset that pairs Sentinel-1 SAR images with optical RSVQA questions and compares end-to-end versus two-stage prompting pipelines for answering them. It reports that a late-fusion of SAR and optical predictions gives the best accuracy, with SAR helping most on water-related questions.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The BEN-MM-61 classifier split is unspecified, leaving a plausible label-leakage path from the RSVQAxBEN-MM test set into Prompt-RSVQA's textual context, so the headline ~10% gain is not yet established.","rationale":"The reader and I converge on the same load-bearing assumption. The manuscript's central numerical claims are Tables IV and V. Both Prompt-RSVQA and End-to-End are evaluated on RSVQAxBEN-MM, which inherits the geographic split from RSVQAxBEN. But the Prompt-RSVQA context generator is a multi-label classifier trained on BEN-MM-61, and Section VI-A does not state which split of BEN-MM-61 is used. Section IV-A says BEN's own split is random; Section IV-B says RSVQAxBEN's is geographic. Under the literal reading, the classifier was trained on a random 60% of all BEN patches, i.e., on many of the same images that later appear in the VQA test set. Because Prompt-RSVQA turns the classifier's predicted class list into natural-language context for DistilBERT, seeing the test image during classifier training means the model is handed class labels that are strongly correlated with the ground-truth answer. End-to-End enjoys no such channel. This asymmetry, not the two-stage prompting idea, could explain the ~10% SAR-only gain and the late-fusion advantage. The concern is concrete and testable: the authors can specify the split and, if needed, retrain the classifier on the exact geographic train split. No independent verification is currently possible because no code or dataset is released. I therefore agree with the reader's REJECT verdict; the contribution may be salvageable as a dataset paper, but the headline empirical claims are not currently supported.","tokens_in":17870,"tokens_out":4492,"duration_ms":42753,"concrete_test":"Recover or re-run the classifier training with a specified split. Concretely: (1) check the released configuration or ask the authors for the exact split used in Table III; (2) compute the patch overlap between the classifier training set and the RSVQAxBEN-MM test set. If overlap is nonzero, retrain the classifier from scratch on the RSVQAxBEN-MM train split (geographic), freeze it, regenerate the Prompt-RSVQA contexts, and rerun Table V. If the Prompt-RSVQA SAR-only overall accuracy drops from 71.78% toward the End-to-End level (~62%), the leakage path is confirmed and the headline comparison is invalid.","verdict_should_be":"REJECT","load_bearing_attack":"The central comparison in Tables IV and V rests on a classifier whose training split is never given. Section VI-A says only \"We first train image encoders on the BEN-MM-61 classification task.\" Section IV-A states that the original BEN benchmark uses a random 60/20/20 patch split, while Section IV-B states that RSVQAxBEN (and hence RSVQAxBEN-MM, Section IV-C) uses a geographic split, with the easternmost 23% as test. If the BEN-MM-61 classifier was trained on the original random BEN split, then a large fraction of the RSVQAxBEN-MM test patches are in its training set. Prompt-RSVQA converts the classifier's output into textual context; for a test image whose label was seen during classifier training, this context is effectively answer leakage. End-to-End RSVQA is trained only on the VQA train split, so the ~10% overall-accuracy advantage of Prompt-RSVQA on SAR (71.78% vs 62.23%, Table V[d] vs Table IV[j]) and the late-fusion gains could be artifacts. This is not a claim of intent; the manuscript simply does not document the split, and the default reading of \"trained on BEN-MM-61\" is the random split. The concern is load-bearing because the entire novelty claim is the two-stage pipeline's superiority and the fusion result.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces RSVQAxBEN-MM, a dataset pairing Sentinel-1 SAR images with the existing RSVQAxBEN question/answer set, and compares an End-to-End RSVQA pipeline against a two-stage Prompt-RSVQA pipeline that converts a visual classifier's output into textual context for a language model. Experiments cover optical-only, SAR-only, and three fusion schemes (early, halfway, late). The headline claims are that Prompt-RSVQA improves overall accuracy by nearly 10% over End-to-End on SAR-only inputs and that late fusion yields the best VQA accuracy, 75.49% overall with F1-micro 75.00%.","tokens_in":18040,"tokens_out":5738,"duration_ms":56448,"significance":"If the reported results are trustworthy, this is a useful contribution: RSVQAxBEN-MM is a sensible extension of an established benchmark, the paper systematically compares SAR and optical modalities for RSVQA, and the analysis of per-class fusion behavior (e.g., SAR helping water-body classes) is informative. The paper also includes a clear discussion of dataset biases and releases detailed classification and VQA tables. The central quantitative claims, however, rest on an experimental protocol that is incompletely specified (the classifier split is not documented) and on single-seed runs with no error bars, so the magnitude of the reported improvements is not yet established.","major_comments":[{"comment":"The split used to train the BEN-MM-61 image classifier is never specified. Section IV-A states that the original BEN benchmark uses a random 60/20/20 patch split, while Section IV-B defines RSVQAxBEN's geographic split with the easternmost 23% of patches as the test set. Section VI-A only says 'We first train image encoders on the BEN-MM-61 classification task' without saying which split is used. If the classifier follows the standard random BEN split, then a substantial fraction of the RSVQAxBEN-MM test patches (those in the eastern region) will have been seen during classifier training. Since Prompt-RSVQA converts the classifier's predicted class list into the textual context used to answer the question, this creates a direct label-leakage path for the test set, which would inflate the ~10% accuracy gap in Table V[d] versus Table IV[j] and the fusion gains. The authors must state the split used; if it was random, they should retrain the classifier on the RSVQAxBEN-MM training split (or otherwise ensure no overlap) and rerun the Prompt-RSVQA and fusion experiments.","section":"§VI-A, §IV-A, §IV-B"},{"comment":"All experiments are reported as single runs with no standard deviations or significance testing. This is particularly problematic for the claimed best result: the late-fusion overall accuracy of 75.49% (Table V[g]) differs from the optical-only result of 75.40% (Table V[b]) by only 0.09 percentage points, which is well within typical run-to-run variability for deep models. Similarly, the SAR-only versus optical-only differences in Table IV are small. Without multiple seeds (at least 3, preferably 5) with mean and standard deviation, or a statistical test, the superiority of late fusion and the value of SAR as a complementary modality are not convincingly demonstrated.","section":"§VI-B/C, Tables IV–V"},{"comment":"The claim that 'SAR proves especially beneficial for questions related to specific land cover types, such as water areas' is supported primarily by classification F1 variations in Figure 4 and by qualitative examples in Figure 5, but there is no quantitative VQA accuracy breakdown by class or by question type for SAR versus optical versus fusion. The paper should report per-class or per-question-type VQA accuracy (e.g., for questions mentioning water bodies or wetlands) to substantiate this claim, since the overall VQA accuracy differences are small and possibly within noise.","section":"§VII and Figure 4"}],"minor_comments":[{"comment":"The abstract states that 'no study has compared SAR and optical imagery in RSVQA,' but reference [8] (by the same authors) appears to be exactly such a comparison; the distinction (e.g., very-high-resolution SAR vs. Sentinel-1, or question/answer design) should be made explicit to avoid overclaiming novelty.","section":"Abstract/Introduction"},{"comment":"The heading 'D. Dataset evaluation)' contains a stray closing parenthesis; it should read 'D. Dataset evaluation'.","section":"§IV-D heading"},{"comment":"The table caption says 'Classification results on BEN-MM' while the text consistently refers to 'BEN-MM-61'; please unify the naming to avoid confusion about the label set.","section":"Table III and §VI-A"},{"comment":"The sentence 'The results of the 10 experiments are presented in Table IV' is fine, but the description of which rows are unfrozen is only given for ResNet-50 initialized on ImageNet; clarify explicitly which rows (e.g., e and j) correspond to fine-tuned networks.","section":"§VI-B"},{"comment":"The LBscore column is reported but the metric is not defined in Section V; add a definition (or a reference to [41]) in the performance-evaluation section.","section":"Tables IV and V"},{"comment":"The confusion matrix is extremely dense and difficult to read, especially the bottom-right zoom; consider splitting into separate figures or increasing resolution.","section":"Figure 6"},{"comment":"The sentence beginning 'In Table III[a,b] we can verify that optical results lose 1.6%' is grammatically incomplete; it should specify what the 1.6% is relative to (presumably ResNet-152 vs. ResNet-50) and complete the thought.","section":"§VII-A"}],"recommendation":"major_revision","confidential_remarks":"The decisive issue is the unspecified classifier split in §VI-A. If the authors used the standard random BEN split, the headline Prompt-RSVQA gain is likely a leakage artifact. The paper should be given the opportunity to clarify and, if necessary, rerun the experiments. The dataset itself remains a potentially useful resource regardless of the experimental outcome."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a workmanlike empirical study that deserves a serious referee, but not without the authors pinning down a few reporting gaps. The headline finding—Prompt-RSVQA beats End-to-End by ~10 points on SAR—is notable but rests on a comparison of two very different pipelines, and the fusion gain over optical-only is 0.09 points, which is noise without error bars.\n\nWhat's new: the RSVQAxBEN-MM dataset, pairing Sentinel-1 SAR with the existing RSVQAxBEN VQA data, is a useful resource. The paper also gives the first controlled SAR-vs-optical comparison on matched patches and a systematic study of early/halfway/late fusion for this task. The finding that SAR is complementary for water bodies and that late fusion works best (with the fewest trainable parameters) is a legitimate empirical result.\n\nThe reader's main worry is the unspecified split for the BEN-MM-61 classifier. On reading the paper, that concern is weaker than it first looks: Section VII-A explicitly says the training set and test set are geographically separated, which only makes sense if the classifier was trained on the RSVQAxBEN geographic split, not the original random BEN split. But the paper should have said so in Section VI-A; the ambiguity is fair to flag and needs a one-line fix.\n\nThe bigger problems are the missing error bars and the absence of code/data release. All runs are single-seed, and the entire case for late fusion over optical-only rests on 0.09 percentage points. The classification threshold for converting class probabilities to text is also never given. The two-stage design means the language model only sees class names, so the comparison to End-to-End is a comparison of two philosophies, not just two backbones. These are not fatal, but they are exactly the things a referee should press on.\n\nBottom line: worth engaging with. The dataset and the fusion comparison are useful to the RSVQA niche. I'd send it to review, with a request for split specification, error bars, threshold value, and a data link.","headline":"The split-leakage worry is overstated—the paper says train/test are geographically separated—but the missing split spec, no error bars, and no data release still make this a major-revision rather than a clean accept.","tokens_in":18671,"tokens_out":4793,"would_cite":true,"duration_ms":43916,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"By turning SAR pixels into a short text list of land-cover classes, a two-stage pipeline answers remote-sensing questions about radar imagery almost 10 percentage points better than an end-to-end model, and late fusion with optical…","keywords":["Deep Learning","Remote Sensing","Visual Question Answering","Multi-Modality","Natural Language Processing","Synthetic Aperture Radar","data fusion","land cover classification"],"falsifier":"Retrain the BEN-MM-61 classifier on a strict geographic split (western-Europe patches for training, eastern-Europe for testing, mirroring RSVQAxBEN), then rerun Prompt-RSVQA with SAR, optical, and late fusion on the VQA test set. If the ~10-point SAR-only gain over end-to-end and the late-fusion lead shrink materially, the mismatch between the classifier's training split and the VQA test split is the cause.","tokens_in":17583,"feed_emoji":"🛰️","tokens_out":8834,"duration_ms":82103,"temperature":0.7,"pith_summary":"Remote-sensing visual question answering (RSVQA) has mostly been an optical-image task; this paper argues that synthetic aperture radar (SAR) can be brought in productively. It introduces RSVQAxBEN-MM, a benchmark that pairs Sentinel-1 radar patches (VV, VH, and a normalized VV/VH ratio) with the existing Sentinel-2 optical questions of RSVQAxBEN, and it studies two pipelines on it. The central claim is that a two-stage pipeline—first classifying the image into a text list of land-cover classes, then letting a language model answer from that list—works far better with SAR than an end-to-end model, gaining nearly 10 percentage points in overall accuracy. The paper further claims that fusing SAR and optical at the decision level, by combining the two classifiers' per-class scores, gives the best results: 75.49% overall accuracy on the VQA test set, with the late-fusion classifier at 75.00% F1-micro. If true, this is a practical route to all-weather, interpretable land-cover question answering, with SAR carrying information optical images miss, especially over water.","feed_headline":"Late fusion of SAR and optical hits 75.49 percent in land-cover Q&A","feed_subtitle":"SAR alone gains ~10 points, and late fusion with optical tops 75 percent accuracy.","key_machinery":"The load-bearing mechanism is the text bottleneck: the Prompt-RSVQA context generated by thresholding multi-label classification scores. By converting image content into a list of class names, it aligns radar-derived information with the language model's pretrained word space, which is why the paper sees SAR-only gains over end-to-end and why late fusion of class-score vectors outperforms feature-level fusion.","core_discovery":"On its own terms, the paper's claim is that SAR imagery can be brought into RSVQA not by feeding radar pixels through one monolithic network, but by a two-stage translation: a multi-label land-cover classifier turns the SAR patch into a short text list (for example 'Forest, Water bodies'), and a language model answers the question from that list plus the question text. The paper constructs RSVQAxBEN-MM, pairing each Sentinel-2 RGB patch and its existing RSVQAxBEN question-answer triplets with the matching Sentinel-1 dual-polarization patch including a normalized VV/VH ratio channel. In experiments, this Prompt-RSVQA pipeline with SAR alone reaches 71.78% overall accuracy, nearly 10 percentage points above the best end-to-end SAR result, and the best fusion—late fusion, which concatenates the separate optical and SAR class-score vectors through a small MLP—reaches 75.49% VQA accuracy and 75.00% F1-micro on the classification step. The paper also argues SAR contributes specifically to water-body classes, and that the two-stage design makes errors inspectable through the intermediate class list.","pith_inferences":["Because the paper never states the train/test split used to fit the BEN-MM-61 classifier, the cleanest way to stress-test the headline is to retrain that classifier on the same geographic split as RSVQAxBEN and watch whether the ~10-point SAR gain survives; this is the immediate next experiment.","The text-bottleneck design can express only what the fixed land-cover vocabulary names; spatial relations, counts, and object locations would require object-level or region-level context generation, which the current dataset does not exercise.","The fusion ranking is probably class-frequency-dependent: late fusion wins on the frequent and water classes that dominate questions, so a more balanced question distribution could reverse the ranking—the paper itself acknowledges the winner is dataset-dependent.","A direct transfer test to other dual-polarization SAR sensors, or to cloud-covered optical scenes, would show whether the VV/VH/ratio recipe generalizes beyond Sentinel-1."],"forward_implications":["SAR-only two-stage RSVQA reaches 71.78% overall accuracy, roughly 10 points over end-to-end SAR, so SAR can support land-cover question answering without optical imagery.","Decision-level fusion of independently trained SAR and optical classifiers yields the best VQA accuracy (75.49%) and best classification F1-micro (75.00%) on the proposed dataset.","Fusion gains are class-dependent: late and halfway fusion improve water-body predictions, while wetlands remain poorly predicted, so SAR's complementary value is not uniform across land-cover types.","The two-stage pipeline's intermediate class text makes answer errors traceable to misclassified land-cover classes, supporting error analysis.","Because the proposed dataset uses an east-west geographic split, the reported numbers are meant to measure geographic generalization rather than in-distribution memorization."],"supporting_citations":[{"why":"Introduces the RSVQA task, datasets, and the end-to-end baseline this paper extends to SAR.","marker":"[5]"},{"why":"Introduces the two-stage Prompt-RSVQA architecture that the paper adapts for SAR input.","marker":"[7]"},{"why":"Prior SAR-plus-RSVQA land-cover study that contributes the VV/VH ratio channel and wetland findings.","marker":"[8]"},{"why":"BigEarthNet, which supplies the CLC land-cover labels and the original random split that underpin the dataset.","marker":"[37]"},{"why":"BigEarthNet-MM, which provides the aligned Sentinel-1 VV/VH image pairs used as the SAR modality.","marker":"[38]"},{"why":"Decision-level fusion precedent for classification that frames the late-fusion design.","marker":"[31]"},{"why":"Provides the Prior, Uniform, and LBscore bias metrics used to characterize the answer distribution of RSVQAxBEN-MM.","marker":"[41]"},{"why":"Explains why wetland classes remain hard (HH polarization absent, less structured vegetation), contextualizing the fusion failures.","marker":"[43]"}],"fun_headline_variants":["Late fusion of SAR and optical hits 75.49% in land-cover Q&A","Radar-to-text step lifts satellite Q&A accuracy by 10 points","Two-stage SAR pipeline beats end-to-end by 10% in RSVQA","SAR shines on water classes: late fusion tops 75% in RSVQA","From patch to prose: a two-stage route to better SAR answers"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The paper never states which train/test split is used to train the image classifier behind Prompt-RSVQA, and the original BigEarthNet split is random while the VQA test split is geographic; if the classifier saw the VQA test patches during training, the reported gains are inflated.","fun_headline_variants_meta":{"raw":{"variants":["Late fusion of SAR and optical hits 75.49% in land-cover Q&A","Radar-to-text step lifts satellite Q&A accuracy by 10 points","Two-stage SAR pipeline beats end-to-end by 10% in RSVQA","SAR shines on water classes: late fusion tops 75% in RSVQA","From patch to prose: a two-stage route to better SAR answers"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.00043,"raw_usage":{"total_tokens":2245,"prompt_tokens":1044,"completion_tokens":1201,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":660,"completion_tokens_details":{"reasoning_tokens":1098}},"tokens_in":660,"tokens_out":1201,"duration_ms":10569,"temperature":1.0,"reasoning_tokens":1098,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T20:30:18.399614+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Retrain the BEN-MM-61 classifier on a strict geographic split (western-Europe patches for training, eastern-Europe for testing, mirroring RSVQAxBEN), then rerun Prompt-RSVQA with SAR, optical, and late fusion on the VQA test set. If the ~10-point SAR-only gain over end-to-end and the late-fusion lead shrink materially, the mismatch between the classifier's training split and the VQA test split is the cause.","supporting_citations":[{"cited_title":"RSVQA: Visual question answering for remote sensing data,","cited_arxiv_id":null,"evidence_quote":"Introduces the RSVQA task, datasets, and the end-to-end baseline this paper extends to SAR."},{"cited_title":"Prompt- RSVQA: Prompting visual context to a language model for remote sensing visual question answering,","cited_arxiv_id":null,"evidence_quote":"Introduces the two-stage Prompt-RSVQA architecture that the paper adapts for SAR input."},{"cited_title":"Can SAR improve RSVQA performance?","cited_arxiv_id":null,"evidence_quote":"Prior SAR-plus-RSVQA land-cover study that contributes the VV/VH ratio channel and wetland findings."},{"cited_title":"BigEarthNet: A large-scale benchmark archive for remote sensing image understanding [dataset],","cited_arxiv_id":null,"evidence_quote":"BigEarthNet, which supplies the CLC land-cover labels and the original random split that underpin the dataset."},{"cited_title":"BigEarthNet-MM: A large- scale, multimodal, multilabel benchmark archive for remote sensing image classification and retrieval [dataset],","cited_arxiv_id":null,"evidence_quote":"BigEarthNet-MM, which provides the aligned Sentinel-1 VV/VH image pairs used as the SAR modality."},{"cited_title":"Decision fusion for the classification of hyperspectral data: Outcome of the 2008 GRS-S data fusion contest,","cited_arxiv_id":null,"evidence_quote":"Decision-level fusion precedent for classification that frames the late-fusion design."},{"cited_title":"Wetland monitoring using sar data: A meta- analysis and comprehensive review,","cited_arxiv_id":null,"evidence_quote":"Explains why wetland classes remain hard (HH polarization absent, less structured vegetation), contextualizing the fusion failures."}],"review_version":1}