{"id":"b87e52b8-a265-4f76-ac70-606c81947789","arxiv_id":"2412.16583","paper_version":1,"verdict":"REJECT","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"high","formal_verification":"none","parameter_count":2,"one_line_summary":"A vision-language model for Earth observation that adds numeric biomass regression to generative question answering, with R^2 up to 0.36 and patch counting still failing.","lead":"This paper creates a 1.6 million sample satellite image-and-text dataset for predicting above-ground biomass, and a vision-language model with a separate regression head. If it works, an Earth observation chatbot could also estimate environmental quantities, but the reported accuracy is modest and below a dedicated baseline.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The AGB text annotations in REO-Instruct contain the regression target, so the reported R^2 gain may come from the LLM hidden states memorizing the numeric answer rather than from scientific reasoning; the paper lacks the control needed to rule this out.","rationale":"The reader's weakest assumption identifies the same load-bearing issue: the LLM is fine-tuned on text that contains the regression target, creating a plausible leakage path through the reverse projection. I agree with that assessment and with the REJECT verdict, so no verdict change is needed. The concern is not a disagreement with the field's consensus; it is an internal control problem. The paper's own Section 4.2 says the generation head learns domain knowledge during training and passes it through R-Proj, so the causal mechanism claimed for the R^2 gain is exactly what the missing control would test. I also credit the dataset scale (1.6M pairs), the explicit separation of generation and regression heads, and the honest reporting that Niconet remains better on AGB RMSE and that patch counting yields negative R^2. These merits do not remove the leakage confound, because no experiment distinguishes 'read the numeric label out of the LLM hidden state' from 'integrate scientific knowledge.' The zero-shot baseline comparisons in Tables 2-4 are also uninformative for superiority claims, but the fine-tuned LLaVA comparison is the fair one, and the leakage control is what would make it interpretable. With no code, data, or model weights released, the central claim is not currently well supported; a targeted re-run could settle it.","tokens_in":13927,"tokens_out":6717,"duration_ms":55817,"concrete_test":"Retrain REO-VLM on REO-Instruct with the same architecture and two-stage protocol, but delete the AGB numerical values from every text annotation used in stage one, leaving all other land-cover, patch-count, and VQA annotations intact and keeping the AGB target in stage two's MSE loss. Evaluate on the same test split. If the MS R^2 falls from 0.36 toward the fine-tuned LLaVA baseline, the Table 4 result is partly label leakage; if R^2 is essentially unchanged, the hidden states carry independent information and the leakage concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that REO-VLM 'integrates scientific regression and generative capabilities' and that its positive AGB R^2 (0.36 for MS, Table 4) proves it 'captures underlying numerical patterns' rests on the reverse projection mechanism (§4.1) passing knowledge from the LLM to the regression head. The weakest step is the controlled source of that knowledge. REO-Instruct's text annotations explicitly include 'quantitative ground-truth estimates of the Above-Ground Biomass' (§3.4), and stage one fine-tunes the LLM on those text annotations with cross-entropy loss (§4.2). The reverse projection then reads 4096-d LLM hidden states and projects them into the visual-feature space consumed by the regression head (§4.1). If those hidden states already encode the AGB value that the LLM was trained to generate, the regression head is reading a memorized label out of the language model, not integrating scientific domain knowledge. The paper reports no ablation in which the reverse projection is removed, no training run in which AGB values are withheld from the text, and no experiment showing that hidden-state content beyond a label copy matters. The improvement over the fine-tuned LLaVA baseline is therefore confounded: it may simply be a more direct readout of information already present in the LLM's next-token supervision. This is load-bearing because the knowledge-driven aspect of REO-VLM is the paper's stated novelty; without it, the contribution reduces to a regression head attached to a VLM, and the patch-counting result is itself negative (R^2 = -0.12 to -0.43, Table 3).","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces REO-Instruct, a large Earth-Observation benchmark with 1.6 million multimodal (RGB, multispectral, SAR) image-text pairs spanning four tasks: land cover classification, ecological patch counting, VQA-based human activity monitoring, and above-ground biomass (AGB) regression. It then proposes REO-VLM, a LLaVA-1.5-based architecture with a dedicated regression head, a reverse projection module that maps LLM hidden states back into the visual feature space, and a two-stage training strategy: stage one fine-tunes the LLM on text annotations with cross-entropy loss, and stage two trains the regression head and reverse projection with MSE loss. On the REO-Instruct test split, the paper reports R²=0.36 for AGB regression with multispectral input, 19.94% overall accuracy on land cover classification, 80.50% on VQA, and negative R² for patch counting. The central claim is that the reverse projection mechanism lets the model exploit scientific domain knowledge stored in the LLM to improve regression accuracy.","tokens_in":14144,"tokens_out":5087,"duration_ms":43353,"significance":"If validated, the paper would make a useful contribution: REO-Instruct appears to be one of the largest EO instruction datasets that explicitly pairs regression targets with rich text annotations, and the two-stage separation of generation and regression training is a sensible way to address conflicting optimization objectives. The paper is also candid about the patch-counting failure, which is a point in its favor. However, the central scientific claim—that language-driven scientific knowledge, rather than memorization of the regression target, is what improves AGB prediction—is not established by the reported experiments. The comparison protocol is asymmetric, the absolute regression accuracy is modest (best R²=0.36), and one of the two regression tasks in the benchmark has negative R² for every model including REO-VLM. With additional controls and a fair baseline protocol, the contribution could be significant; as it stands, the evidence is not sufficient to support the headline claims.","major_comments":[{"comment":"The comparison protocol is asymmetric. REO-VLM is trained on the REO-Instruct training split in both stages, while GeoChat, LHRS-Bot, Qwen2-VL, and ChatGPT-4o are evaluated zero-shot with only guiding prompts. The only baseline fine-tuned on REO-Instruct is LLaVA, and it appears only in the AGB regression table. The large gaps (e.g., VQA accuracy 80.50% vs. 33.79% for ChatGPT-4o; land-cover OA 19.94% vs. 3.97%) therefore do not establish that REO-VLM outperforms these systems; the baselines have not been given the opportunity to adapt to the task distribution. The paper should either fine-tune all baselines on the same training split or explicitly restrict all comparative claims to the fine-tuned setting.","section":"Tables 2–4 and Figure 5"},{"comment":"The design does not rule out label leakage through the reverse projection. Section 3.4 states that the text annotations contain \"quantitative ground-truth estimates of the Above-Ground Biomass,\" and Section 4.2 says stage one fine-tunes the LLM on these annotations with cross-entropy loss. The reverse projection module in Section 4.1 then reads 4096-dimensional LLM hidden states and projects them into the visual feature space consumed by the regression head. If those hidden states already encode the AGB value that the LLM was trained to generate, the regression head may be reading a memorized label out of the language model rather than integrating scientific domain knowledge. This is load-bearing because the knowledge-driven mechanism is the paper's stated novelty. The paper needs at least three controls: (i) an ablation without the reverse projection; (ii) a training run in which AGB values are withheld from or masked in the text annotations during stage one; and (iii) a test of whether the regression head can recover the target from hidden states of a frozen or randomly initialized LLM.","section":"§3.4, §4.1, §4.2"},{"comment":"The ecological patch counting task fails for every model, including REO-VLM, with negative R² in all rows of Table 3. Since patch counting is one of the four core tasks in REO-Instruct and is listed as a supported capability in Table 1, this is a failure on one of the benchmark's own tasks, not a minor limitation. The paper's explanation that numeric values were \"treated as texts\" and trained with cross-entropy does not apply to REO-VLM, which has a dedicated regression head and MSE loss in stage two; the authors should explain why the regression head was not used for counting, and they should not claim that REO-VLM unifies regression and generation while one of the two regression tasks has negative R².","section":"§5.3 and Table 3"},{"comment":"The AGB regression results are not compared against the full set of baselines. Table 4 includes only LLaVA variants for the direct comparison, yet the text states that \"almost all comparison algorithms fail to perform effective AGB regression\"—GeoChat, LHRS-Bot, Qwen2-VL, and ChatGPT-4o are not evaluated on this task. Moreover, the authors cite Niconet's RMSE of 69.0 on the same dataset, whereas REO-VLM's best RMSE is 75.59; the paper therefore does not demonstrate state-of-the-art regression accuracy. The R²=0.36 result should be reported as a weak positive correlation, not as proof that the model \"captures underlying numerical patterns,\" especially given the label-leakage confound described above.","section":"§5.4 and Table 4"}],"minor_comments":[{"comment":"The heading contains a typo: \"Architectue\" should be \"Architecture.\"","section":"§4.1 heading"},{"comment":"Table 1 marks the counting capability (\"Cnt.\") as supported for REO-VLM, but Table 3 reports negative R² for this task; the table should be qualified to avoid overstating the model's capabilities.","section":"Table 1"},{"comment":"The strategies \"Half layers\" and \"Half layers (deep)\" are not sufficiently distinguished in the text; the reader cannot tell exactly which layers are included in each configuration.","section":"Table 5"},{"comment":"The manuscript does not describe how ChatGPT-4o outputs were checked for factual consistency with the land cover and AGB labels; a brief verification protocol would strengthen the benchmark's credibility.","section":"§3.4 and Figure 3"},{"comment":"The paper states that the test set has about 36K pairs and then says each downstream test subset contains approximately 8.6K unique samples; it should clarify whether the 36K is the union of the four subsets and confirm that the four subsets are mutually exclusive.","section":"§3.3 and §5"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about label leakage is, in my reading, the most serious issue in the manuscript. I would not recommend acceptance until the authors run the proposed controls and fix the asymmetric baseline protocol. I also note that no dataset URL or code release is mentioned, which limits reproducibility and the usefulness of the proposed benchmark."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Two things to know about arXiv:2412.16583. The dataset REO-Instruct is a substantial new resource: 1.6M multimodal EO-language pairs built on AGBD, covering AGB regression plus land cover, patch counting, and VQA. The paper is also honest about its own failures – negative R² for ecological patch counting, and an RMSE on AGB that is worse than Niconet's. That transparency is real credit. The architectural idea, a regression head with a reverse projection from LLM hidden states back into visual space, is concrete and not something I've seen in EO-VLMs before. The token-layer ablation is a useful piece of evidence.\n\nThe soft spots are serious, though. The most load-bearing is the leakage path for the 'knowledge-driven' claim. The text annotations in REO-Instruct include the ground-truth AGB values (Section 3.4). Stage one fine-tunes the LLM on those annotations with cross-entropy (Section 4.2). The reverse projection then pulls 4096-d LLM hidden states into the regression head (Section 4.1). If those hidden states already carry the numeric label the LLM was trained to generate, the regression head is reading a memorized copy, not integrating scientific domain knowledge. The paper reports no control: no run without the reverse projection, no run with AGB withheld from the text, no test of what the hidden states contain beyond the label. This matters because the knowledge-integration story is the stated novelty; without it, the contribution is a regression head bolted onto LLaVA-1.5.\n\nThe comparisons are also asymmetric. Most baselines in Tables 2–4 are zero-shot; REO-VLM is fine-tuned on the benchmark. Outperforming zero-shot ChatGPT-4o or Qwen2-VL is expected and does not establish superiority. The only fair comparison, the fine-tuned LLaVA baseline on AGB, shows a real improvement (R² from about -0.45 to 0.36) but the absolute RMSE is still behind Niconet. And the patch-counting task fails for all models, which the paper acknowledges.\n\nFinally, no code, data, or weights are released. That is a serious deficiency for a paper whose main contribution is a dataset.\n\nSo: the paper deserves a serious referee because the dataset is large and the architecture is worth testing, but the current version's central claims are not well supported. The audience is researchers building EO-VLMs and anyone using language models for numeric prediction. The path to a usable paper is clear – fine-tune all baselines, ablate the reverse projection, train a version where AGB values are withheld from the text, and release the dataset. I'd send it to review with those demands, not desk reject it.","headline":"A large and honestly reported EO benchmark, but the 'knowledge-driven' regression claim is confounded by likely label leakage and asymmetric baselines.","tokens_in":14829,"tokens_out":3774,"would_cite":false,"duration_ms":32107,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A single vision-language model can both describe satellite images and estimate above-ground biomass from them.","keywords":["vision-language models","Earth observation","above-ground biomass regression","multimodal remote sensing","instruction tuning benchmark","reverse projection module","scientific regression","SAR and multispectral fusion"],"falsifier":"Train REO-VLM on a version of REO-Instruct in which all numeric AGB values are removed or paraphrased out of the text annotations, keeping land-cover and human-activity text intact, then measure AGB R-squared on the same test set. If R-squared stays at roughly 0.36, the reverse projection is genuinely transferring domain knowledge; if it collapses toward the fine-tuned LLaVA baseline's negative value, the result depended on label leakage through the language pathway.","tokens_in":13608,"feed_emoji":"🛰️","tokens_out":4967,"duration_ms":37650,"temperature":0.7,"pith_summary":"This paper attempts to show that a vision-language model (VLM) can do more than describe satellite imagery: it can also perform scientific regression, namely estimating above-ground biomass (AGB) from multispectral and radar data. To that end it introduces REO-Instruct, a 1.6-million-pair benchmark that pairs RGB, multispectral, and SAR images with land-cover labels, ecological patch counts, human-activity question-answer pairs, and ground-truth AGB values. Building on this dataset, it proposes REO-VLM, a single model with separate generation and regression heads, trained in two stages so the language side first learns domain knowledge and the regression side then consumes it. The key reported result is a positive R-squared of 0.36 for AGB regression on multispectral input, where a fine-tuned LLaVA baseline is negative, which the authors read as evidence that the model captures real numerical patterns rather than memorizing answers. A sympathetic reader would care because it points toward VLMs as joint tools for environmental monitoring and resource management, not just captioning.","feed_headline":"One VLM now predicts biomass and narrates satellite scenes","feed_subtitle":"Language-driven reasoning plus a regression head pushes above-ground biomass prediction into positive R-squared territory.","key_machinery":"The carrying mechanism is the reverse projection module (R-Proj), a linear layer that projects 4096-dimensional LLM hidden features back into the 1024-dimensional visual feature space, feeding language-derived contextual information into a four-layer MLP-mixer-style regression head. Around it sit three supporting components: spectral recombination, which recasts multispectral bands and SAR polarization channels as pseudo-RGB images so the frozen visual encoder can consume them; visual token selection, which harvests features from middle layers of the visual encoder so regression gets fine detail rather than only high-level semantic tokens; and a two-stage training scheme that first fine-tunes the LLM and generation head with cross-entropy, then trains only the regression head and reverse projection module with MSE loss, decoupling the conflicting objectives of discrete generation and continuous regression.","core_discovery":"On the authors' own terms, REO-VLM is the first unified Earth-observation VLM that integrates scientific regression and generative capabilities in a single model. Using REO-Instruct, the model is trained first with the LLM and generation head, then with a regression head and a reverse projection module that maps LLM hidden states back into visual feature space. In the authors' experiments, this design yields an R-squared of 0.20 (RGB), 0.36 (MS), and 0.35 (MS+SAR) for AGB regression, outperforming a fine-tuned LLaVA baseline whose R-squared is negative across modalities, while also handling land-cover classification, patch counting, and human-activity VQA. The paper claims these positive R-squared values prove the model captures underlying numerical patterns in the data, and that the language-driven reasoning supplies scientific domain knowledge that breaks the information bottleneck of image-only regression.","pith_inferences":["The paper's own setup leaves open a label-leakage path: the REO-Instruct text annotations contain the ground-truth AGB values, and stage one fine-tunes the LLM on those annotations. If the LLM hidden states fed to the regression head already encode the numeric answer being generated, the reported gain over the LLaVA baseline could partly come from copying the label through the reverse projection, ","Because the authors report negative R-squared for ecological patch counting when numbers are treated as text tokens, a natural extension of their thesis is that any VLM regression pipeline should route numeric outputs through a dedicated regression head, and that token-level generation alone is insufficient for continuous scientific quantities.","The reliance on a commercial language model to generate text annotations is a potential reproducibility vulnerability: the benchmark's scientific content depends on the exact prompt templates and model version used, and a systematic release of those prompts would make the dataset's construction fully auditable.","If the mechanism generalizes, the same architecture could be applied to other continuous geophysical attributes such as soil moisture, canopy height, or surface temperature, with the reverse projection providing the knowledge bridge."],"forward_implications":["If the central claim holds, a single VLM can jointly answer questions, classify land cover, count ecological patches, and estimate biomass from the same satellite scenes, simplifying EO analysis pipelines.","Positive R-squared on AGB regression with multispectral input implies VLMs are a viable alternative to dedicated image-only regression models, narrowing the gap with specialized baselines like Niconet while adding descriptive abilities.","The two-stage decoupling strategy suggests that regression and generation objectives need not be reconciled within one loss; separating them by training phase may transfer to other scientific regression tasks in EO, such as species range or population density estimation.","The REO-Instruct benchmark itself provides a reusable training and evaluation substrate for future EO-VLMs aiming at regression-plus-generation.","The reverse projection principle, pulling LLM-generated knowledge back into the visual space, could be adapted to other multimodal regression problems beyond remote sensing."],"supporting_citations":[{"why":"Supplies the underlying EO imagery, land-cover data, and GEDI AGB data that REO-Instruct is built from.","marker":"[23]"},{"why":"Provides the LLaVA-1.5 foundation architecture that REO-VLM adapts with regression heads and reverse projection.","marker":"[13]"},{"why":"Serves as the best single-task AGB regression baseline whose RMSE of 69.0 is the comparison point in the authors' discussion.","marker":"[9]"},{"why":"Prior conversational EO-VLM used as a comparison baseline and as source of the general three-component VLM architecture.","marker":"[8]"},{"why":"Prior multi-sensor EO-VLM compared in the tables and referenced for the conversational VLM design pattern.","marker":"[30]"},{"why":"Prior EO-specific VLM baseline in the comparisons, positioned as lacking regression capability.","marker":"[17]"},{"why":"General-domain VLM baseline (Qwen2-VL) evaluated on the benchmark tasks.","marker":"[26]"},{"why":"Source of the MLP-mixer style architecture used for the regression head.","marker":"[25]"},{"why":"The language model used to generate the text annotations in REO-Instruct.","marker":"[1]"}],"fun_headline_variants":["First Earth-observation VLM to fuse regression and generation","REO-VLM predicts biomass and narrates satellite scenes","Language-driven VLM breaks image-only bottleneck in biomass prediction","From negative to positive R-squared: VLM masters AGB regression","One VLM for scientific regression and image understanding in EO"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The claimed advantage of the reverse projection module rests on the assumption that the LLM's hidden states carry scientific knowledge that is independent of the numeric answer being predicted, rather than a copy of the ground-truth AGB value that appears in the training annotations.","fun_headline_variants_meta":{"raw":{"variants":["First Earth-observation VLM to fuse regression and generation","REO-VLM predicts biomass and narrates satellite scenes","Language-driven VLM breaks image-only bottleneck in biomass prediction","From negative to positive R-squared: VLM masters AGB regression","One VLM for scientific regression and image understanding in EO"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000244,"raw_usage":{"total_tokens":1518,"prompt_tokens":919,"completion_tokens":599,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":535,"completion_tokens_details":{"reasoning_tokens":516}},"tokens_in":535,"tokens_out":599,"duration_ms":6445,"temperature":1.0,"reasoning_tokens":516,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T10:27:01.860236+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train REO-VLM on a version of REO-Instruct in which all numeric AGB values are removed or paraphrased out of the text annotations, keeping land-cover and human-activity text intact, then measure AGB R-squared on the same test set. If R-squared stays at roughly 0.36, the reverse projection is genuinely transferring domain knowledge; if it collapses toward the fine-tuned LLaVA baseline's negative value, the result depended on label leakage through the language pathway.","supporting_citations":[{"cited_title":"A high-resolution canopy height model of the earth","cited_arxiv_id":null,"evidence_quote":"Serves as the best single-task AGB regression baseline whose RMSE of 69.0 is the comparison point in the authors' discussion."},{"cited_title":"Geochat: Grounded large vision-language model for remote sensing","cited_arxiv_id":null,"evidence_quote":"Prior conversational EO-VLM used as a comparison baseline and as source of the general three-component VLM architecture."},{"cited_title":"Earthgpt: A universal multi-modal large lan- guage model for multi-sensor image comprehension in re- mote sensing domain","cited_arxiv_id":null,"evidence_quote":"Prior multi-sensor EO-VLM compared in the tables and referenced for the conversational VLM design pattern."}],"review_version":1}