{"id":"4955a795-9af9-43da-b020-3036cd0c6a38","arxiv_id":"2507.22346","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"DeltaVLM is an instruction-tuned vision-language model trained on a new 105k-pair dataset to answer multi-turn questions about changes in pairs of satellite images.","lead":"This paper introduces ChangeChat-105k, a dataset of 105,107 instruction-answer pairs, and DeltaVLM, a vision-language model that answers questions about what changed between two satellite images of the same area. It is a step toward letting analysts query satellite imagery interactively, for example by counting new buildings or locating a new road.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Open-ended QA evaluation is circular: test references are ChatGPT paraphrases of the same text-only captions/counts/contours as training, so BLEU/CIDEr gains may reflect style imitation, not visual grounding; multi-turn SOTA claim lacks quantitative evidence.","rationale":"The paper's central claim is SOTA on single-turn captioning and multi-turn interactive change analysis. The captioning claim is partially supported by Table II against human-written LEVIR-CC captions, though the margin over SFT is mixed (SFT wins BLEU-4, METEOR, CIDEr). The interactive claim, however, rests on the open-ended QA results (Table VI) and on qualitative multi-turn examples. The open-ended QA evaluation is circular: both training targets and test references are generated by ChatGPT from the same text-only inputs (LEVIR-CC captions, MCI counts/contours), with no visual information (Section III.5, Fig. 3). Unless one assumes ChatGPT's text-only paraphrases are a perfect surrogate for human visual judgment, the reported BLEU-1 36.67 and CIDEr 127.38 gains over general VLMs (GPT-4o: 33.08/72.58; GLM-4V-Plus: 34.27/100.66) are as likely to reflect distribution matching to ChatGPT style as improved perception. This is the weakest assumption because the entire 'interactive analysis' contribution is quantified mainly by this one task; the structured tasks are fewer in number and are essentially grounded in change maps, so they provide support but not for the open-ended reasoning claim. The absence of any quantitative multi-turn evaluation and of comparisons to ChangeChat/Change-Agent further weakens the second half of the claim. A human-annotated open-ended QA test set would settle whether the apparent SOTA is real. I agree with the reader's conditional verdict: the contribution is plausible and the structured-task results give independent support, but the open-ended evaluation must be re-grounded before the SOTA claim can be accepted.","tokens_in":19076,"tokens_out":8991,"duration_ms":95359,"concrete_test":"Have expert annotators who see only the bi-temporal images (not the LEVIR-CC captions or MCI counts/contours) write reference answers for a random sample of N=200 open-ended QA test pairs. Recompute Table VI using these human references for DeltaVLM, GPT-4o, GLM-4V-Plus, Gemini-1.5-Pro, and Qwen-VL-Plus. If DeltaVLM's BLEU/CIDEr lead over the general VLMs shrinks or reverses, the ChatGPT-generated references are the source of the apparent SOTA and the interactive-analysis claim must be downgraded. As a secondary check, run the same models on the Change-Agent benchmark's multi-turn tasks to obtain a quantitative comparison against the closest prior interactive system.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The weakest point in the central SOTA claim is the evaluation of open-ended QA and multi-turn dialogue. In Section III.5 (Fig. 3), the authors state that ChatGPT generated all open-ended QA and conversational responses 'from prompts we constructed based on five captions, as well as the change contours and counting information' and explicitly note 'we did not provide any visual information to ChatGPT.' The test references in Table VI are produced by the same pipeline, so the ground truth is a paraphrase of the same text-only captions/counts/contours used to generate training data. High BLEU/CIDEr on this task therefore measures how closely DeltaVLM reproduces ChatGPT's linguistic style and priors, not whether it sees and understands the visual changes. Because the open-ended QA test set (7,527 pairs, Table I) is the largest non-rule-based component, the claimed advantage over GPT-4o/GLM-4V-Plus may be an artifact of distribution match rather than perceptual quality. Moreover, the multi-turn dialogue results are only qualitative (Fig. 5); no quantitative metric or comparison with the closest prior systems ChangeChat [56] and Change-Agent [57] is reported. Thus the 'multi-turn interactive change analysis' half of the central claim is unsupported by the evidence.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces remote sensing image change analysis (RSICA), a multi-turn instruction-following paradigm for bi-temporal remote sensing images, and presents ChangeChat-105k, a dataset with 105,107 instruction-response pairs covering captioning, binary change classification, category-specific quantification, localization, open-ended QA, and multi-turn dialogue. The authors also propose DeltaVLM, an end-to-end VLM with a bi-temporal vision encoder, a cross-semantic relation measuring (CSRM) module, and an instruction-guided Q-former, with a frozen Vicuna-7B decoder. Experiments on ChangeChat-105k report strong results on structured tasks relative to general VLMs, and the abstract claims state-of-the-art performance on both single-turn captioning and multi-turn interactive change analysis. The central architectural ideas are plausible, but the evaluation of open-ended QA and multi-turn dialogue has significant validity gaps that affect the main claim.","tokens_in":19327,"tokens_out":5260,"duration_ms":59130,"significance":"If the results hold, the paper would contribute a useful benchmark and a reasonable architecture for interactive change analysis in remote sensing. The structured-task evaluations (captioning, classification, quantification, localization) use independently annotated LEVIR-CC and LEVIR-MCI data, and the model's design choices—selective Bi-VE fine-tuning, CSRM filtering, and instruction-guided Q-former—are clearly presented. The release of code, dataset, and weights is a practical strength. However, the open-ended QA evaluation is circular because references are generated by ChatGPT from text-only captions/counts/contours, and the multi-turn dialogue claim rests on qualitative examples only. These issues currently limit the significance of the SOTA claim to the structured, rule-based portions of the benchmark.","major_comments":[{"comment":"The open-ended QA evaluation is circular. The test references for Table VI are produced by the same ChatGPT-based pipeline described in §III.5, in which ChatGPT receives only five captions, change counts, and contours, with no visual input, and the model is trained on the same type of GPT-generated references. BLEU and CIDEr on this 7,527-pair test set therefore measure how closely DeltaVLM reproduces ChatGPT's textual style and priors rather than whether it perceptually grounds changes in the images. Because this is the largest non-rule-based component of the evaluation, the claimed advantage over GPT-4o and GLM-4V-Plus is not evidence of visual understanding. Please add image-grounded human-annotated references, or at minimum a human evaluation of answer correctness, and do not use these numbers as the primary evidence for the SOTA claim.","section":"§III.5 and Table VI"},{"comment":"The abstract claims state-of-the-art performance on 'multi-turn interactive change analysis,' but the only multi-turn evidence is the qualitative demo in Fig. 5. No quantitative metric, consistency measure, or comparison with the closest prior systems ChangeChat [56] and Change-Agent [57] is reported. Please add a quantitative multi-turn evaluation protocol, for example per-turn accuracy/consistency or human-rated dialogue quality, and compare with those systems. Without this, the multi-turn half of the central claim is unsupported by the evidence currently in the paper.","section":"§V.D and Abstract"},{"comment":"The baseline comparisons are not fully specified. The paper does not state the prompts or adaptation scheme used for GPT-4o, Qwen-VL-Plus, GLM-4V-Plus, and Gemini-1.5-Pro, nor how the RS change captioning models RSICCFormer, PromptCC, PSNet, and SFT were obtained for the ChangeChat-105k test set, for example original checkpoints versus retraining on the same training split. The paper also reports no error bars or repeated trials. These details are necessary to interpret the magnitude of the reported gains, especially on open-ended QA and localization, which are sensitive to output formatting and prompt wording. Please document the baselines' prompts/adaptation and report variance across at least a few runs.","section":"§V.B and Tables II–VI"}],"minor_comments":[{"comment":"The ablation discussion refers to 'Table II' when it should refer to Table VII, and the sentence 'After freezing the parameters of the Bi-VE, the model exhibits improvements in all metrics' is ambiguous because the w/o Bi-VE FT row improves over w/o CSRM but not over the full DeltaVLM model.","section":"§V.C"},{"comment":"In the third example of Fig. 5, the response 'Bottom-left.' appears under a yes/no question, but it seems to answer the following localization question; the alignment in the figure should be corrected.","section":"Fig. 5"},{"comment":"The phrase 'averaging 35%'s improvement across all metrics' is ungrammatical and not directly derivable from Table IV without a definition of how the average improvement is computed.","section":"§V.B.3"},{"comment":"There are minor typographical errors: 'conditioned on the instuction P' in the text around Eq. (3) should be 'instruction', and 'potential for for practical use' in §V.D contains a duplicated word.","section":"§IV.A and §V.D"}],"recommendation":"major_revision","confidential_remarks":"The paper has a solid structured-task core and a well-described architecture, but the central SOTA claim is overstated given the circular open-ended QA evaluation and the absence of quantitative multi-turn results. I do not see this as a rejection: the issues are fixable with additional evaluation and with a recalibrated claim. I would encourage the editor to require the authors to address the baseline specifications and the multi-turn quantification before considering publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here is the take. DeltaVLM is a real empirical package: a 105k instruction dataset for interactive change analysis, a model with a sensible design, and structured-task results that mostly hold up. But the open-ended QA half of the SOTA claim rests on a circular evaluation. The test references in Table VI are ChatGPT paraphrases of the same text captions, counts, and contours that generated the training data. The paper explicitly says no visual information was given to ChatGPT, so high BLEU/CIDEr there measures style imitation, not perceptual grounding.\n\nWhat is new: ChangeChat-105k, with 105,107 instruction pairs across six task types, is a concrete resource. The architecture, a fine-tuned bi-temporal EV-A ViT, a CSRM gating mechanism, an instruction-guided Q-former, and a frozen Vicuna, is a reasonable design. The ablations show CSRM matters: without it, binary change classification F1 collapses to 0.62. The classification, counting, and localization results against general VLMs are plausible and use LEVIR-MCI annotations, so those parts are not tainted by the GPT loop. The released code and data make this worth engaging with.\n\nSoft spots, in proportion. First, the open-ended QA evaluation is the biggest problem. With 7,527 of 17,172 test pairs coming from the ChatGPT-generated pipeline, the paper's claimed edge over GPT-4o and GLM-4V-Plus on that task is more likely distribution match than visual understanding. No human evaluation or independently grounded references are provided. This does not sink the structured results, but it does mean the interactive analysis claim is only weakly supported. Second, the closest prior systems, ChangeChat, which is the authors' own earlier ICASSP paper, and Change-Agent, are never quantitatively benchmarked. Leaving out ChangeChat is especially hard to justify, since the task overlap is direct. Third, multi-turn dialogue is evaluated only qualitatively, and the demo in Fig. 5 is thin. The absence of a quantitative multi-turn metric is a real gap. Fourth, no error bars or baseline prompting protocols are reported. That is a minor-to-moderate issue and standard in the field, but still worth fixing.\n\nThe reader's conditional verdict is about right. This paper deserves a serious referee: the dataset and architecture are usable, the structured results are credibly grounded, and the flaws are fixable. Send it to review with a request that the authors redo or relabel the open-ended evaluation, add the missing ChangeChat/Change-Agent comparison, and report variance and baseline protocols. If those changes land, this is a solid contribution. If not, the interactive SOTA claim should be withdrawn.","headline":"Useful dataset and credible architecture, but the open-ended QA evaluation is circular, so the multi-turn SOTA claim is not yet supported.","tokens_in":770,"tokens_out":2096,"would_cite":true,"duration_ms":55034,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"DeltaVLM claims that instruction-guided, multi-turn dialogue about changes between two satellite images is achievable, and backs this claim with a new 105,107-pair dataset and state-of-the-art results across six change-analysis tasks.","keywords":["remote sensing image change analysis","change captioning","bi-temporal change detection","visual question answering","vision-language models","instruction following","ChangeChat-105k","multi-turn dialogue"],"falsifier":"Take a random subsample of ChangeChat-105k change pairs, have human annotators answer the open-ended questions by looking only at the two images, and compare DeltaVLM's outputs against those human answers; if agreement is at chance while BLEU and CIDEr against the GPT references are high, the open-ended claim fails. A cheaper check is to paraphrase each GPT-generated answer while preserving the counts and spatial facts, then re-score: a large drop would mean the model is matching surface phrasing rather than the underlying change content.","tokens_in":18798,"feed_emoji":"🛰️","tokens_out":9943,"duration_ms":99498,"temperature":0.7,"pith_summary":"This paper tries to establish remote sensing image change analysis (RSICA) as a task: rather than emitting a change mask or a fixed caption, a model should let a user ask a question about what changed between two satellite images of the same place, get an answer, and then ask a follow-up. To make that possible, the authors construct ChangeChat-105k, a dataset of 105,107 instruction-response pairs derived from the LEVIR-CC and LEVIR-MCI benchmarks, spanning change captioning, binary change classification, category-specific counting, localization in a 3x3 grid, open-ended QA, and multi-turn dialogue. On top of this dataset they propose DeltaVLM, which computes the pixel-level difference between bi-temporal visual features, filters that difference through a cross-semantic relation measuring (CSRM) module, and aligns the surviving change features with the user's instruction through a Q-former before a frozen large language model decodes the answer. The paper reports state-of-the-art numbers on ChangeChat-105k across all six tasks, with ablations showing that the CSRM filter is essential and that fine-tuning the last two blocks of the vision encoder adds a further gain. If these results hold, satellite change analysis becomes an interactive conversation rather than a one-shot prediction.","feed_headline":"DeltaVLM turns before/after satellite images into a dialogue","feed_subtitle":"A new model answers follow-up questions about land-cover change, beating general VLMs on counting and localization.","key_machinery":"The load-bearing mechanism is the instruction-guided difference perception module (IDPM), a two-stage filter. Its first stage, cross-semantic relation measuring (CSRM), computes the raw difference $F_{\\mathrm{diff}} = F_{t_2} - F_{t_1}$ and then, for each temporal feature $F_{t_1}$ and $F_{t_2}$, builds a context vector $C = \\tanh(W_c[F_{\\mathrm{diff}}; F_t]+b_c)$ and a gate vector $G = \\sigma(W_g[F_{\\mathrm{diff}}; F_t]+b_g)$, refining the features as $F'_t = G \\odot C$ so that semantically relevant changes survive and nuisance differences are suppressed. The second stage is a Q-former with 32 learnable queries, in the style of InstructBLIP, that attend to the concatenation of the filtered features and the instruction prompt and compress the result into the language model's input space. The paper's argument is that this pipeline is what lets a frozen 7-billion-parameter language model produce instruction-specific, multi-turn answers about changes.","core_discovery":"On its own terms, the paper's central claim is that instruction-guided difference perception is what makes multi-turn change analysis work. DeltaVLM feeds the two images through a shared bi-temporal vision encoder to get features $F_{t_1}$ and $F_{t_2}$, forms the raw difference $F_{\\mathrm{diff}} = F_{t_2} - F_{t_1}$, and then uses CSRM to compute, for each temporal state, a context vector and a gate vector from the concatenation of difference and original features; the gate multiplies the context to suppress irrelevant variation such as lighting or sensor noise. An instruction-guided Q-former then runs learnable queries through self-attention and cross-attention over the filtered features and the user's prompt, producing 32 tokens that a frozen Vicuna-7B decoder turns into a response. The paper demonstrates the claim on the ChangeChat-105k test split, reporting 85.78 BLEU-1 on captioning, 93.83 F1 on binary change classification, road-count MAE of 0.24, localization F1 of 67.94 for roads and 78.99 for buildings, and CIDEr of 127.38 on open-ended QA; ablations attribute the margin to the CSRM module and to selective fine-tuning of the vision encoder.","pith_inferences":["Because the open-ended QA and dialogue answers were generated by ChatGPT from text captions, counts, and contours rather than from the images, the dataset probably rewards stylistic imitation of ChatGPT's phrasing as much as perceptual accuracy; a human perceptual-accuracy study on a subsample would separate the two.","The 3x3 grid localization protocol cannot distinguish a model that knows the change is in the top-left cell from one that only knows the broad region; refining to continuous or pixel-level localization would likely shrink the reported margin over general vision-language models.","The difference-then-filter recipe is generic: the same CSRM-plus-Q-former structure could be applied to video question answering or longitudinal medical imaging, where nuisance variation between observations must be gated before answering the user's query.","Since the language model is frozen, swapping Vicuna-7B for a stronger instruction-tuned language model at inference time would probably improve open-ended QA and multi-turn coherence without retraining the visual side; that is a cheap experiment the paper does not run."],"forward_implications":["Interactive change analysis of bi-temporal satellite imagery can be delivered by a frozen large language model; only the vision encoder and alignment modules need task-specific training, which lowers the adaptation cost for new remote sensing domains.","The CSRM ablation, which collapses to predicting \"no change\" without the filter, indicates that explicit semantic filtering of the temporal difference is necessary for any model that must answer change questions, not an optional refinement.","The performance gap between DeltaVLM and general-purpose vision-language models on ChangeChat-105k implies that domain-specific instruction tuning on bi-temporal remote sensing data is the main driver of improvement; scale and general vision pretraining alone are not enough.","ChangeChat-105k gives future work a common testbed with six task types, so progress in interactive change analysis can be measured against a fixed distribution rather than bespoke task splits."],"supporting_citations":[{"why":"Supplies the LEVIR-CC bi-temporal image pairs and change captions that anchor the captioning task, and contributes the RSICCFormer baseline against which DeltaVLM is measured.","marker":"[8]"},{"why":"Provides the LEVIR-MCI pixel-level change maps from which binary labels, object counts, and 3x3 grid localization targets are derived.","marker":"[57]"},{"why":"Is the ChatGPT in-context learning engine that generates the open-ended QA and multi-turn dialogue responses in ChangeChat-105k.","marker":"[17]"},{"why":"Is the InstructBLIP design that the instruction-guided Q-former adapts for cross-modal alignment.","marker":"[54]"},{"why":"Is the EVA ViT-g/14 backbone that the bi-temporal vision encoder is initialized from and selectively fine-tuned on.","marker":"[58]"},{"why":"Is Vicuna-7B, the frozen large language model that decodes the aligned difference features into responses.","marker":"[60]"},{"why":"Is the earlier ChangeChat model and dataset that this work extends by adding instruction-guided difference perception and a larger benchmark.","marker":"[56]"},{"why":"Is the sparse focus Transformer (SFT) baseline that posts the best METEOR and CIDEr captioning scores, delimiting where DeltaVLM wins and where it does not.","marker":"[33]"}],"fun_headline_variants":["DeltaVLM chats about land changes in satellite images","Interactive change analysis for satellite image pairs","Ask DeltaVLM follow-ups on before/after maps","Multi-turn satellite change Q&A with DeltaVLM","DeltaVLM lets you interrogate image differences"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that ChatGPT-generated answers, written from text captions, counts, and contours without ever seeing the images, are valid ground truth for the open-ended and dialogue tasks; if those references encode GPT priors rather than actual image content, the reported BLEU and CIDEr gains for those tasks measure style imitation rather than perceptual accuracy.","fun_headline_variants_meta":{"raw":{"variants":["DeltaVLM chats about land changes in satellite images","Interactive change analysis for satellite image pairs","Ask DeltaVLM follow-ups on before/after maps","Multi-turn satellite change Q&A with DeltaVLM","DeltaVLM lets you interrogate image differences"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000591,"raw_usage":{"total_tokens":2865,"prompt_tokens":1134,"completion_tokens":1731,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":750,"completion_tokens_details":{"reasoning_tokens":1658}},"tokens_in":750,"tokens_out":1731,"duration_ms":13315,"temperature":1.0,"reasoning_tokens":1658,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T11:48:00.699890+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Take a random subsample of ChangeChat-105k change pairs, have human annotators answer the open-ended questions by looking only at the two images, and compare DeltaVLM's outputs against those human answers; if agreement is at chance while BLEU and CIDEr against the GPT references are high, the open-ended claim fails. A cheaper check is to paraphrase each GPT-generated answer while preserving the counts and spatial facts, then re-score: a large drop would mean the model is matching surface phrasing rather than the underlying change content.","supporting_citations":[{"cited_title":"Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset,","cited_arxiv_id":null,"evidence_quote":"Supplies the LEVIR-CC bi-temporal image pairs and change captions that anchor the captioning task, and contributes the RSICCFormer baseline against which DeltaVLM is measured."},{"cited_title":"Change-agent: Towards interactive comprehensive remote sensing change interpretation and analysis,","cited_arxiv_id":null,"evidence_quote":"Provides the LEVIR-MCI pixel-level change maps from which binary labels, object counts, and 3x3 grid localization targets are derived."},{"cited_title":"ChatGPT: Optimizing language models for dialogue,","cited_arxiv_id":null,"evidence_quote":"Is the ChatGPT in-context learning engine that generates the open-ended QA and multi-turn dialogue responses in ChangeChat-105k."},{"cited_title":"Instructblip: Towards general-purpose vision-language models with instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Is the InstructBLIP design that the instruction-guided Q-former adapts for cross-modal alignment."},{"cited_title":"EV A: Exploring the limits of masked visual representation learning at scale,","cited_arxiv_id":null,"evidence_quote":"Is the EVA ViT-g/14 backbone that the bi-temporal vision encoder is initialized from and selectively fine-tuned on."},{"cited_title":"Judging llm-as-a-judge with mt-bench and chatbot arena,","cited_arxiv_id":null,"evidence_quote":"Is Vicuna-7B, the frozen large language model that decodes the aligned difference features into responses."},{"cited_title":"Changechat: An interactive model for remote sensing change analysis via multimodal instruction tuning,","cited_arxiv_id":null,"evidence_quote":"Is the earlier ChangeChat model and dataset that this work extends by adding instruction-guided difference perception and a larger benchmark."},{"cited_title":"A lightweight sparse focus transformer for remote sensing image change captioning,","cited_arxiv_id":null,"evidence_quote":"Is the sparse focus Transformer (SFT) baseline that posts the best METEOR and CIDEr captioning scores, delimiting where DeltaVLM wins and where it does not."}],"review_version":1}