{"id":"29b8a950-10ff-44c8-a668-c52de5167b10","arxiv_id":"2411.11360","paper_version":1,"verdict":"REJECT","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"high","formal_verification":"none","parameter_count":5,"one_line_summary":"CCExpert reports S*_m=81.80 on LEVIR-CC change captioning using a difference-aware module and a 200k-pair pretraining dataset, but possible test-set contamination undermines the claim.","lead":"A remote sensing model called CCExpert describes changes between two images of the same place, and reports a new best score on the LEVIR-CC benchmark. The score is suspect because the training data may have included the test images, and the paper is explicitly marked work in progress.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper does not exclude the LEVIR-CC test split from CC-Foundation training, and selects hyperparameters on test metrics; the S*_m=81.80 SOTA claim is therefore unsupported.","rationale":"The reader's weakest assumption identifies exactly the load-bearing concern I find: the evaluation may be contaminated because the LEVIR-CC test split could be present in the CC-Foundation pretraining corpus and because hyperparameters are selected using test-set metrics. The reported S*_m=81.80 is the paper's only quantitative evidence for the 'significant surpassing' claim, and every ablation and comparison in Section IV is evaluated on LEVIR-CC test. If data from that test split was seen during any training stage, the comparison against prior methods is unfair and the claimed state-of-the-art status collapses. The manuscript contains no sentence stating that the official LEVIR-CC train/test split was preserved; instead, Table I and Section IV-A1 list LEVIR-CC as a component of the pretraining data, and Section IV-A4 explicitly says LEVIR-CC is upsampled threefold in training. The burden is on the authors to show the withheld test set was not used. The test-set-based hyperparameter selection in Tables VII and VIII is a second, independent contamination path that would inflate scores even if the train/test split were clean. The paper's limitations section acknowledges the single-benchmark evaluation and noisy annotations but does not address either contamination issue. Since the central claim is empirical and the evaluation protocol is not demonstrated to be clean, the REJECT verdict is appropriate. I see no reason to change it.","tokens_in":20027,"tokens_out":4961,"duration_ms":45525,"concrete_test":"Obtain the exact image-pair identifiers used in the CC-Foundation training set and in the three training stages (including the threefold-upsampled LEVIR-CC data mentioned in Section IV-A4), and compute their intersection with the LEVIR-CC test split from Liu et al. (2022). Report the overlap count. If any test pair appears in training, the evaluation is invalid. If the overlap is zero, rerun the Table VII/VIII ablations with a held-out validation split for model selection; if the chosen layer set or transformer count changes, the reported gains are test-set artifacts.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that CCExpert achieves S*_m=81.80 on LEVIR-CC, surpassing prior methods. That claim requires the LEVIR-CC test split to be excluded from all training and model selection to avoid test labels. The manuscript does not establish either condition. Table I lists LEVIR-CC (10k image pairs) as part of CC-Foundation, and Section IV-A1 says open-source datasets including LEVIR-CC were included. Section IV-A4 states that 'LEVIR-CC, SYSU-CD, and SECOND datasets are upsampled threefold to increase the proportion of remote sensing images in training,' with no statement that the standard train/test split was respected. Moreover, Tables VII and VIII are captioned 'Performances are validated on LEVIR-CC test set' and are used to select the multi-scale layer set {-2,-5,-8,-11} and the number of Change Aware Transformer Layers (2); hyperparameters are thus chosen directly on test-set scores. If any test pair was in training, the reported improvement is a leakage artifact; if not, the exact split must be documented. As written, the SOTA claim is not independently supported.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CCExpert, a remote sensing image change captioning (RSICC) model built on LLaVA-OneVision, with three components: a Difference-aware Integration Module that injects multi-scale change features into the original image representation, a large continued-pretraining corpus called CC-Foundation (200k image pairs, 1.2M captions), and a three-stage training schedule. The authors report S*_m = 81.80 on the LEVIR-CC benchmark, claiming a large improvement over previous state-of-the-art methods, and ablate the base model, the continued-pretraining data, the difference module, the multi-scale layer set, and the number of Change Aware Transformer layers.","tokens_in":20285,"tokens_out":4387,"duration_ms":40631,"significance":"If the reported results were obtained under a clean evaluation protocol, CCExpert would be a meaningful advance: it demonstrates a modular way to inject change information into an MLLM without disrupting the pretrained architecture, and the CC-Foundation dataset (even if only partially released) would be a useful resource for RSICC. The consistent gains across the 0.5B and 7B variants are suggestive. However, the current evaluation protocol does not support the headline claim: the manuscript does not rule out that the LEVIR-CC test split was used in continued pretraining, and hyperparameters are explicitly selected on the LEVIR-CC test set. The reported 81.80 therefore cannot be interpreted as an unbiased estimate of generalization. The significance of the contribution is currently not established.","major_comments":[{"comment":"The manuscript never states whether the LEVIR-CC test split is excluded from the CC-Foundation training corpus. Table I lists LEVIR-CC (10k image pairs) as a component of CC-Foundation, and Section IV-A4 says 'LEVIR-CC, SYSU-CD, and SECOND datasets are upsampled threefold' for training, with no statement that the standard train/test split was respected. If any LEVIR-CC test pair was in the training data, the reported S*_m = 81.80 on LEVIR-CC is a leakage artifact and the state-of-the-art claim is invalid. The authors must state explicitly which LEVIR-CC split is used for training; if the full dataset was used, the experiments must be redone with the test split excluded.","section":"Section IV-A1 and IV-A4, Table I"},{"comment":"The captions of Tables VII and VIII say 'Performances are validated on LEVIR-CC test set,' and the multi-scale layer set {-2,-5,-8,-11} and the number of Change Aware Transformer Layers (2) are selected by comparing test-set scores. This is test-set model selection, which introduces an optimistic bias into the reported S*_m = 81.80. The authors should split LEVIR-CC into train/validation/test (or use cross-validation) for hyperparameter selection, and report test-set performance only for the final chosen configuration. Without this, the reported improvements cannot be taken as evidence of generalization.","section":"Section IV-C4 and IV-C5, Tables VII and VIII"},{"comment":"All experiments appear to be single runs without error bars or significance tests. The main reported gains are small (0.49 to 1.12 in S*_m), and in MLLM fine-tuning such differences can easily fall within run-to-run variance. The authors should report mean and standard deviation over at least three random seeds for the main baseline, the continued-pretraining ablation, and the final CCExpert model, so the reader can assess whether the differences are meaningful.","section":"Section IV-C, Tables IV-VI"}],"minor_comments":[{"comment":"The citation to Semantic-CC appears as '[ ? ]'; please fill in the missing reference.","section":"Section III-B1"},{"comment":"The text states 'All evaluation metrics improved' after continued pretraining, but for CCExpert-7B, BLEU-1 decreases from 86.37 to 86.35. Please correct either the sentence or the table.","section":"Section IV-C2, Table V"},{"comment":"The vision encoder is referred to as both 'siglip-so400m' and 'Siglip-400m' in different places; please standardize the naming.","section":"Section IV-A4"},{"comment":"There is a typo in the caption: 'dfifference' should be 'difference'.","section":"Table VI caption"},{"comment":"The sentence 'Based on these results, we selected {-2,-5,-8,-11} as the optimal layer combination' would be clearer if it specified that the selection was made on a validation set, once the evaluation protocol is corrected.","section":"Section IV-C4"},{"comment":"The header contains the note 'This work is currently in progress (WIP), with ongoing development and refinement.' For a journal submission, this note should be removed.","section":"Title page"}],"recommendation":"major_revision","confidential_remarks":"The central issue is evaluation integrity. Please impress on the authors that the reported state-of-the-art claim cannot stand without a clean train/test split for CC-Foundation (with the LEVIR-CC test set provably excluded) and without moving hyperparameter selection off the test set. If the authors confirm that the full LEVIR-CC was used in training, the paper should be rejected; if they can rerun the experiments with a proper protocol and report variance, a major revision is appropriate."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: the paper has a substantive dataset and a sensible architecture, but the headline S*m=81.80 on LEVIR-CC is not trustworthy as reported. The authors need to fix the evaluation before the claim can be taken seriously.\n\nWhat is genuinely useful: CC-Foundation, 200k image pairs and 1.2M captions, assembled from existing datasets with GPT-4o-generated captions and expert refinement. If released cleanly, that is a real resource for the RSICC subfield. The difference-aware integration module is also a reasonable, if incremental, idea: extract multi-scale differences and inject them into the base MLLM features without disturbing the pretrained backbone. The three-stage training is a sensible way to mount that module. The paper is honest that the base LLaVA-OneVision already hits 80.19, and the module adds roughly half a point.\n\nThe problems are in the evaluation, and they are load-bearing. Table I lists all 10,000 LEVIR-CC pairs as part of CC-Foundation. Section IV-A4 says LEVIR-CC is upsampled threefold in training, with no statement that the standard train/test split was respected. Tables VII and VIII are explicitly validated on the LEVIR-CC test set and are used to pick the multi-scale layer set and the number of Change Aware Transformer layers. That is hyperparameter selection on test labels. Even if the split was actually clean, the paper neither says so nor provides a validation set, and all results are single runs without error bars on a single benchmark. The WIP note and the unresolved \"Semantic-CC [ ? ]\" reference are minor by comparison.\n\nMy read: the authors probably did not deliberately cheat; they may have just treated LEVIR-CC as a monolithic pretraining corpus without thinking about the split. That does not rescue the paper, but it means the flaw is fixable in revision.\n\nWho this is for: someone working specifically on remote sensing change captioning or on MLLM continued pretraining for domain-specific vision tasks. The dataset could be useful, and the module might transfer to other change-understanding tasks. But the evaluation section as written would mislead anyone who took the SOTA claim at face value.\n\nRecommendation: send it to peer review, but make the first question to the authors an explicit demand for the train/test split documentation and a proper validation protocol. If they can show clean separation and re-select hyperparameters on a validation set, the paper could become a solid subfield contribution. If not, reject.","headline":"The dataset and module are real, but the SOTA claim is compromised by a likely train/test leak and test-set hyperparameter selection.","tokens_in":20826,"tokens_out":2131,"would_cite":false,"duration_ms":23559,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper claims CCExpert reaches 81.80 on the LEVIR-CC change-captioning benchmark by injecting multi-scale difference features into a multimodal language model and continuing pretraining on a 200,000-pair dataset.","keywords":["remote sensing image change captioning","multimodal large language model","difference-aware integration module","CC-Foundation dataset","LEVIR-CC benchmark","continued pretraining","multi-scale feature fusion","change captioning"],"falsifier":"A direct check is to inspect the released training lists for LEVIR-CC test filenames, then rerun the CCExpert-7B recipe from scratch with the test split excluded and with the multi-scale layers and transformer count chosen on a held-out validation split instead of the test table; if the composite score falls to or below the 80.19 of the unmodified base model, the claimed gain is an artifact of test-set exposure.","tokens_in":19809,"feed_emoji":"🛰️","tokens_out":9013,"duration_ms":78641,"temperature":0.7,"pith_summary":"The paper tries to establish that a multimodal large language model can become the best remote sensing change captioner without overhauling its core architecture, by explicitly feeding it multi-scale differences between the two images and by continuing pretraining on a large domain dataset. The authors build CCExpert on the LLaVA-OneVision base, add a Difference-aware Integration Module that extracts change features from several vision-encoder layers and weights them into the original image features, and assemble CC-Foundation: 200,000 image pairs with 1.2 million captions. They report a composite score of $S^*_m = 81.80$ on LEVIR-CC, roughly two points above the previous best listed method, and attribute the gain to the dataset and the module. If true, this makes data-driven continued pretraining plus targeted difference injection a practical recipe for adapting general multimodal models to change understanding.","feed_headline":"Difference-aware model reaches 81.80 on change captioning","feed_subtitle":"A 200,000-pair dataset and multi-scale difference features push a multimodal LLM past prior remote-sensing change captioners.","key_machinery":"The carrying mechanism is the Difference-aware Integration Module, which has two parts. A Diff Expert takes features from vision-transformer layers at positions $-11$, $-8$, $-5$, and $-2$, builds initial difference maps by gating each temporal feature with the other through concatenation and a sigmoid, refines those maps through two Change Aware Transformer Layers using self-attention and cross-attention, and injects the refined difference back into both image feature streams. An Adaptive Adjustment submodule computes per-scale weights, sums the weighted multi-scale features, and adds them residually to the original features. This module is what carries the argument that explicit multi-scale difference information, rather than a redesigned vision encoder or task head, lets the pretrained language model reason about changes. The other load-bearing pieces are the CC-Foundation dataset, built from existing change captioning and change detection datasets with LLM refinement, GPT-4o-generated captions from change masks, and expert annotations; and the three-stage training schedule that pretrains the new module, then unfreezes the whole model, then fine-tunes on domain data.","core_discovery":"The central claim is that CCExpert sets a new state of the art on the LEVIR-CC remote sensing image change captioning benchmark, reaching $S^*_m = 81.80$, where $S^*_m$ is the average of BLEU-4, ROUGE-L, METEOR, and CIDEr-D. The unmodified base model reaches 80.19 with supervised fine-tuning alone, and the previous best listed method reaches about 79.6. The authors attribute the gain to three coordinated choices: keep the standard multimodal model pathway intact, inject multi-scale bi-temporal difference features into the visual tokens before the projector, and continue pretraining on CC-Foundation before domain fine-tuning. Ablations show that the continued pretraining adds about 1.1 points and the Difference-aware Integration module adds about 0.5 points in the 7B setting, so the two together move the model from 80.19 to 81.80.","pith_inferences":["The paper's own tables list all 10,000 LEVIR-CC image pairs as pretraining data and select the multi-scale layer set and transformer count from LEVIR-CC test metrics; unless the test split was withheld, part of the reported gain could be memorization rather than understanding, and the paper does not state that it was withheld.","Several ablated choices move different metrics in different directions, so the reported ranking is sensitive to the choice of the composite $S^*_m$ as the selection criterion rather than any single captioning metric.","Because CC-Foundation captions are partly generated by GPT-4o from change-mask prompts, models trained on it may inherit the annotation style and biases of that generator; the paper's own limitation section acknowledges possible errors in these annotations.","The manuscript labels itself as work in progress and evaluates on a single benchmark, so the stated lead should be read as a status report pending independent reproduction on additional datasets."],"forward_implications":["If the reported score is robust, a standard multimodal language model can be converted into a top change captioner by feature-level difference injection and continued pretraining, without replacing the vision encoder or adding task heads.","The CC-Foundation dataset, if released, becomes a reusable resource for pretraining and evaluating remote sensing change captioning models, including the expert-annotated SECOND-derived split the authors propose as a future benchmark.","The three-stage schedule offers a template for adding lightweight modules to multimodal language models without destructively fine-tuning the base weights.","The 0.5B variant reaching 80.99 suggests most of the gain does not require a 7B decoder, so the recipe may transfer to smaller deployable models."],"supporting_citations":[{"why":"It provides the LLaVA-OneVision base model whose vision tower, projector, and language model weights initialize CCExpert, and it supplies the 80.19 supervised-fine-tuning baseline.","marker":"[19]"},{"why":"It provides the LEVIR-CC dataset and the composite captioning metric used for all evaluation and ablation comparisons.","marker":"[4]"},{"why":"It supplies the inspiration for explicitly enhancing difference features in a vision-language model, which the Difference-aware Integration Module adapts.","marker":"[21]"},{"why":"It supplies the Change Feature Init mechanism and the Semantic-CC method that CCExpert is compared against as an earlier multimodal-model approach.","marker":"[1]"},{"why":"It identifies the Qwen2 family as the language model decoder used inside the language model component of CCExpert.","marker":"[13]"},{"why":"It describes the GPT-4 model used to generate change descriptions from change-mask prompts for several CC-Foundation subsets.","marker":"[14]"},{"why":"It contributes the SYSU-CD change detection data that CC-Foundation expands with generated captions.","marker":"[51]"},{"why":"It contributes the SECOND semantic change detection data that CC-Foundation includes with expert-refined annotations.","marker":"[61]"}],"fun_headline_variants":["New model and dataset push change captioning to 81.80","CCExpert: difference-aware MLLM hits 81.80 on LEVIR-CC","200K image pairs fuel CCExpert to state-of-the-art","Difference-aware integration lifts change captioning score","CCExpert's three-stage training yields benchmark win"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the LEVIR-CC test partition is absent from the CC-Foundation pretraining data and from the ablation hyperparameter choices, so the reported 81.80 measures genuine generalization rather than memorization.","fun_headline_variants_meta":{"raw":{"variants":["New model and dataset push change captioning to 81.80","CCExpert: difference-aware MLLM hits 81.80 on LEVIR-CC","200K image pairs fuel CCExpert to state-of-the-art","Difference-aware integration lifts change captioning score","CCExpert's three-stage training yields benchmark win"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000841,"raw_usage":{"total_tokens":3710,"prompt_tokens":1038,"completion_tokens":2672,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":654,"completion_tokens_details":{"reasoning_tokens":2600}},"tokens_in":654,"tokens_out":2672,"duration_ms":18213,"temperature":1.0,"reasoning_tokens":2600,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-12T18:37:19.685966+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A direct check is to inspect the released training lists for LEVIR-CC test filenames, then rerun the CCExpert-7B recipe from scratch with the test split excluded and with the multi-scale layers and transformer count chosen on a held-out validation split instead of the test table; if the composite score falls to or below the 80.19 of the unmodified base model, the claimed gain is an artifact of test-set exposure.","supporting_citations":[{"cited_title":"A deeply supervised attention metric-based network and an open aerial image dataset for remote sensing change detection,","cited_arxiv_id":null,"evidence_quote":"It contributes the SYSU-CD change detection data that CC-Foundation expands with generated captions."},{"cited_title":"Semantic change detection with asymmetric siamese networks,","cited_arxiv_id":null,"evidence_quote":"It contributes the SECOND semantic change detection data that CC-Foundation includes with expert-refined annotations."}],"review_version":1}