REVIEW 4 major objections 5 minor 29 references
Robust Change Captioning in Remote Sensing: SECOND-CC Dataset and MModalCC Framework
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read The paper introduces SECOND-CC, a 6,041-pair remote sensing change captioning dataset with semantic maps, and MModalCC, a multimodal attention framework that outperforms prior RSICC models by +4.6% on BLEU4 and +9.6% on CIDEr.
desk verdict SECOND-CC is a genuinely new and useful benchmark, but the headline SOTA gains are confounded by MModalCC receiving ground-truth semantic maps that baselines don't get. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing object is the MModalCC architecture operating on two Siamese ResNet encoders, one for RGB pairs and one for semantic-map pairs. Its feature enhancement stage alternates Cross-Modal Cross Attention (CMCA), which lets each modality attend to the other, with Unimodal Difference Cross Attention (UDCA), which attends to the temporal difference within each modality, followed by a residual convolutional block; the decoder then uses Multimodal Gated Cross Attention (MGCA), which computes sigmoid gates over the word, RGB, and semantic representations and fuses them as a weighted sum before each next-word prediction. The dataset supplies the semantic maps that make this machinery possible: pixel-level land-cover labels for six categories, with changed regions colored and unchanged regions left black, so that no-change scenes have entirely black semantic maps.
What would settle it
Take the SECOND-CC test split, replace the provided ground-truth semantic maps with maps predicted by a standard semantic segmentation network trained on SECOND, and rerun MModalCC; if the BLEU4 and CIDEr advantages over RGB-only baselines drop sharply or vanish, the claim that the method is robust to real-world conditions fails in the setting the paper did not test.
Extended reading notes
Core claim
The central claim is that multimodal fusion of RGB and semantic segmentation maps, through a specifically designed attention pipeline, yields more accurate and more robust change captions under realistic acquisition distortions than unimodal RGB methods. The authors show this by evaluating on SECOND-CC, where MModalCC achieves an overall average score $S^*_m$ of 0.487, against 0.447 for the best prior method, with the largest gains on BLEU4 and CIDEr. The paper also claims the dataset itself is a contribution: it provides 6,041 bitemporal image pairs with semantic maps, multiple change captions per pair, and a no-change subset that exposes models to distractors; the augmented version has 10,855 pairs and 54,275 captions. Ablations show that dual cross-attention and dual-modality decoding each improve performance, and attention visualizations support the mechanism: semantic maps often correct RGB attention that is misled by viewpoint or illumination artifacts.
Load-bearing premise
The reported gains assume the model is given ground-truth semantic segmentation maps at test time, not maps the model must predict from the RGB images.
Editorial extensions
If this is right
- If the central claim holds, RSICC systems can be pushed past the RGB-only plateau by adding a semantic channel, with the reported +4.6% BLEU4 and +9.6% CIDEr gains on SECOND-CC AUG.
- Dual cross-attention (CMCA+UDCA) raises the overall $S^*_m$ from 0.462 (CMCA only) and 0.455 (UDCA only) to 0.487, so the combination, not either attention alone, is what carries the improvement.
- The no-change subset becomes a useful stress test: MModalCC identifies unchanged scenes correctly when semantic maps are all-black, even when RGB differences come from viewpoint or illumination.
- The augmented dataset raises overall $S^*_m$ from 0.474 to 0.487, suggesting that simple blur, brighten, mirror, and rotate augmentation is effective for this benchmark.
- The dataset's 30 change-category labels, such as low vegetation to building, enable fine-grained evaluation and could support models that reason explicitly about the type of transition.
Reading between the lines
- The paper's own conclusion (Section V) notes that errors in predicted semantic maps can hurt change captioning and leaves that untested, so the practical deployment claim depends on how accurately such maps can be predicted from RGB images alone.
- The gated fusion idea could transfer to other multimodal remote sensing change tasks, such as optical and SAR combination, where the two modalities have complementary failure modes rather than one being a label-derived oracle.
- The category-transition labels in SECOND-CC could support a two-stage model that first predicts the land-cover transition and then verbalizes it; the paper does not explore that route.
- The no-change subset, with viewpoint and illumination distractors, could also serve as a stress test for change detection systems, not just captioning models.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces SECOND-CC, a new remote sensing change captioning dataset derived from the SECOND change detection dataset, with added human captions and semantic segmentation maps, and an augmented version SECOND-CC AUG. The paper also proposes MModalCC, a multimodal encoder-decoder that fuses RGB images and semantic maps via Cross-Modal Cross Attention (CMCA), Unimodal Difference Cross Attention (UDCA), and a Multimodal Gated Cross Attention (MGCA) decoder. Experiments on SECOND-CC AUG report that MModalCC outperforms RSICCformer, Chg2Cap, and PSNet baselines on standard captioning metrics, and ablation studies analyze the contribution of attention modules and decoder configurations.
Significance. The dataset is a potentially valuable resource: it introduces real-world distortions (illumination, viewpoint, blur, misregistration) and semantic maps into RSICC evaluation, addressing a gap in existing benchmarks. The architecture is a reasonable attempt to combine two modalities, and the authors commit to releasing data and code, which is a strength. However, the headline claim that MModalCC outperforms state-of-the-art methods is currently not supported by a controlled comparison, because the baselines receive only one modality while MModalCC receives both RGB and ground-truth semantic maps. The internal inconsistency in the S*_m metric and the lack of error bars further weaken confidence. If the comparison is made apples-to-apples and the metric issue is fixed, the dataset and architecture could be a useful contribution.
major comments (4)
- [Section IV-F, Table VII] The central claim that MModalCC outperforms RSICCformer, Chg2Cap, and PSNet is not supported by a controlled comparison. MModalCC is trained and tested with both RGB images and ground-truth semantic maps, while each baseline receives only RGB or only SEM. Since MModalCC has strictly more input information, including oracle-level semantic labels at test time, the reported BLEU4 and CIDEr gaps do not isolate the architecture's contribution. The paper's own conclusion (Section V, last paragraph) concedes that errors in predicted semantic maps can negatively affect captioning, yet no experiment uses predicted maps. Please add baselines that also receive both modalities (e.g., two-stream inputs or semantic maps as auxiliary input) and/or an MModalCC variant that uses predicted semantic maps, so the comparison is apples-to-apples.
- [Section IV-E, Table VI] The decoder ablation labeled 'RGB-only' does not actually evaluate a unimodal model. As stated in Section IV-E, 'we retain both CMCA and UDCA modules active throughout this study,' so the RGB stream has already been influenced by semantic features through cross-attention in the encoder. Thus the comparison between RGB-only, SEM-only, and dual-modality decoder configurations cannot attribute the gains to the decoder fusion. A true RGB-only baseline would need to remove the semantic branch (or at least sever the cross-modal connections) so that xrgb is not contaminated by semantic information.
- [Section IV-A, Eq. (12), Table IV] The computation of S*_m is internally inconsistent for no-change rows. The text states that CIDEr is omitted and S*_m is calculated from the remaining four metrics, but Table IV's no-change rows report values that match averaging five metrics with CIDEr set to 0 (e.g., 0.940+0.735+0.972+0.413 = 3.060; /5 = 0.612). Table V's no-change rows, by contrast, average only four metrics (e.g., 0.919+0.710+0.959+0.433 = 3.021; /4 = 0.755). Please define S*_m uniformly and recompute all affected values; currently the no-change scores are not comparable across tables.
- [Section IV-A, Section IV-F, Abstract] The reported improvements are inconsistent in magnitude. The abstract and conclusion claim '+4.6% improvement on BLEU4' and '+9.6% improvement on CIDEr', which are absolute differences (0.386-0.341=0.045; 0.933-0.837=0.096). In Section IV-F, however, the text states that MModalCC outperforms Chg2Cap-RGB by 13.20% for BLEU4 and 11.48% for CIDEr, which are relative percentages. Please clarify which convention is used throughout and ensure the abstract numbers match the table. In addition, no error bars or repeated runs are reported; given that the overall S*_m margins (0.487 vs. 0.447) are modest, a single run is insufficient to establish the ranking.
minor comments (5)
- [Section II-C and II-E] The total number of captions is reported as 30,205 in Section II-C and as 30,159 in Section II-E; please correct the discrepancy.
- [Section III-C and Figure 9] The abbreviation 'MCGA' appears in the text and figure caption; the correct abbreviation defined earlier is 'MGCA' (Multimodal Gated Cross Attention).
- [Algorithm 1] Line 12 of Algorithm 1 contains broken formatting: 'UDCA(r1', r2',... r3', r4', ...' should be written cleanly, and the trailing ellipsis should be removed.
- [Section III-B1, Eq. (3)] In Eq. (3), the sentence lists 'Wi_Q, Wi_K, and Wi_K'; the last term should be Wi_V.
- [References] Reference [18] is incomplete: it ends with '(waiting to be published in IEEE Xplore)'. Please provide full publication details or update the reference.
Circularity Check
No significant circularity: the empirical SOTA claim is evaluated on a held-out test split, not derived from its inputs; the only self-citation is a minor, non-load-bearing augmentation detail.
full rationale
This is an empirical dataset-and-architecture paper, so there is no analytic derivation chain whose conclusion could reduce to its premises. The central claim that MModalCC outperforms RSICCformer, Chg2Cap, and PSNet rests on Table VII, computed on a held-out test split (Section II-E) with standard external captioning metrics, so the BLEU4 and CIDEr margins are not defined into existence by the model equations. The strongest caveats are experimental-design limitations, not circularity: MModalCC is given ground-truth semantic maps in addition to RGB at test time while baselines receive one modality, and Section V concedes that errors in predicted semantic maps could negatively affect performance; likewise, Table VI's 'RGB-only' decoder row still runs the full two-branch encoder with CMCA and UDCA, so it does not isolate the RGB stream. These are confounds for the architecture claim, but the outputs are still measured against human captions rather than reconstructed from the inputs. The only self-citation is [18] for the augmentation recipe in Section II-D; it is a training detail whose benefit is verified by the paper's own Table IV, and the central fusion and captioning result does not rely on it, so it is not load-bearing.
Assumptions & free parameters
free parameters (5)
- beam_size =
4
- learning_rate =
5e-5
- max_epochs =
30
- gaussian_blur_kernel =
5x5
- cmca_udca_iterations =
2
assumptions (4)
- domain assumption The human captions in SECOND-CC are accurate and consistent descriptions of the image-pair changes.
- domain assumption The semantic maps in SECOND are accurate pixel-level labels of changed regions.
- domain assumption The SECOND-CC test split is representative of the claimed real-world challenges.
- standard math Standard captioning metrics correlate with caption quality in this domain.
Cite this review
Pith. "Pith review of Robust Change Captioning in Remote Sensing: SECOND-CC Dataset and MModalCC Framework." pith.science (2026). https://pith.science/paper/E7NWHBN2
@misc{pith2026250110075,
author = {Pith},
title = {Pith review of: Robust Change Captioning in Remote Sensing: SECOND-CC Dataset and MModalCC Framework},
year = {2026},
howpublished = {\url{https://pith.science/paper/E7NWHBN2}},
note = {Machine review of arXiv:2501.10075}
}
read the original abstract
Remote sensing change captioning (RSICC) aims to describe changes between bitemporal images in natural language. Existing methods often fail under challenges like illumination differences, viewpoint changes, blur effects, leading to inaccuracies, especially in no-change regions. Moreover, the images acquired at different spatial resolutions and have registration errors tend to affect the captions. To address these issues, we introduce SECOND-CC, a novel RSICC dataset featuring high-resolution RGB image pairs, semantic segmentation maps, and diverse real-world scenarios. SECOND-CC which contains 6,041 pairs of bitemporal RS images and 30,205 sentences describing the differences between images. Additionally, we propose MModalCC, a multimodal framework that integrates semantic and visual data using advanced attention mechanisms, including Cross-Modal Cross Attention (CMCA) and Multimodal Gated Cross Attention (MGCA). Detailed ablation studies and attention visualizations further demonstrate its effectiveness and ability to address RSICC challenges. Comprehensive experiments show that MModalCC outperforms state-of-the-art RSICC methods, including RSICCformer, Chg2Cap, and PSNet with +4.6% improvement on BLEU4 score and +9.6% improvement on CIDEr score. We will make our dataset and codebase publicly available to facilitate future research at https://github.com/ChangeCapsInRS/SecondCC
Figures
Figures from the paper (8 more)
Reference graph
Works this paper leans on
-
[1]
Change captioning: A new paradigm for multitemporal remote sensing image analysis,
G. Hoxha, S. Chouaf, F. Melgani, and Y . Smara, “Change captioning: A new paradigm for multitemporal remote sensing image analysis,” IEEE Transactions on Geoscience and Remote Sensing , vol. 60, pp. 1–14, 2022
2022
-
[2]
Y . Li, X. Zhang, X. Cheng, P. Chen, and L. Jiao, “Inter-temporal interaction and symmetric difference learning for remote sensing image change captioning,” IEEE Transactions on Geoscience and Remote Sensing, 2024
work page 2024
-
[3]
Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset,
C. Liu, R. Zhao, H. Chen, Z. Zou, and Z. Shi, “Remote sensing image change captioning with dual-branch transformers: A new method and a large scale dataset,” IEEE Transactions on Geoscience and Remote Sensing, vol. 60, pp. 1–20, 2022
2022
-
[4]
Progressive Scale-aware Network for Remote sensing Image Change Captioning
C. Liu, J. Yang, Z. Qi, Z. Zou, and Z. Shi, “Progressive scale-aware network for remote sensing image change captioning,” 2023. [Online]. Available: https://arxiv.org/abs/2303.00355
work page Pith review arXiv 2023
-
[5]
Multi-scale attentive fusion network for remote sensing image change captioning,
C. Chen, Y . Wang, and K.-H. Yap, “Multi-scale attentive fusion network for remote sensing image change captioning,” in 2024 IEEE International Symposium on Circuits and Systems (ISCAS) . IEEE, May 2024, p. 1–5. [Online]. Available: http://dx.doi.org/10.1109/ ISCAS58744.2024.10558583
arXiv 2024
-
[6]
Interactive change-aware transformer network for remote sensing image change captioning,
C. Cai, Y . Wang, and K.-H. Yap, “Interactive change-aware transformer network for remote sensing image change captioning,” Remote Sensing, vol. 15, no. 23, p. 5611, Dec. 2023
work page 2023
-
[7]
Y . Zhu, L. Li, K. Chen, C. Liu, F. Zhou, and Z. Shi, “Semantic- cc: Boosting remote sensing image change captioning via foundational knowledge and semantic guidance,” arXiv preprint arXiv:2407.14032 , 2024
arXiv 2024
-
[8]
Change-agent: Toward interactive comprehensive remote sensing change interpretation and analysis,
C. Liu, K. Chen, H. Zhang, Z. Qi, Z. Zou, and Z. Shi, “Change-agent: Toward interactive comprehensive remote sensing change interpretation and analysis,” IEEE Transactions on Geoscience and Remote Sensing , vol. 62, p. 1–16, 2024. [Online]. Available: http://dx.doi.org/10.1109/TGRS.2024.3425815
arXiv 2024
Show all 29 references
-
[9]
Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain,
W. Zhang, M. Cai, T. Zhang, Y . Zhuang, and X. Mao, “Earthgpt: A universal multimodal large language model for multisensor image comprehension in remote sensing domain,” IEEE Transactions on Geo- science and Remote Sensing , vol. 62, pp. 1–20, 2024
2024
-
[10]
Ccexpert: Ad- vancing mllm capability in remote sensing change captioning with difference-aware integration and a foundational dataset,
Z. Wang, M. Wang, S. Xu, Y . Li, and B. Zhang, “Ccexpert: Ad- vancing mllm capability in remote sensing change captioning with difference-aware integration and a foundational dataset,” arXiv preprint arXiv:2411.11360, 2024
2024 arXiv
-
[11]
Changes to captions: An attentive network for remote sensing change captioning,
S. Chang and P. Ghamisi, “Changes to captions: An attentive network for remote sensing change captioning,” IEEE Transactions on Image Processing, 2023
2023
-
[12]
Image difference captioning with instance-level fine-grained feature representation,
Q. Huang, Y . Liang, J. Wei, Y . Cai, H. Liang, H.-f. Leung, and Q. Li, “Image difference captioning with instance-level fine-grained feature representation,” IEEE transactions on multimedia , vol. 24, pp. 2004– 2017, 2021
2004
-
[13]
Semantic change detection with asymmetric siamese networks,
K. Yang, G.-S. Xia, Z. Liu, B. Du, W. Yang, M. Pelillo, and L. Zhang, “Semantic change detection with asymmetric siamese networks,” arXiv preprint arXiv:2010.05687, 2020
2010 arXiv
-
[14]
Asymmetric siamese networks for semantic change detection in aerial images,
——, “Asymmetric siamese networks for semantic change detection in aerial images,” IEEE Transactions on Geoscience and Remote Sensing , vol. 60, pp. 1–18, 2022
2022
-
[15]
Changemamba: Re- mote sensing change detection with spatio-temporal state space model,
H. Chen, J. Song, C. Han, J. Xia, and N. Yokoya, “Changemamba: Re- mote sensing change detection with spatio-temporal state space model,” arXiv preprint arXiv:2404.03425 , 2024
2024 arXiv
-
[16]
Joint spatio-temporal modeling for semantic change detection in remote sensing images,
L. Ding, J. Zhang, H. Guo, K. Zhang, B. Liu, and L. Bruzzone, “Joint spatio-temporal modeling for semantic change detection in remote sensing images,” IEEE Transactions on Geoscience and Remote Sensing, 2024
2024
-
[17]
Change- agent: Towards interactive comprehensive change interpretation and analysis from change detection and change captioning,
C. Liu, K. Chen, H. Zhang, Z. Qi, Z. Zou, and Z. Shi, “Change- agent: Towards interactive comprehensive change interpretation and analysis from change detection and change captioning,” arXiv preprint arXiv:2403.19646, 2024
2024 arXiv
-
[18]
Data augmentation in remote sensing image change captioning,
O. Karimli, I. Mustafazade, A. C. Karaca, and M. F. Amasyali, “Data augmentation in remote sensing image change captioning,” in2024 IEEE 9th International Conference on Computer Science and Engineering , 2024, p. (waiting to be published in IEEE Xplore)
2024
-
[19]
Attention is all you need,
A. Vaswani, “Attention is all you need,” Advances in Neural Information Processing Systems, 2017
2017
-
[20]
Masked multi-head self-attention for causal speech enhancement,
A. Nicolson and K. K. Paliwal, “Masked multi-head self-attention for causal speech enhancement,” Speech Communication, vol. 125, pp. 80– 96, 2020
2020
-
[21]
Adam: A method for stochastic optimization,
D. P. Kingma and J. Ba, “Adam: A method for stochastic optimization,” CoRR, vol. abs/1412.6980, 2014. [Online]. Available: https://api.semanticscholar.org/CorpusID:6628106
2014 arXiv
-
[22]
Deep residual learning for image recognition,
K. He, X. Zhang, S. Ren, and J. Sun, “Deep residual learning for image recognition,” CoRR, vol. abs/1512.03385, 2015. [Online]. Available: http://arxiv.org/abs/1512.03385
2015 arXiv
-
[23]
Imagenet: A large-scale hierarchical image database,
J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei, “Imagenet: A large-scale hierarchical image database,” in 2009 IEEE Conference on Computer Vision and Pattern Recognition , 2009, pp. 248–255
2009
-
[24]
Global visual feature and linguistic state guided attention for remote sensing image captioning,
Z. Zhang, W. Zhang, M. Yan, X. Gao, K. Fu, and X. Sun, “Global visual feature and linguistic state guided attention for remote sensing image captioning,” IEEE Transactions on Geoscience and Remote Sensing , vol. 60, pp. 1–16, 2022
2022
-
[25]
Bleu: a method for automatic evaluation of machine translation,
K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, “Bleu: a method for automatic evaluation of machine translation,” in Proceedings of the 40th Annual Meeting on Association for Computational Linguistics , ser. ACL ’02. USA: Association for Computational Linguistics, 2002, p. 311...
2002
-
[26]
ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks,
K. Ganesan, “ROUGE 2.0: Updated and improved measures for evaluation of summarization tasks,” CoRR, vol. abs/1803.01937, 2018. [Online]. Available: http://arxiv.org/abs/1803.01937
2018 arXiv
-
[27]
METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,
S. Banerjee and A. Lavie, “METEOR: An automatic metric for MT evaluation with improved correlation with human judgments,” in Proceedings of the ACL Workshop on Intrinsic and Extrinsic Evaluation Measures for Machine Translation and/or Summarization , J. Goldstein, A. Lavie, C....
2005
-
[28]
Cider: Consensus- based image description evaluation,
R. Vedantam, C. L. Zitnick, and D. Parikh, “Cider: Consensus- based image description evaluation,” CoRR, vol. abs/1411.5726, 2014. [Online]. Available: http://arxiv.org/abs/1411.5726
2014 arXiv
-
[2016]
Available: http://arxiv.org/abs/1607.08822
[Online]. Available: http://arxiv.org/abs/1607.08822
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.