Pith. sign in

REVIEW 3 major objections 4 minor 24 references

Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation

T0 review · 3 major / 4 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that self-supervised pretraining based on predicting relative patch positions transfers to multimodal satellite semantic segmentation better than masked-reconstruction pretraining, reporting 74.62 flood IoU on Sen1Floods11.

desk verdict Solid empirical ablations of a sensible multimodal LOCA adaptation, but the headline claim that position prediction beats reconstruction is undercut by test-set selection, a confounded baseline comparison, and the fact that the best config masks the position-prediction signal away. read the letter →

arxiv 2506.06852 v2 pith:Q4ZUZBVR submitted 2025-06-07 cs.CV cs.AI

John Waithaka , Moise Busogi This is my paper
classification cs.CVcs.AI
keywords self-supervisedlearningpositionpredictionsemanticsegmentationsatelliteimagerymultimodalpretrainingmaskedautoencoderfloodmappingchannelgrouping
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper adapts LOCA, a self-supervised method originally built for natural images, to multimodal satellite imagery by making its relative-patch-position prediction task work across Sentinel-2 multispectral bands, Sentinel-1 synthetic-aperture radar, and elevation data. It claims that this position-prediction objective, rather than the masked-image-reconstruction objective used by SatMAE, SatMAE++, ScaleMAE, and MMEarth, learns patch representations that transfer better to semantic segmentation. On the Sen1Floods11 flood-mapping benchmark, the adapted method reaches 74.62 flood IoU and 85.49 mIoU, versus 65.28/80.56 for SatMAE and 68.92/82.34 for MMEarth. The practical reason to care is that pixel-level labels for satellite imagery are scarce and expensive, so a self-supervised pretraining recipe that gets more segmentation accuracy from the same unlabelled data would directly reduce annotation cost.

What carries the argument

The load-bearing object is the relative patch position prediction pretext task: a query view and a much larger reference view are sampled from the same multimodal image, each query patch is asked to classify which of the $N_{\mathrm{ref}} = 196$ reference positions it corresponds to, and the loss is cross-entropy against the known overlap mapping. Around that task sit three adaptations: SatMAE-style channel grouping (channels partitioned into groups, each with its own patch embedding), same-group attention masking (a binary mask that forbids attention within a channel group in both self- and cross-attention blocks), and group sampling (one group per spatial position, giving a ${\times}12.2$ FLOP reduction with almost no accuracy loss). A patch cluster prediction loss with Sinkhorn-Knopp pseudo-labels is added with equal weight. The mechanism is supposed to force the encoder to relate information across modalities rather than reconstruct from within one modality.

What would settle it

Train the same multimodal pretraining pipeline with the relative-position prediction head and the cross-attention block removed entirely, keeping cluster prediction, channel grouping, same-group masking, and group sampling, then fine-tune on Sen1Floods11; if flood IoU stays near 74.62, the paper's central claim that position prediction is the source of the gain is false.

Watch

Extended reading notes

Core claim

The central claim is that the right pretext task for satellite-image semantics is location, not reconstruction: a query patch should know where it sits relative to a reference view, and forcing that spatial reasoning produces representations better suited to pixel-level classification than forcing the model to rebuild masked pixels. The paper realizes this by concatenating modalities along the channel axis, partitioning channels into groups, giving each group its own patch embedding, and adding a mask that stops patches from attending to other patches in their own group, so the model must combine cross-modal evidence. The reported best model masks 100% of the reference view, so query patches never actually see reference representations; the paper states this means the cross-attention block can be dropped without hurting accuracy and still attributes the gain to the position-prediction setup plus its auxiliary cluster-prediction objective.

Load-bearing premise

The paper's headline result comes from a configuration in which the reference view is fully masked and position prediction accuracy is about 1.57%, near chance, so the claim that position prediction drives the gain rests on the untested assumption that the other components, namely cluster prediction, channel grouping, and attention masking, do not by themselves produce the entire improvement.

Editorial extensions

If this is right

  • A position-prediction pretraining objective can replace masked reconstruction as the default self-supervised pretraining for multimodal satellite segmentation, at least on flood mapping.
  • Channel grouping by modality is the mechanism that makes cross-modal interaction happen; grouping bands together or mixing modalities into one group removes the gain.
  • Same-group attention masking is most effective when the reference-masking ratio is low ($\eta=60\%$), improving flood IoU by 1.89 points, and is no longer needed at $\eta=100\%$.
  • The best configuration is also the cheapest: with full reference masking the cross-attention block is unnecessary, and group sampling cuts pretraining FLOPs by a factor of 12.2 with a 0.03 mIoU penalty.
  • Patch cluster prediction is a substantial part of the recipe, contributing +1.77 mIoU when it is removed.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because the top-scoring configuration masks the full reference view, the reference branch contributes no information; a straightforward ablation that deletes the position-prediction head and cross-attention block while keeping cluster prediction, channel grouping, and same-group masking would determine whether position prediction is actually the cause of the reported gain, and that ablation is not
  • The near-chance position accuracy at $\eta=100\%$ suggests the 'Best' model is effectively a cluster-prediction plus masking pretraining, so the comparison with reconstruction baselines may support dense cluster-based self-supervised learning over reconstruction rather than position prediction per se.
  • A testable extension would vary which modality is grouped with which, such as pairing DEM with multispectral imagery instead of SAR, to see whether the optimal grouping follows a measurable principle like spatial-resolution similarity or sensor geometry.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. This paper adapts LOCA, a relative-patch-position-prediction self-supervised learning method, to multimodal satellite imagery by extending SatMAE-style channel grouping to MSI, SAR, and DEM, and by adding same-group attention masking, group sampling, and a patch cluster prediction loss. The authors pretrain on a 300,000-sample subset of MMEarth and finetune on Sen1Floods11 for flood segmentation. They report that their method, 'Satellite LOCA', outperforms MMEarth, ScaleMAE, SatMAE++, and SatMAE, and they conclude that position prediction, when properly adapted for multimodal satellite imagery, learns representations more effective for semantic segmentation than reconstruction-based approaches.

Significance. If the claims were supported, this would be a useful contribution: it is among the first applications of position prediction to multimodal satellite self-supervised learning, and it includes systematic ablations of channel grouping, group sampling, same-group attention masking, and cluster prediction, with source code released. The computational-efficiency result from group sampling (a 12.2x FLOP reduction at a cost of only 0.03 mIoU) is a concrete strength. However, the central comparative claim is not currently supported because the best configuration is selected on the evaluation benchmark and the Table 7 baselines differ in architecture, input modalities, and selection protocol. The significance is therefore potential rather than demonstrated.

major comments (3)
  1. [Section 4, Tables 2 and 4] The 'Best' group setting is defined as the setting that yields the best performance on Sen1Floods11, and this same setting is then used for the headline comparison in Table 7. Since Sen1Floods11 is the only downstream evaluation set, this constitutes test-set-based model selection; the reported 74.62 IoU and 85.49 mIoU are the maximum over the explored configurations rather than a fair estimate of expected performance. The authors should use a held-out validation split or nested cross-validation for configuration selection and report test results for the selected configuration, ideally with variance across finetuning runs.
  2. [Table 7] The comparison is not controlled: Satellite LOCA uses a ViT-Small encoder with Sentinel-1, Sentinel-2, and DEM, while MMEarth uses a ConvNext-T encoder with Sentinel-1 and Sentinel-2 only, and the modality sets for ScaleMAE, SatMAE++, and SatMAE are not specified. The observed gains could be due to the additional DEM channel or to architecture differences rather than to the position-prediction objective. To support the abstract's claim, the reconstruction baselines must be pretrained with the same encoder, the same modalities, the same pretraining data and budget, and the same model-selection protocol. In addition, Table 7 reports a single finetuning run, which contradicts Section 3's statement that results are averaged over three runs.
  3. [Section 3, 'Masking reference patches'; Tables 4 and 5] The best configuration uses a reference masking ratio of eta=100%, where position-prediction accuracy is 1.57% and the paper itself states 'there is no need for the query patches to look at the reference view representations.' At this setting the position-prediction loss is effectively unusable, so the improvement over reconstruction baselines cannot be attributed to the position-prediction task. Table 6 shows that the cluster-prediction loss alone contributes +1.77 mIoU, and Tables 4 and 5 show that channel grouping and same-group attention masking also contribute. The paper needs an experiment that isolates the position-prediction loss, for example by varying eta over values where the task is solvable and by ablating the position loss while keeping the other components fixed, before the causal claim in the abstract is justified.
minor comments (4)
  1. [Section 3, equation] The word 'maatrices' in the attention-masking equation should be 'matrices'.
  2. [Table 2 caption] The caption reads 'Sen1Food11'; this should be 'Sen1Floods11'.
  3. [Section 4] There are spelling errors: 'transponse convolution' should be 'transpose convolution', and 'incoporating' should be 'incorporating'.
  4. [Section 3, 'Training and Evaluation' vs. Table 7 caption] The text says evaluation results are averaged over three runs, while Table 7 says results are from a single finetuning run; please reconcile this inconsistency.

Circularity Check

1 steps flagged · score 4.0 of 10

No derivational circularity in the loss equations, but the headline comparison is partially circular: the 'Best' group configuration is selected on the Sen1Floods11 benchmark and then reported as evidence that position prediction outperforms reconstruction.

  1. fitted input called prediction [Section 4 ('Adding DEM modality as a channel group', Table 4) and Table 7]
    "The “Best” group setting is the one that yields the best performance on Sen1Floods11. It separates the MSI and SAR modalities but mixes DEM into SAR. ... Tab. 7 shows that our adopted LOCA method does significantly better than the other methods on satellite imagery semantic segmentation on the Sen1Floods11 dataset."

    The configuration used for the headline comparison is explicitly chosen by maximizing IoU/mIoU on the same Sen1Floods11 benchmark that Table 7 uses to claim superiority over reconstruction-based methods. Thus the reported Satellite LOCA result is the outcome of a selection procedure on the evaluation set, not an independent prediction of transfer performance. Reconstruction baselines are not given the same per-dataset configuration selection, so the numerical advantage is partly an artifact of test-set selection rather than evidence about the position-prediction objective. This is a fitted-input-called-prediction circularity for the headline numerical claim, although it does not by itself make the position-prediction concept definitionally equivalent to the result.

full rationale

The paper is an empirical study, not a derivation, and the loss equations (1) and (2) are not constructed to force the downstream IoU values. There are no load-bearing self-citations or imported uniqueness theorems. The only genuine circularity concern is the way the 'Best' group setting is defined and then reused: the manuscript states that 'Best' is the setting with the best Sen1Floods11 performance, and the Table 7 comparison then presents that same test-set-tuned configuration as demonstrating the superiority of position prediction over reconstruction. This is a mild form of fitting the reported result to the benchmark, which I score as partial circularity. Separately, the paper undermines its own causal attribution by reporting position-prediction accuracy of only 1.57% at eta=100% and by stating 'there is no need for the query patches to look at the reference view representations'; these are evidence-quality problems rather than circularity, as is the uncontrolled comparison across encoders and modalities in Table 7.

Assumptions & free parameters 3 free parameters · 3 assumptions · 0 invented entities

The central empirical result depends on several free hyperparameters, the most important being the channel group layout and the reference masking ratio, both tuned on the evaluation benchmark. The paper also implicitly assumes that the pretraining and evaluation datasets are representative and that the position prediction signal remains useful even when fully masked, which its own numbers contradict.

free parameters (3)
  • reference masking ratio eta = 100% (best)
    Chosen by evaluating settings on the downstream Sen1Floods11 benchmark; at 100% the position prediction signal is destroyed and accuracy collapses to 1.57%.
  • channel group configuration ('Best') = {(B1,B2),(B3,B7),(B4,B8A),(B11),(DEM,A-VV,A-VH,D-VH),(A-HH,A-HV,D-VV,D-HH)}
    Selected as the grouping that gives the highest mIoU on Sen1Floods11, so it is fitted to the evaluation data.
  • number of pretraining samples = 300,000 from MMEarth
    Chosen to reduce pretraining time; affects all methods equally but may limit comparison quality.
assumptions (3)
  • domain assumption MMEarth is a suitable pretraining source and Sen1Floods11 is a valid transfer benchmark for satellite semantic segmentation.
    The entire evaluation rests on transfer from MMEarth to Sen1Floods11; if this transfer is not representative, results do not generalize.
  • ad hoc to paper It is acceptable to select model hyperparameters using the target evaluation set and then report that configuration as the method's result.
    The 'Best' group setting is chosen by downstream performance, which makes the headline comparison an optimistic estimate.
  • ad hoc to paper The position prediction loss remains a meaningful training signal even when the reference view is fully masked (eta=100%).
    The paper reports best performance at eta=100% where position prediction accuracy is near chance, so the benefit must come from another component, contradicting the stated mechanism.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation." pith.science (2026). https://pith.science/paper/Q4ZUZBVR

@misc{pith2026250606852,
  author       = {Pith},
  title        = {Pith review of: Position Prediction Self-Supervised Learning for Multimodal Satellite Imagery Semantic Segmentation},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/Q4ZUZBVR}},
  note         = {Machine review of arXiv:2506.06852}
}
read the original abstract

Semantic segmentation of satellite imagery is crucial for Earth observation applications, but remains constrained by limited labelled training data. While self-supervised pretraining methods like Masked Autoencoders (MAE) have shown promise, they focus on reconstruction rather than localisation-a fundamental aspect of segmentation tasks. We propose adapting LOCA (Location-aware), a position prediction self-supervised learning method, for multimodal satellite imagery semantic segmentation. Our approach addresses the unique challenges of satellite data by extending SatMAE's channel grouping from multispectral to multimodal data, enabling effective handling of multiple modalities, and introducing same-group attention masking to encourage cross-modal interaction during pretraining. The method uses relative patch position prediction, encouraging spatial reasoning for localisation rather than reconstruction. We evaluate our approach on the Sen1Floods11 flood mapping dataset, where it significantly outperforms existing reconstruction-based self-supervised learning methods for satellite imagery. Our results demonstrate that position prediction tasks, when properly adapted for multimodal satellite imagery, learn representations more effective for satellite image semantic segmentation than reconstruction-based approaches.

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

24 extracted references · 9 canonical work pages

  1. [1]

    Ankit, Oehmcke Stefan, Belongie Serge, Igel Christian, Lang Nico Nedungadi Vishal, and Kariryaa. 2025. MMEarth: Exploring Multi-modal Pretext Tasks for Geospatial Representation Learning. In Computer Vision – ECCV 2024 (Cham), Elisa, Roth Stefan, Russakovsky Olga, Sattler Torsten, Varol Gül Leonardis Aleš, and Ricci (Eds.). Springer Nature Switzerland, 164–182

    work page 2025

  2. [2]

    Kumar Ayush, Burak Uzkent, Chenlin Meng, Kumar Tanmay, Marshall Burke, David Lobell, and Stefano Ermon. 2021. Geography-Aware Self- Supervised Learning. In 2021 IEEE/CVF International Conference on Computer Vision (ICCV) . 10161–10170. doi:10.1109/ICCV48922.2021.01002

    arXiv 2021

  3. [3]

    Tadas Baltrusaitis, Chaitanya Ahuja, and Louis-Philippe Morency. 2019. Multimodal Machine Learning: A Survey and Taxonomy. IEEE Trans. Pattern Anal. Mach. Intell. 41, 2 (Feb. 2019), 423–443. doi:10.1109/TPAMI.2018.2798607

    arXiv 2019

  4. [4]

    Derrick Bonafilia, Beth Tellman, Tyler Anderson, and Erica Issenberg. 2020. Sen1Floods11: a georeferenced dataset to train and test deep learning flood algorithms for Sentinel-1. 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) 2020-June (6 2020), 835–845. doi:10.1109/CVPRW50498.2020.00113

    arXiv 2020

  5. [5]

    Mathilde Caron, Neil Houlsby, and Cordelia Schmid. 2024. Location-Aware Self-Supervised Transformers for Semantic Segmentation. Proceedings - 2024 IEEE Winter Conference on Applications of Computer Vision, W ACV 2024 (1 2024), 116–126. doi:10.1109/WACV57701.2024.00019

    arXiv 2024

  6. [6]

    Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David B Lobell, and Stefano Ermon. 2022. SatMAE: Pre-training Transformers for Temporal and Multi-Spectral Satellite Imagery. Advances in Neural Information Processing Systems 35 (12 2022), 197–211. https://sustainlab-group.github.io/SatMAE/

    work page 2022

  7. [7]

    Jacob Devlin, Ming-Wei Chang, Kenton Lee, Kristina Toutanova Google, and A I Language. 2019. BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding. Proceedings of the 2019 Conference of the North (2019), 4171–4186. doi:10.18653/V1/N19-1423 Manuscript submitted to ACM 10 Waithaka et al

    doi:10.18653/v1/n19-1423 2019

  8. [8]

    Carl Doersch, Abhinav Gupta, and Alexei A. Efros. 2015. Unsupervised Visual Representation Learning by Context Prediction. In 2015 IEEE International Conference on Computer Vision (ICCV) . 1422–1430. doi:10.1109/ICCV.2015.167

    work page doi:10.1109/iccv.2015.167 2015

Show all 24 references
  1. [9]

    William Emery and Adriano Camps. 2017. Introduction to Satellite Remote Sensing . Elsevier. https://www.sciencedirect.com/book/9780128092545/ introduction-to-satellite-remote-sensing

    2017

  2. [10]

    Atkinson, and Jon Atli Benediktsson

    Pedram Ghamisi, Behnood Rasti, Naoto Yokoya, Qunming Wang, Bernhard Hofle, Lorenzo Bruzzone, Francesca Bovolo, Mingmin Chi, Katharina Anders, Richard Gloaguen, Peter M. Atkinson, and Jon Atli Benediktsson. 2019. Multisource and multitemporal data fusion in remote sensing: A co...

    2019

  3. [11]

    Astruc Guillaume, Gonthier, Nicolas, Mallet Clement, and Landrieu Loic. 2025. OmniSat: Self-supervised Modality Fusion for Earth Observation. In Computer Vision – ECCV 2024 (Cham), Elisa, Roth Stefan, Russakovsky Olga, Sattler Torsten, Varol Gül Leonardis Aleš, and Ricci (Eds....

    2025

  4. [14]

    Zhengtao Li, Guokun Chen, and Tianxu Zhang. 2020. A CNN-Transformer Hybrid Approach for Crop Classification Using Multitemporal Multisensor Images. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing 13 (2020), 847–858. doi:10.1109/JSTARS.2020.2971763

    2020

  5. [15]

    Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwar, Salman Khan, and Fahad Shahbaz Khan. 2024. Rethinking Transformers Pre-training for Multi-Spectral Satellite Imagery. 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (6 2024), 278...

    2024

  6. [16]

    Mehdi Noroozi and Paolo Favaro. 2016. Unsupervised Learning of Visual Representations by Solving Jigsaw Puzzles. In Computer Vision – ECCV 2016, Bastian Leibe, Jiri Matas, Nicu Sebe, and Max Welling (Eds.). Springer International Publishing, Cham, 69–84

    2016

  7. [17]

    Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell

    Colorado J. Reed, Ritwik Gupta, Shufan Li, Sarah Brockman, Christopher Funk, Brian Clipp, Kurt Keutzer, Salvatore Candido, Matt Uyttendaele, and Trevor Darrell. 2023. Scale-MAE: A Scale-Aware Masked Autoencoder for Multiscale Geospatial Representation Learning. Proceedings of ...

    2023

  8. [18]

    Maofeng Tang, Andrei Cozma, Konstantinos Georgiou, Hairong Qi, and Min H Kao. 2023. Cross-Scale MAE: A Tale of Multiscale Exploitation in Remote Sensing. Advances in Neural Information Processing Systems 36 (12 2023), 20054–20066

    2023

  9. [19]

    Gabriel Tseng, Ruben Cartuyvels, Ivan Zvonkov, Mirali Purohit, David Rolnick, and Hannah Kerner. 2024. Lightweight, Pre-trained Transformers for Remote Sensing Timeseries. arXiv:2304.14065 [cs.CV] https://arxiv.org/abs/2304.14065

    2024 arXiv

  10. [20]

    Green, Evan Shelhamer, Hannah Kerner, and David Rolnick

    Gabriel Tseng, Anthony Fuller, Marlena Reil, Henry Herzog, Patrick Beukema, Favyen Bastani, James R. Green, Evan Shelhamer, Hannah Kerner, and David Rolnick. 2025. Galileo: Learning Global & Local Features of Many Remote Sensing Modalities. arXiv:2502.09356 [cs.CV] https: //ar...

    2025 arXiv

  11. [21]

    Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Ł ukasz Kaiser, and Illia Polosukhin. 2017. Attention is All you Need. In Advances in Neural Information Processing Systems, I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. V...

    2017

  12. [22]

    Peng Xu, Xiatian Zhu, and David A. Clifton. 2023. Multimodal Learning With Transformers: A Survey. IEEE Transactions on Pattern Analysis and Machine Intelligence 45, 10 (10 2023), 12113–12132. doi:10.1109/TPAMI.2023.3275156

    2023

  13. [23]

    Shuangfei Zhai, Navdeep Jaitly, Jason Ramapuram, Dan Busbridge, Tatiana Likhomanenko, Joseph Y Cheng, Walter Talbott, Chen Huang, Hanlin Goh, and Joshua M Susskind. 2022. Position Prediction as an Effective Pretraining Strategy. In Proceedings of the 39th International Confere...

    2022

  14. [24]

    Cheng Zhang, Wanshou Jiang, Yuan Zhang, Wei Wang, Qing Zhao, and Chenjie Wang. 2022. Transformer and CNN Hybrid Deep Neural Network for Semantic Segmentation of Very-High-Resolution Remote Sensing Imagery. IEEE Transactions on Geoscience and Remote Sensing 60 (2022). doi:10.11...

    2022

  15. [25]

    Bolei Zhou, Hang Zhao, Xavier Puig, Sanja Fidler, Adela Barriuso, and Antonio Torralba. 2017. Scene Parsing through ADE20K Dataset. In2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) . 5122–5130. doi:10.1109/CVPR.2017.544

    2017 doi

  16. [26]

    Adrian Ziegler and Yuki M. Asano. 2022. Self-Supervised Learning of Object Parts for Semantic Segmentation . In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) . IEEE Computer Society, Los Alamitos, CA, USA, 14482–14491. doi:10.1109/CVPR52688.2022.01...

    2022

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.