Pith. sign in

REVIEW 5 major objections 6 minor 1 cited by

MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks

T0 review · 5 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read Pre-training a shared ViT encoder to reconstruct six Earth-observation modalities—four band-grouped Sentinel-2 targets plus elevation and land-cover labels—transfers better to downstream EO classification and segmentation than prior…

desk verdict A solid incremental adaptation of MultiMAE to Earth Observation, where the flexibility result is credible but the superiority claim is confounded by the segmentation-label pre-training modality and an incomplete baseline comparison. read the letter →

arxiv 2505.14951 v1 pith:HJAA5WK3 submitted 2025-05-20 cs.CV

Jose Sosa , Danila Rukhovich , Anis Kacem , Djamila Aouada This is my paper
classification cs.CV
keywords multi-modalpre-trainingmaskedautoencodersEarthobservationSentinel-2transferlearningself-supervisedlandcoverclassificationimagesegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper tries to establish that multi-modal, multi-task masked autoencoding is a flexible and effective pre-training recipe for Earth observation models. It adapts the MultiMAE architecture—one Vision Transformer encoder shared across inputs, with a separate lightweight decoder per modality—and pre-trains it on 1.24 million MMEarth samples to reconstruct six targets: four groups of Sentinel-2 spectral bands, elevation, and land-cover segmentation labels. The paper claims that the resulting encoder transfers better than single-modality and contrastive baselines (SatMAE, MAE, CROMA, DOFA) on GEO-Bench classification and segmentation tasks, and that it can be fine-tuned with any subset of the pre-training modalities, including only RGB, without retraining. If the claim holds, one pre-trained encoder could serve many downstream EO tasks with differing data availability, avoiding modality-specific pre-training for each setup.

What carries the argument

The central object is the multi-modal, multi-task masked autoencoder built on a ViT-B encoder with patch size 8. Each of the six modalities gets its own linear projection and modality embedding, and visible tokens from all modalities are concatenated into one sequence for the shared encoder; six shallow decoders then reconstruct the masked patches, each using a cross-attention layer that pulls information from tokens of all modalities. Masking drops 5/6 of tokens per modality, with the visible tokens sampled from a symmetric Dirichlet distribution so no modality is favored. The pre-training objective is the average MSE reconstruction loss across the six decoders. The band-grouping design is what carries the flexibility claim: because each Sentinel-2-derived band group is its own input modality, fine-tuning can simply discard an unavailable modality rather than replicate bands or train a separate model.

What would settle it

Pre-train the same MultiMAE on MMEarth with the SEG target removed or replaced by non-semantic random labels, keeping RGB, IRED, SIRED, EB, and DEPTH, and run the paper's linear-probing and fine-tuning evaluation on m-eurosat, m-SA-crop-type, and m-cashew-plantation. If the scores stay at the reported level, the multi-modal reconstruction drives the transfer; if they fall to the SatMAE/DOFA range, the WorldCover land-cover labels carried the semantic signal that explains the gains.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central discovery is that splitting the ten most-used Sentinel-2 bands into four modality groups—RGB (B4, B3, B2), IRED (B5, B6, B7), SIRED (B11, B12), and EB (B8, B8A)—and pre-training a MultiMAE to reconstruct those plus DEPTH (Aster DEM elevation) and SEG (ESA WorldCover land-cover labels) yields a ViT-B encoder whose linear-probe and end-to-end fine-tuned top-1 accuracies exceed prior EO pre-training methods on the GEO-Bench classification datasets, fMoW, and EuroSAT, and whose frozen-encoder and fine-tuned mIoU exceed or match them on segmentation benchmarks. The paper further claims that fine-tuning the same encoder works when only a subset of the pre-training modalities is available: single-modality RGB fine-tuning loses only a few accuracy points on classification compared with all four S2-derived modalities, while segmentation degrades more, suggesting modality count matters most for dense prediction.

Load-bearing premise

The load-bearing premise is that the transfer gains come from multi-modal, multi-task pre-training rather than from the semantic labels included in that pre-training: Section 3.3 adds a SEG modality built from ESA WorldCover land-cover labels as one of six reconstruction targets, and the downstream benchmarks in Tables 1 and 2 are themselves land-cover and crop-type classification and segmentation, so an ablation that removes or randomizes SEG is needed to keep the multi-modality explanation intact.

Editorial extensions

If this is right

  • A single pre-trained encoder covers both classification and segmentation across the evaluated datasets, so downstream users do not need a separate pre-trained model per sensor-band configuration.
  • Fine-tuning with only RGB stays close to full multi-spectral performance on classification, so the model remains usable when only optical data are available.
  • Fine-tuning on the multi-temporal crop dataset with RGB, IRED, and pseudo-depth beats the original Prithvi baseline, indicating that the pre-trained alignment of depth with other modalities transfers to unseen data.
  • The consistent wins over DOFA, CROMA, and SatMAE++ on the tested benchmarks suggest that reconstructing several semantically distinct EO modalities in one MAE transfers better than reconstructing a single spectral stack or using contrastive objectives.
  • The same recipe can be extended to more MMEarth modalities, such as SAR or text, as the authors suggest for future work.
  • The same recipe can be extended to more MMEarth modalities, such as SAR or text, as the authors suggest for future work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper never ablates the SEG target; since ESA WorldCover labels are land-cover semantics and the downstream tasks are land-cover and crop-type semantics, part of the transfer gain may come from that label supervision rather than from multi-modality per se, and an ablation with SEG removed would settle this.
  • The band-grouping recipe is directly portable: any downstream dataset can map its available bands into the pre-trained RGB/IRED/SIRED/EB slots, and missing slots are simply dropped, which suggests a practical standardization for EO transfer learning that the paper does not fully formalize.
  • A natural test is to swap SEG for a non-semantic modality already in MMEarth, such as SAR backscatter, and see whether the gains persist; if they do, the claim is about modality diversity, not label leakage.
  • The symmetric Dirichlet masking balances modalities in expectation, but the paper does not test skewed masking; a testable extension would be modality-weighted masking to boost performance on datasets that resemble a particular modality.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 6 minor

Summary. The paper adapts the MultiMAE architecture to Earth observation data. It pre-trains a ViT-B encoder on 1.24 million samples from MMEarth using six modalities: four Sentinel-2 band groups (RGB, IRED, SIRED, EB), Aster-DEM elevation (DEPTH), and ESA WorldCover land-cover labels (SEG). The model reconstructs masked patches of all six modalities via six task-specific decoders. Transfer is evaluated by linear probing and fine-tuning on GEO-Bench classification and segmentation datasets, plus EuroSAT, fMoW, and a multi-temporal crop segmentation dataset. The paper also reports fine-tuning with fewer modalities, including RGB-only, to demonstrate flexibility.

Significance. If the reported results are reproducible and the gains are attributable to the multi-modal multi-task objective, the work would provide a practical recipe for pre-training flexible EO foundation models that can be fine-tuned with arbitrary subsets of Sentinel-2 bands. The paper's core experimental design—splitting S2 bands into four modalities and adding elevation and segmentation—is sensible, and the flexibility experiments (Table 3 and Table 4) are a useful addition. However, the headline claim of consistent improvement over state-of-the-art is not supported by the current evidence because the pre-training uses supervised land-cover labels (SEG) that baselines lack, the comparison matrix is incomplete, and no variance estimates are reported. The contribution is therefore promising but requires substantial additional validation.

major comments (5)
  1. [Section 3.3 / Appendix Table 2 / Eq. (1)] The SEG pre-training modality consists of ESA WorldCover categorical land-cover labels, and in the MultiMAE architecture these labels are linearly projected and passed through the shared encoder, in addition to being a reconstruction target. Downstream tasks (land-cover classification, crop-type segmentation) are semantically aligned with these labels, so the observed improvements over SatMAE, CROMA, and DOFA—which do not see any such labels—may be due to label supervision rather than to the multi-modal, multi-task masked autoencoder objective. No ablation is provided that removes SEG or replaces it with an unsupervised counterpart (e.g., another spectral modality), so the central attribution claim is not established.
  2. [Section 4.3 / Tables 1 and 2] The claim that 'our approach consistently outperforms previous methods on all the GEO-Bench datasets under both settings' (Section 4.3) is not supported by the tables. Many cells are empty (dashes), so for m-bigearthnet no non-MMEarth baseline is reported, and for m-eurosat and m-brick-kiln no fine-tuning results for baselines appear. Even on the filled cells the claim is contradicted: on EuroSAT (Table 1, columns 11-12), our LP/FT values (96.20/99.11) are below CROMA (97.59/99.22) and SatMAE (96.61/99.20). The comparison matrix needs to be completed or the claim needs to be qualified.
  3. [Tables 1-4 / Section 4.2] All reported numbers appear to come from single runs without error bars or multiple seeds. Many differences are small (e.g., sub-percent gaps on EuroSAT), and several fine-tuning datasets are small (e.g., 50 test images for m-cashew-plantation, Appendix Table 3). Without variance estimates or at least two seeds, the indicated improvements cannot be distinguished from optimization noise.
  4. [Section 4.3 / Tables 1-2] The comparisons against MMEarth variants are not controlled: those baselines use a ConvNeXt V2 backbone and a two-stage fine-tuning strategy, so performance differences cannot be attributed to the pre-training objective. The paper lacks a same-architecture, same-data baseline (e.g., a ViT-B MAE pre-trained on the S2 bands without SEG), which is necessary to isolate the effect of the multi-modal multi-task design.
  5. [Section 5 (Conclusions and limitations)] The limitations section does not acknowledge the SEG label confound or the incompleteness of the comparison matrix. Since these issues directly qualify the abstract's claims, they should be discussed explicitly.
minor comments (6)
  1. [Figure 1 caption] The caption contains a typo: 'Pre-traning' should be 'Pre-training'.
  2. [Table 3] The row labels 'RGB' and 'S2' are ambiguous; 'S2' here means the four S2-derived modalities, but S2 is also used for the dataset name. Use 'RGB-only' and 'All four S2 modalities'.
  3. [References] Reference [3] has a corrupted author list ('Hanxi (Steve) Kordi, Fatemehand Li'); please fix the bibliography.
  4. [Section 4.3 / Table 4] The Prithvi baseline is not described; it is unclear whether it uses the same input bands, fine-tuning budget, or segmentation head. A short setup description is needed.
  5. [Figure 4 caption] The caption contains a typo: 'Segmetation' should be 'Segmentation'.
  6. [Section 2] The related work briefly mentions DOFA and CROMA but does not discuss their input modality flexibility; adding a sentence would help position the contribution.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: the paper is an empirical transfer-learning study; the reported gains are measurements over held-out data, not consequences of a fitted equation or a self-citation chain.

full rationale

The paper's derivation chain is not circular. It adopts the MultiMAE architecture [1] and MAE-style reconstruction loss (Appendix Eq. 1) as external, prior-work components, modifies the modality set, and then measures transfer by linear probing and fine-tuning on GEO-Bench, fMoW, EuroSAT, and segmentation benchmarks. No model parameter is fitted to the evaluation results and then renamed as a prediction; the flexibility claim is supported by ablations that drop modalities at fine-tuning time (Table 3, Table 4), not by construction. The only noteworthy issue is the SEG pre-training modality: ESA WorldCover land-cover labels are used as both input and reconstruction target, and downstream tasks are also semantic, which may confound the comparison against baselines that never receive such labels. This is a fairness/validity concern and a missing limitation note, not circularity, because the downstream labels are not identical to WorldCover labels and the superiority claim rests on reported empirical numbers rather than an equation that reduces to its inputs. The self-citation [20] in Section 2 motivates the flexibility problem but is not load-bearing for any result in this paper.

Assumptions & free parameters 2 free parameters · 3 assumptions · 0 invented entities

The paper is an empirical study with no derivation, so there are no fitted constants of the physics-derivation type. The central claim depends on standard training hyperparameters and on domain assumptions about band grouping, dataset alignment, and the multi-task loss. No new entities, forces, or conserved quantities are introduced.

free parameters (2)
  • mask ratio = 5/6
    Chosen following MultiMAE [1], Section 3.1. The masking fraction affects reconstruction difficulty and is not tuned per dataset.
  • fine-tuning input size = 96x96 for classification, 256x256 for segmentation
    Section 4.3. Fixed to align with prior work; influences patch sequence length and downstream accuracy.
assumptions (3)
  • domain assumption Sentinel-2 bands can be meaningfully grouped into four modalities: RGB, IRED, SIRED, and EB.
    Section 3.2: the paper chooses these groupings partly following SatMAE++ [19], arguing that they align with available EO datasets. If the groups were semantically poor, the pre-training signal would weaken.
  • domain assumption MMEarth provides correctly aligned multi-modal samples across Sentinel-2, Aster DEM, and ESA WorldCover.
    Section 4.1 and Appendix 1.2: pre-training relies on 1.24M samples from MMEarth [18]. Misalignment or missing data in the dataset would propagate to the pre-trained representations.
  • domain assumption Averaging per-modality MSE reconstruction losses without weights is a valid multi-task objective.
    Appendix 2.1, Equation 1: the total loss is a simple average over six modalities, assuming equal contribution of each modality to the shared representation.

how reviews work

0 comments
Cite this review

Pith. "Pith review of MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks." pith.science (2026). https://pith.science/paper/HJAA5WK3

@misc{pith2026250514951,
  author       = {Pith},
  title        = {Pith review of: MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencoders for Earth Observation Tasks},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/HJAA5WK3}},
  note         = {Machine review of arXiv:2505.14951}
}
read the original abstract

Multi-modal data in Earth Observation (EO) presents a huge opportunity for improving transfer learning capabilities when pre-training deep learning models. Unlike prior work that often overlooks multi-modal EO data, recent methods have started to include it, resulting in more effective pre-training strategies. However, existing approaches commonly face challenges in effectively transferring learning to downstream tasks where the structure of available data differs from that used during pre-training. This paper addresses this limitation by exploring a more flexible multi-modal, multi-task pre-training strategy for EO data. Specifically, we adopt a Multi-modal Multi-task Masked Autoencoder (MultiMAE) that we pre-train by reconstructing diverse input modalities, including spectral, elevation, and segmentation data. The pre-trained model demonstrates robust transfer learning capabilities, outperforming state-of-the-art methods on various EO datasets for classification and segmentation tasks. Our approach exhibits significant flexibility, handling diverse input configurations without requiring modality-specific pre-trained models. Code will be available at: https://github.com/josesosajs/multimae-meets-eo.

Figures

Figures reproduced from arXiv: 2505.14951 by the authors.

Figure 1
Figure 1. Pre-traning and fine-tuning stages of our MultiMAE [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 2
Figure 2. MultiMAE pre-training with EO data. Patches are randomly sampled from six input modalities from EO data, RGB, IRED, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 1
Figure 1. Spatial and temporal distribution of MMEarth dataset. [PITH_FULL_IMAGE:figures/full_fig_p008_1.png] view at source ↗
Figures from the paper (5 more)
Figure 2
Figure 2. Figure 2: MultiMAE pre-training and fine-tuning with EO data. The top part of the figure illustrates the pre-training stage with six input [PITH_FULL_IMAGE:figures/full_fig_p009_2.png]
Figure 3
Figure 3. Figure 3: Decoders design. The tokens from the encoder are [PITH_FULL_IMAGE:figures/full_fig_p010_3.png]
Figure 5
Figure 5. Figure 5: Visualisation of reconstructions across different input [PITH_FULL_IMAGE:figures/full_fig_p010_5.png]
Figure 4
Figure 4. Figure 4: Fine-tuning setups for segmentation and classification [PITH_FULL_IMAGE:figures/full_fig_p010_4.png]
Figure 6
Figure 6. Figure 6: Visualisations for segmentation tasks. The figure visu [PITH_FULL_IMAGE:figures/full_fig_p011_6.png]

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. Using Multiple Input Modalities Can Improve Data-Efficiency and O.O.D. Generalization for ML with Satellite Imagery

    cs.CV 2025-07 conditional novelty 6.0 of 10

    Adding auxiliary geographic data layers to satellite imagery improves label efficiency and out-of-sample generalization across four SatML tasks, with frozen or hand-coded fusion beating fine-tuned variants.

Reference graph

Works this paper leans on

34 extracted references · 26 canonical work pages · cited by 1 Pith paper

  1. [1]

    Multimae: Multi-modal multi-task masked autoen- coders

    Roman Bachmann, David Mizrahi, Andrei Atanov, and Amir Zamir. Multimae: Multi-modal multi-task masked autoen- coders. InECCV, pages 348–367. Springer, 2022. 2, 3, 5, 6

    work page 2022

  2. [2]

    Satlaspretrain: A large- scale dataset for remote sensing image understanding

    Favyen Bastani, Piper Wolters, Ritwik Gupta, Joe Ferdi- nando, and Aniruddha Kembhavi. Satlaspretrain: A large- scale dataset for remote sensing image understanding. In CVPR, pages 16772–16782, 2023. 1, 2

    work page 2023

  3. [3]

    HLS Multi Temporal Crop Classification, 2023

    Michael Cecil, Hanxi (Steve) Kordi, Fatemehand Li, Sam Khallaghi, and Hamed Alemohammad. HLS Multi Temporal Crop Classification, 2023. 5, 6, 4

    work page 2023

  4. [4]

    Satmae: Pre-training transformers for tem- poral and multi-spectral satellite imagery.NeurIPS, 35:197– 211, 2022

    Yezhen Cong, Samar Khanna, Chenlin Meng, Patrick Liu, Erik Rozi, Yutong He, Marshall Burke, David Lobell, and Stefano Ermon. Satmae: Pre-training transformers for tem- poral and multi-spectral satellite imagery.NeurIPS, 35:197– 211, 2022. 1, 2, 4, 5

    work page 2022

  5. [5]

    Imagenet: A large-scale hierarchical image database

    Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. InCVPR, pages 248–255. Ieee, 2009. 1

    2009

  6. [6]

    An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020

    Alexey Dosovitskiy. An image is worth 16x16 words: Transformers for image recognition at scale.arXiv preprint arXiv:2010.11929, 2020. 2, 3, 4

    arXiv 2010

  7. [7]

    Croma: Remote sensing representations with contrastive radar- optical masked autoencoders.NeurIPS, 36, 2024

    Anthony Fuller, Koreen Millard, and James Green. Croma: Remote sensing representations with contrastive radar- optical masked autoencoders.NeurIPS, 36, 2024. 2, 4, 5

    work page 2024

  8. [8]

    Skysense: A multi-modal remote sens- ing foundation model towards universal interpretation for earth observation imagery

    Xin Guo, Jiangwei Lao, Bo Dang, Yingying Zhang, Lei Yu, Lixiang Ru, Liheng Zhong, Ziyuan Huang, Kang Wu, Dingxiang Hu, et al. Skysense: A multi-modal remote sens- ing foundation model towards universal interpretation for earth observation imagery. InProceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 27672–27683, 2024. 1

    work page 2024

Show all 34 references
  1. [9]

    Masked autoencoders are scalable vision learners

    Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Doll´ar, and Ross Girshick. Masked autoencoders are scalable vision learners. InCVPR, pages 16000–16009, 2022. 2, 3, 4, 5

    2022

  2. [10]

    Patrick Helber, Benjamin Bischke, Andreas Dengel, and Damian Borth. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226, 2019. 4, 5, 1, 2

    2019

  3. [11]

    Masked image mod- eling: A survey.arXiv preprint arXiv:2408.06687, 2024

    Vlad Hondru, Florinel Alin Croitoru, Shervin Minaee, Radu Tudor Ionescu, and Nicu Sebe. Masked image mod- eling: A survey.arXiv preprint arXiv:2408.06687, 2024. 2

    2024 arXiv

  4. [12]

    Masked autoencoders that listen

    Po-Yao Huang, Hu Xu, Juncheng Li, Alexei Baevski, Michael Auli, Wojciech Galuba, Florian Metze, and Christoph Feichtenhofer. Masked autoencoders that listen. Advances in Neural Information Processing Systems, 35: 28708–28720, 2022. 2

    2022

  5. [13]

    Johannes Jakubik, Sujit Roy, Christopher Phillips, Paolo Fraccaro, Denys Godwin, Bianca Zadrozny, Daniel Szwar- cman, Carlos Gomes, Gabby Nyirjesy, Blair Edwards, Daiki Kimura, Naomi Simumba, Linsong Chu, S. Karthik Mukkavilli, Devyani Lambhate, Kamal Das, Ranjini Ban- galore,...

    arXiv

  6. [14]

    Geo- bench: Toward foundation models for earth monitoring

    Alexandre Lacoste, Nils Lehmann, Pau Rodriguez, Evan Sherwin, Hannah Kerner, Bj¨orn L¨utjens, Jeremy Irvin, David 6 Dao, Hamed Alemohammad, Alexandre Drouin, et al. Geo- bench: Toward foundation models for earth monitoring. NeurIPS, 36, 2024. 1, 4, 5, 2

    2024

  7. [15]

    Multimodality helps unimodality: Cross- modal few-shot learning with multimodal models

    Zhiqiu Lin, Samuel Yu, Zhiyi Kuang, Deepak Pathak, and Deva Ramanan. Multimodality helps unimodality: Cross- modal few-shot learning with multimodal models. InPro- ceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 19325–19337, 2023. 2

    2023

  8. [16]

    Re- moteclip: A vision language foundation model for remote sensing.IEEE Transactions on Geoscience and Remote Sensing, 2024

    Fan Liu, Delong Chen, Zhangqingyun Guan, Xiaocong Zhou, Jiale Zhu, Qiaolin Ye, Liyong Fu, and Jun Zhou. Re- moteclip: A vision language foundation model for remote sensing.IEEE Transactions on Geoscience and Remote Sensing, 2024. 2

    2024

  9. [17]

    A convnet for the 2020s

    Zhuang Liu, Hanzi Mao, Chao-Yuan Wu, Christoph Feicht- enhofer, Trevor Darrell, and Saining Xie. A convnet for the 2020s. InCVPR, pages 11976–11986, 2022. 5, 3

    2022

  10. [18]

    Mmearth: Explor- ing multi-modal pretext tasks for geospatial representation learning.arXiv preprint arXiv:2405.02771, 2024

    Vishal Nedungadi, Ankit Kariryaa, Stefan Oehmcke, Serge Belongie, Christian Igel, and Nico Lang. Mmearth: Explor- ing multi-modal pretext tasks for geospatial representation learning.arXiv preprint arXiv:2405.02771, 2024. 2, 4, 5, 1

    2024 arXiv

  11. [19]

    Rethinking transformers pre-training for multi- spectral satellite imagery

    Mubashir Noman, Muzammal Naseer, Hisham Cholakkal, Rao Muhammad Anwer, Salman Khan, and Fahad Shah- baz Khan. Rethinking transformers pre-training for multi- spectral satellite imagery. InCVPR, pages 27811–27819,

  12. [20]

    How effective is pre-training of large masked au- toencoders for downstream earth observation tasks?arXiv preprint arXiv:2409.18536, 2024

    Jose Sosa, Mohamed Aloulou, Danila Rukhovich, Rim Sleimi, Boonyarit Changaival, Anis Kacem, and Djamila Aouada. How effective is pre-training of large masked au- toencoders for downstream earth observation tasks?arXiv preprint arXiv:2409.18536, 2024. 2

    2024 arXiv

  13. [21]

    Ssl4eo-l: Datasets and foundation models for landsat imagery.Advances in Neural Information Processing Systems, 36:59787–59807,

    Adam Stewart, Nils Lehmann, Isaac Corley, Yi Wang, Yi- Chia Chang, Nassim Ait Ait Ali Braham, Shradha Sehgal, Caleb Robinson, and Arindam Banerjee. Ssl4eo-l: Datasets and foundation models for landsat imagery.Advances in Neural Information Processing Systems, 36:59787–59807,

  14. [22]

    Torch- geo: deep learning with geospatial data

    Adam J Stewart, Caleb Robinson, Isaac A Corley, Anthony Ortiz, Juan M Lavista Ferres, and Arindam Banerjee. Torch- geo: deep learning with geospatial data. InProceedings of the 30th international conference on advances in geographic information systems, pages 1–12, 2022. 1, 2

    2022

  15. [23]

    Con- vnext v2: Co-designing and scaling convnets with masked autoencoders

    Sanghyun Woo, Shoubhik Debnath, Ronghang Hu, Xinlei Chen, Zhuang Liu, In So Kweon, and Saining Xie. Con- vnext v2: Co-designing and scaling convnets with masked autoencoders. InCVPR, pages 16133–16142, 2023. 2

    2023

  16. [24]

    Simmim: A simple framework for masked image modeling

    Zhenda Xie, Zheng Zhang, Yue Cao, Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. InCVPR, pages 9653–9663, 2022. 2

    2022

  17. [25]

    Earthnets: Empowering ai in earth obser- vation.arXiv preprint arXiv:2210.04936, 2022

    Zhitong Xiong, Fahong Zhang, Yi Wang, Yilei Shi, and Xiao Xiang Zhu. Earthnets: Empowering ai in earth obser- vation.arXiv preprint arXiv:2210.04936, 2022. 1, 2

    2022 arXiv

  18. [26]

    Neural plasticity-inspired foundation model for observing the earth crossing modalities.arXiv e-prints, pages arXiv–2403, 2024

    Zhitong Xiong, Yi Wang, Fahong Zhang, Adam J Stewart, Jo¨elle Hanna, Damian Borth, Ioannis Papoutsis, Bertrand Le Saux, Gustau Camps-Valls, and Xiao Xiang Zhu. Neural plasticity-inspired foundation model for observing the earth crossing modalities.arXiv e-prints, pages arXiv–2...

    2024

  19. [27]

    Depth anything: Unleashing the power of large-scale unlabeled data

    Lihe Yang, Bingyi Kang, Zilong Huang, Xiaogang Xu, Jiashi Feng, and Hengshuang Zhao. Depth anything: Unleashing the power of large-scale unlabeled data. InCVPR, pages 10371–10381, 2024. 6 7 MultiMAE Meets Earth Observation: Pre-training Multi-modal Multi-task Masked Autoencode...

    2024

  20. [28]

    Data details 1.1. Sentinel-2 data Band Description Resolution Wavelength (nm) B1 Ultra blue (Aerosol) 60 443 B2 Blue 10 490 B3 Green 10 560 B4 Red 10 665 B5 Red edge 1 (near infrared) 20 705 B6 Red edge 2 (near infrared) 20 740 B7 Red edge 3 (near infrared) 20 783 B8 Near infr...

  21. [29]

    Expert knowledge incorporation

  22. [30]

    Original train, validation, and test splits

  23. [31]

    Overall, [14] comprises multiple modified versions of standard geospatial datasets for classification and segmen- tation tasks

    Permissive license. Overall, [14] comprises multiple modified versions of standard geospatial datasets for classification and segmen- tation tasks. We use a subset of those datasets as shown in Table 3. For fine-tuning on classification tasks, we add a couple of standard datas...

  24. [32]

    Pre-training objective We pre-train our approach (depicted in Figure 2) using six input modalities: RGB, IRED, SIRED, EB, DEPTH, and SEG

    Pre-training MultiMAE 2.1. Pre-training objective We pre-train our approach (depicted in Figure 2) using six input modalities: RGB, IRED, SIRED, EB, DEPTH, and SEG. Four of them come from Sentinel-2 data. We use all available samples in the MMEarth dataset as indicated by subs...

  25. [33]

    Fine-tuning setups for segmentation and classification EO tasks

    Fine-tuning setups INPUT(S) Pre-trained MultiMAE encoder ClassLinear classifier Linear probing End-to-end fine-tuning INPUT(S) Pre-trained MultiMAE encoder ClassLinear classifier Fine-tuning on classification tasks INPUT(S) Pre-trained MultiMAE encoder Frozen backbone End-to-e...

  26. [34]

    Pre-training visualisations Masked input Prediction Masked input Prediction TargetTarget RGBDEPTHSEGRGBDEPTHSEG Figure 5

    Qualitative results 4.1. Pre-training visualisations Masked input Prediction Masked input Prediction TargetTarget RGBDEPTHSEGRGBDEPTHSEG Figure 5. Visualisation of reconstructions across different input modalities. Randomly chosen reconstructions of EO input modal- ities after...

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.