REVIEW 4 major objections 6 minor 2 cited by
TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation
T0 review · 4 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash
Pith's one-line read TerraFM is a self-supervised multisensor Earth-observation model that treats Sentinel-1 radar and Sentinel-2 optical imagery as interchangeable views and reports state-of-the-art results on GEO-Bench and Copernicus-Bench.
desk verdict Solid, competitive remote sensing FM, but the 'consistently outperforms' claim and the 23T pixel count don't survive a close read; worth refereeing after cleanup. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The load-bearing machinery is a DINO-style teacher-student contrastive learner in which the three sensing modalities are treated as natural augmentations of one another. Modality-specific patch embeddings map each sensor's channels into tokens, a shared projection aligns them, and a lightweight cross-attention block with five learned spatial queries fuses tokens from different sensors at each spatial location before the shared ViT encoder. Teacher and student receive randomly chosen modality subsets, so the student must predict the teacher's output from a different sensor view. A dual-centering term subtracts both a global running mean of teacher logits and a class-frequency-aware mean computed from high-frequency land-cover classes, which prevents the representation from collapsing onto dominant categories. The model is pretrained on 534-by-534 tiles from 1.53 million globally distributed grid cells, giving broad spatial context and roughly 23 trillion pixels of pretraining data.
What would settle it
One controlled experiment would settle the claim: re-run every compared model on GEO-Bench and Copernicus-Bench under a single protocol identical to TerraFM's, with the same tile pre-processing, class splits, kNN and linear-probe layer choices, and hyperparameter grids, and check whether TerraFM still leads on the majority of tasks.
Extended reading notes
Core claim
The paper's discovery is that a contrastively trained vision transformer can absorb radar and optical sensors into one representation if the sensors are treated as views of the same location rather than separate data types. TerraFM is pretrained on 1.53 million globally distributed grid cells from Major-TOM, filtered to remove most ocean-heavy tiles and sampled with WorldCover, climate, and world-region priors. Each 534-by-534 tile passes through modality-specific patch embeddings; when multiple modalities are present, a cross-attention block with five learned spatial queries fuses their tokens before the shared encoder. During pretraining the teacher and student randomly receive different modality subsets, so the student must match the teacher's view from another sensor, and a dual-centering term subtracts both a global running mean and a high-frequency-class mean from teacher logits. On GEO-Bench and Copernicus-Bench the paper reports top results, including 95.1 percent kNN accuracy on m-EuroSAT, 73.1 percent F1 on m-BigEarthNet, and 37.2 percent mIoU on m-Cashew-Plant.
Load-bearing premise
The claimed state-of-the-art results rest on the assumption that the benchmark numbers reported for earlier models, several taken directly from Galileo [28], were obtained under the same evaluation protocol as TerraFM, including identical preprocessing, class splits, and hyperparameter selection.
Editorial extensions
If this is right
- A single TerraFM backbone can serve radar-only, optical-only, or mixed sensor inputs, because stochastic modality assignment during pretraining teaches the encoder to work with whichever sensors are available.
- Large-tile pretraining at 534-by-534 resolution gives the encoder broader spatial context, which the paper links to stronger segmentation on tasks such as crop-type and cashew-plantation mapping.
- Dual centering pushes the model toward higher-entropy, more diverse predictions for rare land-cover classes, so downstream land-cover maps should be less dominated by frequent classes like tree cover and grassland.
- TerraFM reaches its reported accuracy at lower inference cost in MACs than several larger baselines, suggesting the fusion design is a compute-efficient route to high-performing Earth-observation models.
Reading between the lines
- The dual-centering mechanism is not tied to remote sensing: any contrastive learner facing a long-tailed training distribution could reuse a frequency-aware center to keep rare classes from collapsing into frequent prototypes.
- If modality-as-augmentation is what drives the gains, then a new sensor should be adoptable by adding only a modality-specific patch embedding and a short alignment stage, avoiding a full retraining run.
- The pretraining data curation, which discards most ocean tiles and samples across land-cover, climate, and world-region priors, implies that where training tiles come from may matter as much as how many there are; an experiment that varies only the sampling prior would quantify this.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper introduces TerraFM, a self-supervised vision-transformer foundation model for multimodal Earth observation, pretrained on globally distributed Sentinel-1 and Sentinel-2 (L1C/L2A) tiles from the Major-TOM dataset. The main technical contributions are modality-specific patch embeddings, a cross-attention fusion module that treats modalities as natural augmentations, and a dual-centering mechanism using WorldCover class-frequency statistics to counteract long-tailed land-cover distributions. The model is evaluated on GEO-Bench and Copernicus-Bench, where the authors claim state-of-the-art performance across classification and segmentation tasks. Code and pretrained models are publicly released.
Significance. If the reported results are reliable, TerraFM would be a useful and significant contribution: it combines a large-scale (claimed 23.32T pixels) multimodal pretraining pipeline with a clean DINO-style student-teacher objective, and the public release of code and weights would benefit the EO community. The paper also provides a useful component ablation. However, the central empirical claim of consistent state-of-the-art performance is currently undermined by cross-paper evaluation-protocol ambiguities and by internal inconsistencies in the reported training scale, so the significance is not yet fully established.
major comments (4)
- [Section 5.3, Tables 2-4] The state-of-the-art claim rests on comparisons with baseline numbers whose evaluation protocol is not shown to match that of TerraFM. Table 2 states that results other than Copernicus-FM and TerraFM are 'directly taken from [28]', while Table 4 and Table 3 do not state the provenance or evaluation protocol of the baseline numbers. Section 5.2 describes TerraFM's own kNN protocol (sweep over k, features from the last 4 transformer layers) and linear/UperNet probing protocols, but it is not established that the cited baselines used the same layer selection, k sweep, classifier training schedule, or segmentation probing setup. Since kNN accuracy and mIoU are sensitive to these choices, the claimed 'consistently outperforms' may be an artifact of protocol mismatch. The authors should either re-run the baselines under the same protocol or provide evidence that the cited numbers were obtained under an identical protocol.
- [Section 4, Table 1, Section 5.1] The reported pretraining pixel count is internally inconsistent. Section 4 states that pretraining yielded 18.7 million modality-specific training tiles, and Table 1 lists a tile size of 534×534, which gives roughly 18.7M × 534 × 534 ≈ 5.3T pixels, not 23.32T as reported in Table 1. If the 23.32T figure includes multi-crop augmentation, multiple epochs, or some other counting, this must be stated explicitly. Relatedly, Section 5.1 says the model is pretrained with an input size of 224×224, while the contributions highlight 'large spatial tiles (534)'; the relationship between the 534-pixel source tiles and the 224-pixel model input should be clarified.
- [Section 5.3, Table 3, Table 2] The narrative claim that TerraFM 'consistently outperforms existing foundation models across tasks and modalities' is contradicted by the paper's own tables. In Table 3, TerraFM obtains a lower BigEarthNet-S1 mAP than SoftCon (76.9 vs 78.7) and a lower DFC2020-S2 mIoU than CROMA (63.8 vs 66.5). In Table 2, TerraFM is below Galileo on m-So2Sat at the 1% training setting (41.1/41.6 vs 43.2). The text should be revised to describe the actual pattern of wins and losses rather than claiming uniform superiority.
- [Section 5.4, Table 5 vs Table 4] There is an inconsistency in the segmentation evaluation protocol that affects interpretation of the results. Table 5 reports m-Cashew-Plant values in the range 50.6-64.6, while Table 4 reports 34.1-37.2 for the same dataset under 'segmentation (mIoU)'. The discussion of Table 4 refers to 'segmentation (with linear probing)', but Section 5.2 describes a 'UperNet Probing Evaluation' for segmentation, not a linear-probing protocol. The authors need to clarify which protocol is used for each table and ensure that the baseline numbers in Table 4 are obtained with the same protocol.
minor comments (6)
- [Section 5.1] There is a typo in 'with a16 × 16 patch resolution'; it should read 'with a 16 × 16 patch resolution'.
- [Section 4] The parenthetical note '(Need to highlight the disk space usage, it takes around 76 terabytes to store 1.5M samples in uint16)' appears to be an incomplete author note and should be either removed or integrated into a proper sentence.
- [Table 5 caption] The caption contains a typo: 'Imapct' should be 'Impact'.
- [Section 5.1] 'where as' should be written as 'whereas'.
- [Figure 6] The x-axis label 'MACs (×10 )' appears to have a missing exponent; it should probably read 'MACs (×10^9)'.
- [Sections 4 and 5.1] The paper alternately refers to 'around 1.5 million grid cells', 'around 1.53 million multi-modal samples', and '18.7 million modality-specific training tiles'; the relationship between these counts should be stated explicitly to avoid ambiguity.
Circularity Check
No material circularity: TerraFM's pretraining is self-supervised, WorldCover enters only as an external sampling/regularization prior, and evaluation uses independent held-out benchmarks; baseline-protocol caveats are validity risks, not circular reductions.
full rationale
TerraFM's derivation chain is not circular. The pretraining objective is a DINO-style student–teacher contrastive loss (Sec. 3.2, Eq. 1 and the multi-crop cross-entropy sum) and uses no downstream labels. ESA WorldCover is used only as an external prior: it drives tile filtering (Sec. 4), LULC metadata for sampling, and the dual-centering center c_h (Sec. 3.2); none of these uses GEO-Bench or Copernicus-Bench labels. Hyperparameters are selected on validation splits, and test numbers are reported afterward (Sec. 5.2). The main caveat is comparability of baselines: Table 2 says results other than Copernicus-FM and TerraFM are 'directly taken from [28]', and Tables 3–4 do not fully document baseline protocols. A protocol mismatch could change the SOTA verdict, but this is an external-validity concern, not a prediction that reduces to its inputs by construction. Self-citations to GeoChat [16] and Panopticon [29] appear only in related-work enumerations and are not load-bearing for TerraFM's design or results. No uniqueness theorem, ansatz-smuggling, or fitted-parameter-renamed-as-prediction occurs. The parenthetical 'Need to highlight the disk space usage' note in Sec. 4 flags a manuscript polish issue, not a circular step. Overall the paper is self-contained against external benchmarks; the circularity score is 0.
Assumptions & free parameters
free parameters (4)
- alpha (dual centering balance) =
0.8
- Nq (number of cross-attention learned queries) =
5
- Input tile size =
534x534 pixels (5.34 km per side)
- Patch size =
16x16 pixels
assumptions (6)
- domain assumption Self-supervised contrastive learning on large-scale multimodal satellite data produces representations that transfer to downstream classification and segmentation tasks.
- domain assumption Treating different sensors (S1, S2-L1C, S2-L2A) as natural augmentations encourages modality-invariant features without losing task-relevant information.
- domain assumption The land cover aware sampling strategy (removing ocean tiles, sampling by WorldCover class, climate zone, and ESRI region) yields a more balanced and useful pretraining distribution.
- domain assumption Dual centering with WorldCover-derived class frequencies improves representations for rare classes without degrading common classes.
- domain assumption ESA WorldCover provides accurate global land cover labels for computing class statistics.
- domain assumption The GEO-Bench and Copernicus-Bench benchmarks are fair and the reported baseline numbers from external papers are comparable.
Cite this review
Pith. "Pith review of TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation." pith.science (2026). https://pith.science/paper/SH3HMDBJ
@misc{pith2026250606281,
author = {Pith},
title = {Pith review of: TerraFM: A Scalable Foundation Model for Unified Multisensor Earth Observation},
year = {2026},
howpublished = {\url{https://pith.science/paper/SH3HMDBJ}},
note = {Machine review of arXiv:2506.06281}
}
read the original abstract
Modern Earth observation (EO) increasingly leverages deep learning to harness the scale and diversity of satellite imagery across sensors and regions. While recent foundation models have demonstrated promising generalization across EO tasks, many remain limited by the scale, geographical coverage, and spectral diversity of their training data, factors critical for learning globally transferable representations. In this work, we introduce TerraFM, a scalable self-supervised learning model that leverages globally distributed Sentinel-1 and Sentinel-2 imagery, combined with large spatial tiles and land-cover aware sampling to enrich spatial and semantic coverage. By treating sensing modalities as natural augmentations in our self-supervised approach, we unify radar and optical inputs via modality-specific patch embeddings and adaptive cross-attention fusion. Our training strategy integrates local-global contrastive learning and introduces a dual-centering mechanism that incorporates class-frequency-aware regularization to address long-tailed distributions in land cover.TerraFM achieves strong generalization on both classification and segmentation tasks, outperforming prior models on GEO-Bench and Copernicus-Bench. Our code and pretrained models are publicly available at: https://github.com/mbzuai-oryx/TerraFM .
Figures
Forward citations
Cited by 2 Pith papers
-
Multi-Year Geospatial Reasoning using Interannually-Consistent Historical Predictions as a Free Input Modality
Feeding a crop-type model its own interannual-fixed historical predictions, encoded as confidence-scaled categorical tokens, raises crop-only F1 by 1.6 points and rebalances precision and recall.
-
OpenEarthAgent: A Unified Framework for Tool-Augmented Geospatial Agents
A 4B model fine-tuned on tool-augmented geospatial reasoning traces outperforms larger general-purpose models on executable GIS/spectral tool-use benchmarks and matches frontier models on trajectory fidelity.
Reference graph
Works this paper leans on
- [28]
-
[1]
Astruc, G., N. Gonthier, C. Mallet, and L. Landrieu (2024). Anysat: An earth observation model for any resolutions, scales, and modalities. arXiv preprint arXiv:2412.14123
arXiv 2024
-
[2]
Bastani, F., P. Wolters, R. Gupta, J. Ferdinando, and A. Kembhavi (2023). Satlaspretrain: A large-scale dataset for remote sensing image understanding
work page 2023
-
[3]
Beck, H. E., N. E. Zimmermann, T. R. McVicar, N. Vergopolan, A. Berg, and E. F. Wood (2018). Present and future köppen-geiger climate classification maps at 1-km resolution. Scientific data 5(1), 1–12
work page 2018
-
[4]
Caron, M., H. Touvron, I. Misra, H. Jégou, J. Mairal, P. Bojanowski, and A. Joulin (2021). Emerging properties in self-supervised vision transformers. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 9650–9660
work page 2021
-
[5]
Chan-To-Hing, H. and B. Veeravalli (2024). Fus-mae: A cross-attention-based data fusion approach for masked autoencoders in remote sensing. In IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium, pp. 6953–6958. IEEE
work page 2024
-
[6]
Dosovitskiy, A., L. Beyer, A. Kolesnikov, D. Weissenborn, X. Zhai, T. Unterthiner, M. Dehghani, M. Min- derer, G. Heigold, S. Gelly, J. Uszkoreit, and N. Houlsby (2021). An image is worth 16x16 words: Trans- formers for image recognition at scale. In International Conference on Learning Representations
work page 2021
-
[7]
Drusch, M., U. Del Bello, S. Carlier, O. Colin, V . Fernandez, F. Gascon, B. Hoersch, C. Isola, P. Laberinti, P. Martimort, et al. (2012). Sentinel-2: Esa’s optical high-resolution mission for gmes operational services. Remote sensing of Environment 120, 25–36
work page 2012
Show all 38 references
-
[8]
Central Intelligence Agency (2025)
Esri, Global Mapping International, and U.S. Central Intelligence Agency (2025). World Regions. https: //www.arcgis.com/home/item.html?id=84dbc97915244e35808e87a881133d09. Layer package rep- resenting boundaries for 25 commonly recognized world regions. Updated April 29, 2025....
2025
-
[9]
Francis, A. and M. Czerkawski (2024). Major tom: Expandable datasets for earth observation. In IGARSS 2024-2024 IEEE International Geoscience and Remote Sensing Symposium, pp. 2935–2940. IEEE
2024
-
[10]
Millard, and J
Fuller, A., K. Millard, and J. Green (2023). Croma: Remote sensing representations with contrastive radar-optical masked autoencoders. Advances in Neural Information Processing Systems 36, 5506–5538. 11
2023
-
[11]
Gao, P., T. Ma, H. Li, Z. Lin, J. Dai, and Y . Qiao (2022). MCMAE: Masked convolution meets masked autoencoders. In A. H. Oh, A. Agarwal, D. Belgrave, and K. Cho (Eds.), Advances in Neural Information Processing Systems
2022
-
[12]
Guo, X., J. Lao, B. Dang, Y . Zhang, L. Yu, L. Ru, L. Zhong, Z. Huang, K. Wu, D. Hu, et al. (2024). Skysense: A multi-modal remote sensing foundation model towards universal interpretation for earth observation imagery. In Proceedings of the IEEE/CVF Conference on Computer Vis...
2024
-
[13]
Zhang, X
Han, B., S. Zhang, X. Shi, and M. Reichstein (2024). Bridging remote sensors with multisensor geospatial foundation models. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27852–27862
2024
-
[14]
He, K., X. Chen, S. Xie, Y . Li, P. Dollár, and R. Girshick (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 16000–16009
2022
-
[15]
Huot, F., R. L. Hu, N. Goyal, T. Sankar, M. Ihme, and Y .-F. Chen (2022). Next day wildfire spread: A machine learning dataset to predict wildfire spreading from remote-sensing data. IEEE Transactions on Geoscience and Remote Sensing 60, 1–13
2022
-
[16]
Kuckreja, K., M. S. Danish, M. Naseer, A. Das, S. Khan, and F. S. Khan (2024). Geochat: Grounded large vision-language model for remote sensing. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 27831–27840
2024
-
[17]
Lavreniuk, S
Kussul, N., M. Lavreniuk, S. Skakun, and A. Shelestov (2017). Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters 14(5), 778–782
2017
-
[18]
Hong, and J
Li, X., D. Hong, and J. Chanussot (2024). S2mae: A spatial-spectral pretraining foundation model for spectral remote sensing data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 24088–24097
2024
-
[19]
Darcet, T
Oquab, M., T. Darcet, T. Moutakanni, H. V o, M. Szafraniec, V . Khalidov, P. Fernandez, D. Haziza, F. Massa, A. El-Nouby, et al. (2023). Dinov2: Learning robust visual features without supervision. arXiv preprint arXiv:2304.07193
2023 arXiv
-
[20]
Prodhan, F. A., J. Zhang, F. Yao, L. Shi, T. P. Pangali Sharma, D. Zhang, D. Cao, M. Zheng, N. Ahmed, and H. P. Mohana (2021). Deep learning for monitoring agricultural drought in south asia using remote sensing data. Remote sensing 13(9), 1715
2021
-
[21]
Chowdhury, A
Rahnemoonfar, M., T. Chowdhury, A. Sarkar, D. Varshney, M. Yari, and R. R. Murphy (2021). Floodnet: A high resolution aerial imagery dataset for post flood scene understanding. IEEE Access 9, 89644–89654
2021
-
[22]
Reed, C. J., R. Gupta, S. Li, S. Brockman, C. Funk, B. Clipp, K. Keutzer, S. Candido, M. Uyttendaele, and T. Darrell (2023). Scale-mae: A scale-aware masked autoencoder for multiscale geospatial representation learning. In Proceedings of the IEEE/CVF International Conference o...
2023
-
[23]
Russwurm, C
Roscher, R., M. Russwurm, C. Gevaert, M. Kampffmeyer, J. A. Dos Santos, M. Vakalopoulou, R. Hänsch, S. Hansen, K. Nogueira, J. Prexl, et al. (2024). Better, not just more: Data-centric machine learning for earth observation. IEEE Geoscience and Remote Sensing Magazine
2024
-
[24]
Chowdhury, R
Sarkar, A., T. Chowdhury, R. R. Murphy, A. Gangopadhyay, and M. Rahnemoonfar (2023). Sam-vqa: Supervised attention-based visual question answering model for post-disaster damage assessment on remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing 61, 1–16
2023
-
[25]
Szwarcman, D., S. Roy, P. Fraccaro, Þ. E. Gíslason, B. Blumenstiel, R. Ghosal, P. H. de Oliveira, J. L. d. S. Almeida, R. Sedona, Y . Kang, et al. (2024). Prithvi-eo-2.0: A versatile multi-temporal foundation model for earth observation applications. arXiv preprint arXiv:2412.02732
2024
-
[26]
Cozma, K
Tang, M., A. Cozma, K. Georgiou, and H. Qi (2023). Cross-scale mae: A tale of multiscale exploitation in remote sensing. Advances in Neural Information Processing Systems 36, 20054–20066
2023
-
[27]
Snoeij, D
Torres, R., P. Snoeij, D. Geudtner, D. Bibby, M. Davidson, E. Attema, P. Potin, B. Rommen, N. Floury, M. Brown, et al. (2012). Gmes sentinel-1 mission. Remote sensing of environment 120, 9–24
2012
-
[29]
Waldmann, L., A. Shah, Y . Wang, N. Lehmann, A. J. Stewart, Z. Xiong, X. X. Zhu, S. Bauer, and J. Chuang (2025). Panopticon: Advancing any-sensor foundation models for earth observation. arXiv preprint arXiv:2503.10845
2025 arXiv
-
[30]
Wang, Y ., C. M. Albrecht, N. A. A. Braham, C. Liu, Z. Xiong, and X. X. Zhu (2024). Decoupling common and unique representations for multimodal self-supervised learning. In European Conference on Computer Vision, pp. 286–303. Springer
2024
-
[31]
Wang, Y ., Y . Sun, X. Cao, Y . Wang, W. Zhang, and X. Cheng (2023). A review of regional and global scale land use/land cover (lulc) mapping products generated from satellite remote sensing. ISPRS Journal of Photogrammetry and Remote Sensing 206, 311–334
2023
-
[32]
Xiong, C
Wang, Y ., Z. Xiong, C. Liu, A. J. Stewart, T. Dujardin, N. I. Bountos, A. Zavras, F. Gerken, I. Papoutsis, L. Leal-Taixé, et al. (2025). Towards a unified copernicus foundation model for earth vision. arXiv preprint arXiv:2503.11849
2025 arXiv
-
[33]
Prabha, T
Wang, Z., R. Prabha, T. Huang, J. Wu, and R. Rajagopal (2024). Skyscript: A large and semantically diverse vision-language dataset for remote sensing. In Proceedings of the AAAI Conference on Artificial Intelligence, V olume 38, pp. 5805–5813
2024
-
[34]
Xiao, T., Y . Liu, B. Zhou, Y . Jiang, and J. Sun (2018). Unified perceptual parsing for scene understanding. In European Conference on Computer Vision. Springer
2018
-
[35]
Zhang, Y
Xie, Z., Z. Zhang, Y . Cao, Y . Lin, Y . Wei, Q. Dai, and H. Hu (2023). On data scaling in masked image modeling. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 10365–10374
2023
-
[36]
Xiong, Z., Y . Wang, F. Zhang, A. J. Stewart, J. Hanna, D. Borth, I. Papoutsis, B. Le Saux, G. Camps-Valls, and X. X. Zhu (2024). Neural plasticity-inspired foundation model for observing the earth crossing modalities. arXiv e-prints, arXiv–2403
2024
-
[37]
Yu, D. and C. Fang (2023). Urban remote sensing with spatial big data: A review and renewed perspective of urban studies in recent decades. Remote Sensing 15(5), 1307
2023
-
[38]
𝑥!" 𝐸𝑛𝑐!ProjectorStudent Network 1𝑓!
Zanaga, D., R. Van De Kerchove, D. Daems, W. De Keersmaecker, C. Brockmann, G. Kirches, J. Wevers, O. Cartus, M. Santoro, S. Fritz, et al. (2022). Esa worldcover 10 m 2021 v200. 13 Supplementary Material This supplementary material presents additional experiments, analyses, an...
2022
Reviewed August 7, 2026 · model on record in the stance chip above.
Discussion (0). Sign in to comment.