Landsat-Bench: Datasets and Benchmarks for Landsat Foundation Models

Isaac Corley; Lakshay Sharma; Ruth Crasto

REVIEW 3 major objections 6 minor 14 references

Landsat-Bench: Datasets and Benchmarks for Landsat Foundation Models

T0 review · 3 major / 6 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper introduces Landsat-Bench, three Landsat-8 evaluation datasets, and argues that SSL4EO-L-pretrained models transfer better to downstream tasks than ImageNet-pretrained baselines.

desk verdict Good benchmark resource, but the headline SSL4EO-L vs. ImageNet gains are cherry-picked and the unvalidated label transfer from Sentinel to Landsat needs to be addressed before the rankings can be trusted. read the letter →

arxiv 2506.08780 v1 pith:QZZYMZX7 submitted 2025-06-10 cs.CV cs.LG

Isaac Corley , Lakshay Sharma , Ruth Crasto This is my paper

classification cs.CVcs.LG

keywords Landsatfoundationmodelsremotesensingbenchmarksself-supervisedlearningEuroSAT-LBigEarthNet-LLC100-Ltransfer

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper seeks to fill the absence of standardized benchmarks for Landsat satellite imagery, which has slowed the development of Landsat-based geospatial foundation models. It constructs three benchmark datasets, EuroSAT-L, BigEarthNet-L, and LC100-L, by downloading Landsat 8 imagery for the locations and times of existing Sentinel-based datasets and reusing their labels. Across KNN and linear probing evaluations with ResNet and ViT architectures, it reports that SSL4EO-L pretrained models extract stronger features than ImageNet pretrained weights, with gains of +4% overall accuracy on EuroSAT-L and +5.1% mAP on BigEarthNet-L. The paper also notes that the LC100-L variant is much harder than its Sentinel-3 original, with the best Landsat model reaching only 63.8 mAP.

What carries the argument

The load-bearing object is Landsat-Bench itself: three benchmarks built by aligning Landsat 8 Collection 2 Level-2 imagery with existing labels from Sentinel-derived datasets, constrained by a minimum 80% valid pixel ratio and cloud cover of at most 10% (relaxed progressively until all samples are downloaded). The evaluation machinery is frozen-feature transfer: KNN with k=5 and one-vs-rest logistic regression linear probing over extracted embeddings, with ImageNet pretrained weights as the baseline and SSL4EO-L MoCo v2 and SimCLR weights as the Landsat-pretrained options.

What would settle it

Take a subset of Landsat-Bench patches and have independent higher-resolution reference data or expert annotators label them, then measure agreement with the transferred labels; if agreement is low, the reported performance gains could be measuring label noise rather than transfer. A simpler check would be to compare EuroSAT-L results using the guessed 2018 acquisition window against a small manually dated subset.

Watch

Extended reading notes

Core claim

The central claim is that pretraining on Landsat imagery with self-supervised contrastive methods yields representations that transfer to Landsat downstream tasks better than ImageNet pretraining. This is demonstrated on Landsat-Bench for common architectures including ResNet-18, ResNet-50, and ViT-S/16, using KNN with k=5 and linear probing. In the KNN setting, SSL4EO-L MoCo v2 and SimCLR weights outperform ImageNet weights on EuroSAT-L and BigEarthNet-L by margins of up to +4% overall accuracy and +5.1% mAP, while LC100-L shows little difference under KNN and only linear probing separates the models.

Load-bearing premise

The benchmarks reuse patch-level labels from Sentinel-based datasets for Landsat 8 imagery acquired at different times, without verifying that the land cover label still matches the downloaded Landsat pixels.

Editorial extensions

If this is right

If SSL4EO-L pretraining generalizes as claimed, Landsat-based geospatial foundation models become a practical default for 30-meter Earth observation tasks instead of relying on ImageNet features.
The benchmarks provide a standardized way to compare future Landsat foundation models through KNN and linear probing scores on three diverse tasks.
Landsat's 30-meter resolution allows roughly nine times cheaper global inference than Sentinel-2 at equivalent patch size, with the paper estimating 1.83 hours versus 16.5 hours for global ResNet-50 inference.
Multi-sensor ensembling with Sentinel could raise effective revisit rates and support time-sensitive monitoring such as wildfire and deforestation tracking.
The sharp performance drop on LC100-L indicates that Landsat's seven OLI bands may not carry all the spectral information of Sentinel-3's 21 bands for some land-cover classes.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Because the label transfer assumes patch-level labels remain valid across sensors and acquisition dates, a natural extension is to validate a sample of labels against independent higher-resolution reference data; if the labels are noisy, reported gains could partly reflect label noise rather than representation quality.
EuroSAT-L relies on a guessed acquisition window of March to August 2018, so its temporal alignment is weaker than that of BigEarthNet-L and LC100-L; future versions could reconstruct exact acquisition dates or provide per-sample uncertainty metadata.
The reversal in ranking between MoCo v2 and SimCLR under KNN versus linear probing suggests the evaluation protocol materially changes conclusions, so a standardized protocol reporting both metrics is itself a contribution.
If Landsat foundation models mature, the 50-year Landsat archive combined with this benchmark format could enable temporal generalization studies that Sentinel-era benchmarks cannot support.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

Quick take: this is a dataset/benchmark paper, not a method paper. The three Landsat-adapted datasets are new and fill an actual gap; the evaluation protocol is clean and reproducible; code and data are released. If you do geospatial foundation model work, this is a useful resource to have on hand. But the paper overstates its main result, and the stress-test note is on target about label transfer.

What is genuinely good: the idea of taking well-known Sentinel-2/Sentinel-3 labeled datasets and downloading corresponding Landsat 8 scenes is straightforward but useful. The authors use existing metadata and Planetary Computer, keep the preprocessing simple, and provide baselines with standard architectures and the public SSL4EO-L weights. The KNN and linear probe evaluation is about as ungameable as it gets, and the split choices are standard. The new comparisons themselves are a contribution, even where the conclusion is not uniformly "Landsat pretraining wins."

Soft spots, in order of importance. First, label validity. Section 2 explicitly says they relax cloud cover to 20% and expand the timestamps until all samples are downloaded. For EuroSAT they guess a six-month window in 2018. None of this is validated against the label. Seasonal or inter-annual land cover change can flip a class between the Sentinel label date and the Landsat acquisition. That means some benchmark labels are wrong, and nobody knows how many. This is not fatal for a resource, but it is fatal for the paper's claim that SSL4EO-L features are better because they beat ImageNet on these labels. If the SSL4EO-L features are simply more robust to label noise, the conclusion changes. The authors need to either check a sample, report how many patches used relaxed constraints, or soften the claim to "better under label noise." Second, the headline numbers. The +4% OA and +5.1% mAP come from the best ViT-S KNN rows. Across Table 2, ImageNet is competitive or better in several configurations, especially LC100-L. The abstract should say "in selected configurations," not make a blanket statement. Third, minor: the self-comparison with SSL4EO-L weights has an author overlap, but the benchmark is independent and the weights are public, so I do not weight that heavily.

Bottom line: the benchmark deserves to exist, and the paper deserves refereeing. I would send it to review and ask for a major revision that adds a label-consistency check, quantifies the relaxed-constraint subset, and recalibrates the abstract to match the tables.

Referee Report

3 major / 6 minor

Summary. The paper introduces Landsat-Bench, a benchmark suite of three Landsat-8 variants of existing remote sensing datasets (EuroSAT-L, BigEarthNet-L, LC100-L), constructed by downloading Landsat Collection 2 Level-2 imagery via Microsoft Planetary Computer using timestamps and geospatial bounds from the original Sentinel-based datasets. It reports KNN (k=5) and linear probing baselines for ImageNet-pretrained ResNet-18/50 and ViT-S/16, and for SSL4EO-L MoCo v2 and SimCLR pretrained weights, and claims that SSL4EO-L models extract better representations than ImageNet, with headline gains of +4% OA on EuroSAT-L and +5.1% mAP on BigEarthNet-L. The main contributions are the datasets and the standardized evaluation pipeline, with code and data released.

Significance. If the label-transfer assumption and the baseline results hold, Landsat-Bench would fill a real gap: it is one of the first dedicated evaluation suites for Landsat-based Geospatial Foundation Models, and the SSL4EO-L comparisons provide useful reference numbers for future work. The evaluation protocol is simple, deterministic, and reproducible, and the authors ship code and datasets, which are concrete strengths. However, the validity of the benchmark hinges on the untested transfer of labels from Sentinel to Landsat imagery, and the abstract's central empirical claim is stronger than what Table 2 actually shows. These issues are load-bearing for the paper's primary artifact and its main conclusion, so they need to be resolved before the benchmark can be relied upon.

major comments (3)

[Section 2, Data Acquisition] The transfer of labels from Sentinel-2/Sentinel-3 to Landsat-8 imagery is the key data-construction assumption, but the paper provides no verification that the transferred labels are correct. For EuroSAT-L, the acquisition window 2018/03/01–2018/08/31 is explicitly described as an initial guess and is never validated against the actual Sentinel-2 acquisition dates. For BigEarthNet-L and LC100-L, the ±30-day search window is iteratively expanded and cloud cover is relaxed from ≤10% to ≤20%, so some patches are paired with Landsat scenes from a different season or potentially a different year, and those scenes may contain up to 20% invalid pixels. The paper does not report how many samples required relaxed constraints or what the temporal offset distribution looks like. A direct label-consistency check is needed—for example, a manual audit, a comparison against an independent land-cover product, or an automatic stability filter that excludes patches whose label is likely to have changed between the Sentinel label date and the Landsat acquisition date. Without this, the headline gains in Table 2 and the claims in Section 4 could reflect robustness to label noise rather than genuinely better representations.
[Abstract and Section 4, Table 2] The abstract's claim that 'SSL4EO-L pretrained GFMs extract better representations for downstream tasks in comparison to ImageNet' is too strong given the paper's own results. In Table 2, several configurations show ImageNet matching or beating the SSL4EO-L models: for EuroSAT-L KNN, the ResNet-50 ImageNet baseline (76.4 OA) outperforms both MoCo v2 (75.1) and SimCLR (72.6); for LC100-L KNN, all methods are within roughly 1–2 points and ImageNet is best or tied in most cells. The +4% and +5.1% figures are obtained by selecting the best ViT-S KNN result for each task. The claim should be qualified as 'in several configurations' or 'when the best SSL4EO-L configuration is selected per task,' and the paper should report confidence intervals or otherwise support that the differences are not due to noise.
[Section 4, Benchmark Difficulty and Table 3] The comparison in Table 3 mixes architectures, model sizes, and pretraining datasets, yet is used to draw a conclusion about the difficulty of the Landsat variants. The LC100-L drop from 93.8 to 63.8 mAP is attributed to the reduced spectral information of Landsat, but this attribution is speculative because the comparison is not controlled for model capacity (Copernicus-FM ViT-B/16 vs. ResNet-18) or for the fact that the original LC100 is evaluated on Sentinel-3 imagery while the Landsat variant is evaluated on a different sensor. Please either present a more controlled comparison (e.g., same architecture evaluated on both original and Landsat variants) or soften the conclusion to acknowledge confounds.

minor comments (6)

[Table 2] The table is difficult to read because the numeric entries run together and the superscript asterisks are visually lost, e.g., '81.683.4' and '64.567.0'. Please reformat the table with clearly separated columns and distinct marker styles so each value can be unambiguously assigned to a model.
[Section 4 vs. Table 3] Table 3 reports LC100-S3 Copernicus-FM mAP as 93.3, but the text in Section 4 says performance drops from 93.8 to 63.8; please correct this numerical inconsistency.
[Section 2, Data Acquisition] The paper does not report the fraction of samples that required the relaxed cloud-cover or timestamp constraints, nor the distribution of temporal offsets between the original timestamp and the downloaded Landsat scene. Even if a full label-consistency validation is deferred, this information is essential for users to assess label reliability.
[Section 4, Computational Efficiency] The sentence 'approximately1.83 hoursfor Landsat-8 and16.5 hours' has spacing issues and should read 'approximately 1.83 hours for Landsat-8 and 16.5 hours'; also, superscripts in units such as m2 should be rendered consistently as m².
[Figure 1] The sample images in Figure 1 are too small to be informative; please provide larger thumbnails or a higher-resolution figure, and state which band combination (e.g., RGB or false color) is shown for each dataset.
[Section 3, Baseline Models] The description of the ImageNet weight adaptation states that 'the first convolutional layer adaptive initialization scheme ... repeats the RGB weights for additional channels'; please clarify whether this is applied identically to all architectures and whether any channel-wise scaling is used, since the Landsat bands are not equally weighted in the original RGB channels.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: Landsat-Bench evaluation is self-contained and only uses external pretrained weights as benchmarks.

full rationale

The paper's main contribution is a new benchmark suite obtained by matching Landsat 8 imagery to existing EuroSAT, BigEarthNet, and LC100-S3 labels. The headline comparison (SSL4EO-L pretrained weights vs. ImageNet weights) uses SSL4EO-L models from Stewart et al. (2023) as external artifacts made available in torchgeo; those weights were not trained on Landsat-Bench and no Landsat-Bench output is fed back into them. The evaluation protocol (KNN with k=5, logistic-regression linear probing, fixed preprocessing from Corley et al. 2024) is a standard, parameter-free measurement rather than a fitted parameter renamed as a prediction. The +4% OA and +5.1% mAP figures are therefore empirical results, not identities derived from the input labels or from the authors' own prior work. One author of the present paper co-authored SSL4EO-L and torchgeo, but that self-citation is not load-bearing in a circular sense: the pretrained weights are code-reproduced, externally available, and their quality is independently falsifiable on the new benchmark. The unvalidated transfer of Sentinel-era labels to Landsat patches (Section 2, Data Acquisition) is a potential correctness and label-noise concern, but it does not make the paper's derivation equivalent to its inputs by construction, so it does not constitute circularity under the given criteria. No step in the claimed derivation chain reduces to self-definition, fitted-input prediction, or a uniqueness claim imported from the authors' own work.

Assumptions & free parameters 0 free parameters · 3 assumptions · 0 invented entities

The central claim rests on the validity of transferring labels from Sentinel-based datasets to Landsat imagery and on the fairness of the evaluation protocol. No free parameters or invented entities are introduced; the baselines are standard and available.

assumptions (3)

domain assumption Patch-level labels from EuroSAT, BigEarthNet, and LC100 remain valid for Landsat 8 imagery acquired at different times.
Section 2 Data Acquisition; EuroSAT timestamps are estimated as 2018/03 to 08, and no validation of label consistency across sensors is provided.
domain assumption A minimum valid pixel ratio of 80% and cloud cover up to 20% ensure sufficient image quality for benchmark use.
Section 2 Data Acquisition; thresholds are chosen heuristically and not validated.
domain assumption ImageNet pretrained weights with repeated RGB channels for 7 multispectral bands is a fair transfer baseline.
Section 3 Baseline Models; this is a standard approach but not necessarily optimal for multispectral data.

how reviews work

0 comments

Cite this review

Pith. "Pith review of Landsat-Bench: Datasets and Benchmarks for Landsat Foundation Models." pith.science (2026). https://pith.science/paper/QZZYMZX7

@misc{pith2026250608780,
  author       = {Pith},
  title        = {Pith review of: Landsat-Bench: Datasets and Benchmarks for Landsat Foundation Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QZZYMZX7}},
  note         = {Machine review of arXiv:2506.08780}
}

read the original abstract

The Landsat program offers over 50 years of globally consistent Earth imagery. However, the lack of benchmarks for this data constrains progress towards Landsat-based Geospatial Foundation Models (GFM). In this paper, we introduce Landsat-Bench, a suite of three benchmarks with Landsat imagery that adapt from existing remote sensing datasets -- EuroSAT-L, BigEarthNet-L, and LC100-L. We establish baseline and standardized evaluation methods across both common architectures and Landsat foundation models pretrained on the SSL4EO-L dataset. Notably, we provide evidence that SSL4EO-L pretrained GFMs extract better representations for downstream tasks in comparison to ImageNet, including performance gains of +4% OA and +5.1% mAP on EuroSAT-L and BigEarthNet-L.

Figures

Figures reproduced from arXiv: 2506.08780 by the authors.

**Figure 1.** We introduce Landsat-Bench: a new benchmark for Geospatial Foundation Models (GFM) pretrained on Landsat imagery. Our benchmark contains three Landsat variants (denoted with the suffix “—L”.) of existing remote sensing datasets. Samples of each original dataset and our proposed Landsat variants are provided above (from top-to-bottom by row: LC100-L, LC100-S3, EuroSAT-L, EuroSAT, BigEarthNet-L & BigEarthNet). Abstrac… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

14 extracted references · 6 canonical work pages

[4]

Engebretson, C

URL https://doi.org/ 10.5066/P9KZCM54. Engebretson, C. Landsat 8–9 Operational Land Imager (OLI) - Thermal Infrared Sensor (TIRS) Collection 2 Level 1 (L1) Data Format Control Book (DFCB). Tech- nical report, Department of the Interior, U.S. Geological Survey, September

doi:10.5066/p9kzcm54
[8]

microsoft/planetarycomputer: October 2022, October

Microsoft. microsoft/planetarycomputer: October 2022, October

work page 2022
[10]

J., Lehmann, N., Corley, I

Stewart, A. J., Lehmann, N., Corley, I. A., Wang, Y ., Chang, Y .-C., Braham, N. A. A., Sehgal, S., Robinson, C., and Banerjee, A. Ssl4eo-l: Datasets and foundation models for landsat imagery.arXiv preprint arXiv:2306.09424,

arXiv
[11]

Bigearthnet: A large-scale benchmark archive for remote sensing image understanding

Sumbul, G., Charfuelan, M., Demir, B., and Markl, V . Bigearthnet: A large-scale benchmark archive for remote sensing image understanding. InIGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Sym- posium, pp. 5901–5904. IEEE,

work page 2019
[31]

2017.06.031

URL https://doi.org/10.1016/j.rse. 2017.06.031. Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226,

doi:10.1016/j.rse 2017
[2011]

Coper- nicus global land service: land cover 100 m: collec- tion 3: epoch 2019: Globe.Zenodo,

Buchhorn, M., Smets, B., Bertels, L., De Roo, B., Lesiv, M., Tsendbazar, N.-E., Herold, M., and Fritz, S. Coper- nicus global land service: land cover 100 m: collec- tion 3: epoch 2019: Globe.Zenodo,

work page 2019
[2017]

doi: 10.1016/j.rse.2017.06

work page doi:10.1016/j.rse.2017.06 2017
[2019]

Wang, Y ., Braham, N. A. A., Xiong, Z., Liu, C., Albrecht, C. M., and Zhu, X. X. Ssl4eo-s12: A large-scale multi- modal, multitemporal dataset for self-supervised learning in earth observation [software and data sets].IEEE Geo- science and Remote Sensing Magazine, 11(3):98–106, 2023a. Wang, Y ., Hern´andez, H. H., Albrecht, C. M., and Zhu, X. X. Feature g...

arXiv

Show all 14 references

[2020]

URL https://doi.org/ 10.5281/zenodo.3939050

doi: 10.5281/zenodo.3939050. URL https://doi.org/ 10.5281/zenodo.3939050. Chen, T., Kornblith, S., Norouzi, M., and Hinton, G. A simple framework for contrastive learning of visual rep- resentations. InInternational conference on machine learning, pp. 1597–1607. PMLR, 2020a. C...

arXiv
[2021]

and Survey, U

Dewitz, J. and Survey, U. G. National Land Cover Database (NLCD) 2019 products (ver. 2.0). U.S. Geological Survey data release, June

2019
[2022]

Neumann, M., Pinto, A

URL https://doi.org/10.5281/ zenodo.7261897. Neumann, M., Pinto, A. S., Zhai, X., and Houlsby, N. In- domain representation learning for remote sensing.arXiv preprint arXiv:1911.06721,

1911 arXiv
[2023]

On the opportu- nities and challenges of foundation models for geospatial artificial intelligence.arXiv preprint arXiv:2304.06798,

Mai, G., Huang, W., Sun, J., Song, S., Mishra, D., Liu, N., Gao, S., Liu, T., Cong, G., Hu, Y ., et al. On the opportu- nities and challenges of foundation models for geospatial artificial intelligence.arXiv preprint arXiv:2304.06798,

arXiv
[2024]

J., Dujardin, T., Bountos, N

Wang, Y ., Xiong, Z., Liu, C., Stewart, A. J., Dujardin, T., Bountos, N. I., Zavras, A., Gerken, F., Papoutsis, I., Leal- Taix´e, L., et al. Towards a unified copernicus foundation model for earth vision.arXiv preprint arXiv:2503.11849,

arXiv
[2025]

Watch, G. F. Global forest watch.World Resources Institute, Washington, DC Available from http://www. globalforest- watch. org (accessed March 2002),

2002

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.

[1] [4]

Engebretson, C

URL https://doi.org/ 10.5066/P9KZCM54. Engebretson, C. Landsat 8–9 Operational Land Imager (OLI) - Thermal Infrared Sensor (TIRS) Collection 2 Level 1 (L1) Data Format Control Book (DFCB). Tech- nical report, Department of the Interior, U.S. Geological Survey, September

doi:10.5066/p9kzcm54

[2] [8]

microsoft/planetarycomputer: October 2022, October

Microsoft. microsoft/planetarycomputer: October 2022, October

work page 2022

[3] [10]

J., Lehmann, N., Corley, I

Stewart, A. J., Lehmann, N., Corley, I. A., Wang, Y ., Chang, Y .-C., Braham, N. A. A., Sehgal, S., Robinson, C., and Banerjee, A. Ssl4eo-l: Datasets and foundation models for landsat imagery.arXiv preprint arXiv:2306.09424,

arXiv

[4] [11]

Bigearthnet: A large-scale benchmark archive for remote sensing image understanding

Sumbul, G., Charfuelan, M., Demir, B., and Markl, V . Bigearthnet: A large-scale benchmark archive for remote sensing image understanding. InIGARSS 2019-2019 IEEE International Geoscience and Remote Sensing Sym- posium, pp. 5901–5904. IEEE,

work page 2019

[5] [31]

2017.06.031

URL https://doi.org/10.1016/j.rse. 2017.06.031. Helber, P., Bischke, B., Dengel, A., and Borth, D. Eurosat: A novel dataset and deep learning benchmark for land use and land cover classification.IEEE Journal of Se- lected Topics in Applied Earth Observations and Remote Sensing, 12(7):2217–2226,

doi:10.1016/j.rse 2017

[6] [2011]

Coper- nicus global land service: land cover 100 m: collec- tion 3: epoch 2019: Globe.Zenodo,

Buchhorn, M., Smets, B., Bertels, L., De Roo, B., Lesiv, M., Tsendbazar, N.-E., Herold, M., and Fritz, S. Coper- nicus global land service: land cover 100 m: collec- tion 3: epoch 2019: Globe.Zenodo,

work page 2019

[7] [2017]

doi: 10.1016/j.rse.2017.06

work page doi:10.1016/j.rse.2017.06 2017

[8] [2019]

Wang, Y ., Braham, N. A. A., Xiong, Z., Liu, C., Albrecht, C. M., and Zhu, X. X. Ssl4eo-s12: A large-scale multi- modal, multitemporal dataset for self-supervised learning in earth observation [software and data sets].IEEE Geo- science and Remote Sensing Magazine, 11(3):98–106, 2023a. Wang, Y ., Hern´andez, H. H., Albrecht, C. M., and Zhu, X. X. Feature g...

arXiv