Pith. sign in

REVIEW 3 major objections 5 minor 27 references

Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data

T0 review · 3 major / 5 minor · reviewed 2026-08-07 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read The paper claims that standard deep-learning data loaders, configured with tile-aligned reads and worker thread pools, can stream cloud-stored GeoTIFF imagery fast enough for remote training to match local-disk accuracy in the same time…

desk verdict Useful, reproducible throughput benchmark for cloud GeoTIFF streaming, but the training-parity claim is confounded by differences in patch size and sampling. read the letter →

arxiv 2506.06235 v1 pith:LGWZOFJY submitted 2025-06-06 cs.CV

Akram Zaytar , Caleb Robinson , Girmaw Abebe Tadesse , Tammy Glazer , Gilles Hacheme , Anthony Ortiz , Rahul M Dodhia , Juan M Lavista Ferres This is my paper
classification cs.CV
keywords cloudobjectstorageGeoTIFFtile-alignedreadsBayesianoptimizationGPUutilizationdataloadingthroughputEarthobservationsemanticsegmentation
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper aims to show that the standard deep-learning data-loading stack, configured sensibly and without any custom file format, can stream GeoTIFF imagery from cloud object storage fast enough to keep a GPU fully occupied. It reports that optimized loader settings raise remote loading throughput roughly 20-fold and local SSD loading 4-fold over the default configuration, and that on three public Earth-observation segmentation benchmarks, models trained with cloud-stored data reach the same validation accuracy as locally trained models within a fixed five-minute budget. The practical stake is that petabyte-scale satellite archives could be used for direct cloud training instead of being downloaded or mirrored locally.

What carries the argument

The load-bearing mechanism is tile-aligned reading: a 'blocked' sampler restricts random patch windows to stay inside one internal k-by-k GeoTIFF tile, so each training patch is served by exactly one block read instead of up to four intersecting blocks. Around this sits a worker thread pool that hides the roughly 164 ms cloud round-trip latency behind concurrent range requests, with the whole configuration space of compression, patch size, workers, threads, prefetch factor, and alignment searched by Bayesian optimization.

What would settle it

Train the same segmentation model on an Earth-observation dataset where objects routinely cross tile boundaries, comparing the blocked sampler against unrestricted random cropping with throughput held equal; if validation accuracy drops measurably under blocked sampling, the claim that optimized remote loading matches local training would fail for that setting. A second check is to reproduce the 20x remote throughput figure under a different cloud region or provider to see whether the worker/thread optimum is provider-specific.

Watch

Extended reading notes

Core claim

On the paper's own terms, the central claim is that the remote data-loading bottleneck for GeoTIFF training is a configuration problem, not a format problem. By combining tile-aligned patch sampling that keeps each read inside one internal block, an intra-worker thread pool that issues concurrent range requests, and Bayesian optimization over compression, patch size, worker count, thread count, and prefetch factor, the authors report 20.5x higher remote throughput (849 vs 41 MB/s) and 4.1x higher local throughput (1285 vs 313 MB/s) relative to default settings. On the Vaihingen, Potsdam, and DFC-22 segmentation benchmarks, the cloud-optimized configuration sustained 80-90% GPU utilization and matched local-training validation IoU within a fixed time budget, while the default remote configuration left GPUs idle 0-30% of the time and trailed by 6-15 IoU points.

Load-bearing premise

The paper's equivalence claim assumes that restricting random crops to stay inside GeoTIFF tiles does not materially change the training distribution, since tile-aligned reads prevent ordinary random cropping; the authors state this in the limitations section, and if it degrades generalization elsewhere the cloud/local match would not transfer.

Editorial extensions

If this is right

  • Cloud-trained Earth-observation models can match local-disk accuracy within a fixed time budget, removing the need to copy petabyte archives locally before each experiment.
  • Optimal loader settings are storage-specific: uncompressed imagery wins on local SSD, while LERC-ZSTD compression plus 64 workers and a prefetch factor of 8 wins for remote object storage.
  • Tile-aligned reads pay off more as patch size grows, with remote throughput gains of 45% at 128-pixel patches and 79% at 1024-pixel patches.
  • Sustained 85-95% GPU utilization replaces the 0-30% utilization of default remote loading, cutting idle compute and experiment cost.
  • The recommendations generalize to any chunked raster workflow: match patch size to tile structure, use aligned sampling, scale workers, and pre-fetch aggressively.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The blocked sampler changes the training distribution by outlawing crops that cross tile boundaries; if small objects or boundary-spanning features matter, accuracy could suffer on datasets beyond the three tested, so the equivalence claim should be re-checked with epoch budgets rather than time budgets.
  • The same Bayesian tuning recipe likely transfers to other chunked raster formats such as Zarr arrays, though the optimal compression and worker counts would shift.
  • The worker-versus-thread tradeoff (more workers, fewer threads) may reflect rate limits of the specific cloud provider and region; co-locating storage and compute could move the optimum.
  • A practical middle path would be to use tile-aligned reads only for prefetching and then reassemble cross-tile patches in GPU memory, preserving random-crop augmentation while keeping aligned I/O.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 5 minor

Summary. The paper addresses data-loading bottlenecks when training segmentation models on GeoTIFF data streamed from cloud object storage. It introduces two loader modifications—tile-aligned sampling (blocked=True) and per-worker thread pools—and uses Bayesian optimization to search over compression, patch size, worker counts, thread counts, and prefetch factors. Throughput benchmarks on Azure Blob Storage and local SSD report 20.5x and 4.1x improvements over a default PyTorch configuration. The authors then train a ResNet-18 UNet on three segmentation benchmarks (Vaihingen, Potsdam, DFC-22) under a fixed 5-minute budget, reporting that the optimized remote configuration matches local-optimal accuracy while maintaining 80–90% GPU utilization. Code is released.

Significance. The throughput measurements are directly measured, the search procedure is clearly described, and the paper addresses a practically important problem: avoiding local copies of petabyte-scale EO data. The public code and the systematic comparison of compression and loader settings are useful contributions. If the training-parity result were cleanly established, the work would provide strong evidence that standard PyTorch loaders (without custom formats) can be configured for cloud-native COG training. However, as discussed below, the training comparison currently contains a confound that prevents the parity claim from being isolated to storage location.

major comments (3)
  1. [Section 5; Table 2; Figures 2–3] The central parity claim compares remote-optimal (1024-pixel patches, LERC-ZSTD, blocked sampler) with local-optimal (512-pixel patches, uncompressed, blocked sampler). These configurations differ in patch size, which changes the input distribution and the number of gradient steps within the fixed 5-minute budget. The sentence in Section 5 that the IoU differences are 'attributable solely to data loading configuration' is therefore not justified for the remote-vs-local comparison. To support the abstract's claim that cloud training incurs no accuracy penalty, the authors should hold the training distribution (at least patch size and sampling) constant across storage locations, or provide an ablation that varies patch size under both local and remote storage.
  2. [Section 5] The paper does not report how many optimization steps or epochs each configuration completes in the 5-minute budget. Without this, 'identical time budgets' is only a wall-clock statement; the remote and local configurations may be at very different points in training. Please include completed batches/epochs for each dataset version, or report loss/IoU versus number of steps.
  3. [Section 6] The limitation that tile-aligned reads 'prevent random cropping... potentially reducing model performance' is acknowledged, but the paper does not quantify this effect. Since both optimal configurations use blocked sampling, a direct comparison of blocked versus unblocked sampling with the same patch size is needed to understand whether the optimized configurations' results are tied to this distributional shift. This is relevant to the generalizability of the parity claim beyond the three tested benchmarks.
minor comments (5)
  1. [Section 3.2; Table 1; Table 7] The range for num_threads is given as {1,2,...,32} in Section 3.2 and Table 7 includes 32 threads, but Table 1 lists candidate values {1,2,4,8,16}. Please align these.
  2. [References] The reference list contains two identical entries for Mohan et al. (2020a and 2020b); the in-text citation to 2020b should be distinguished or consolidated.
  3. [Section 5] The remote-optimal patch size (1024) is not stated in the dataset-version description; readers must infer it from Table 2. Please state the patch size explicitly for each dataset version.
  4. [Figures 2 and 3] Final IoU values are only given in the figure captions; a table with the final IoU values (and standard deviations across seeds, if available) would make the quantitative claims easier to verify.
  5. [Section 2.1] Please clarify whether any OS-level file caching is cleared before local throughput measurements; otherwise the local numbers may reflect cached reads rather than cold local SSD performance.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: throughput speedups are measured ratios, and the accuracy-parity claim is a transfer test on external benchmarks with no self-citations and no fitted value renamed as a prediction.

full rationale

This paper is an empirical benchmarking study whose central claims are measured quantities rather than derived results, and I find no step that reduces by construction or by self-citation to its own inputs. The throughput speedups (20.5x remote, 4.1x local, Table 2) are ratios of independently wall-clock-measured MB/s values, where the paper defines throughput operationally: 'Throughput is measured in megabytes per second (MB/s) and represents how much image data the pipeline can process per unit time.' The Bayesian optimization of Section 3.3 fits hyperparameters to this throughput objective, but no fitted value is later renamed as a prediction: the optimized configurations are then transferred to three different public benchmarks (Vaihingen, Potsdam, DFC-22) that played no role in the hyperparameter search, and accuracy is measured as validation IoU under a fixed 5-minute budget. That is a genuine transfer test, not a circular fit. The reference list contains no self-citations by any of the seven authors, so the self-citation, uniqueness-imported, and ansatz-via-citation patterns do not apply. The 'blocked' sampler (Section 3.1) and worker thread pools (Section 3.2) are introduced as new mechanisms and evaluated directly, not borrowed as unexamined prior-authority premises. One in-scope concern is the Section 5 claim that the observed IoU gaps are 'attributable solely to data loading configuration,' since the local-optimal and remote-optimal versions also differ in compression (uncompressed vs LERC-ZSTD) and patch size (512 vs 1024 pixels), and the remote-optimal configuration additionally uses the blocked sampler that the authors concede 'prevent[s] random cropping during training, potentially reducing model performance' (Section 6). This is a threat to the causal attribution of the accuracy-parity result, and it is disclosed in the paper's own limitations section, but it is not circularity: the accuracy numbers are measured outcomes, not constructed from the configurations, and the claim does not define its conclusion into existence. I therefore score 0 with no circular steps.

Assumptions & free parameters 8 free parameters · 4 assumptions · 0 invented entities

The central claims depend on the selected loader hyperparameters, found by Bayesian optimization on the benchmark data, and on several measurement assumptions about the cloud environment. The 'blocked' jitter sampler is a code-level option, not a new entity. No new particles, forces, or conserved quantities are introduced.

free parameters (8)
  • remote_num_workers = 64
    Selected by Optuna for Azure Blob remote loading; higher worker counts hide network latency.
  • remote_num_threads = 1
    Optimized; more threads caused rate-limiting and synchronization stalls in the tested cloud environment.
  • remote_patch_size = 1024
    Found optimal; aligns to 2x2 tiles of 512 pixels in the benchmark COGs, but the authors note it is the search upper bound.
  • remote_blocked = True
    Enables tile-aligned reads, reducing unnecessary data transfers.
  • remote_prefetch_factor = 8
    Optimized to mask network latency.
  • remote_compression = LERC-ZSTD
    Best trade-off between transfer size and decompression cost for remote access.
  • local_compression = None
    Uncompressed data is fastest on local SSD.
  • local_patch_size = 512
    Optimized local throughput in the Bayesian search.
assumptions (4)
  • domain assumption PyTorch DataLoader with 4 workers and 256-pixel patches represents the standard default configuration for remote GeoTIFF training.
    Used as the baseline; prefetch factor and other PyTorch defaults are not fully specified in the paper.
  • domain assumption The measured throughput in MB/s is a valid proxy for end-to-end training speed with a fixed model and compute.
    The paper compares validation IoU after fixed wall-clock time, so loading throughput is assumed to map directly to steps completed.
  • domain assumption Byte-range requests to Azure Blob experience approximately 164 ms round-trip latency and are not rate-limited in a way that invalidates the worker and thread counts.
    The remote experiments use one Azure region pair; results may not generalize to other providers or co-located storage.
  • ad hoc to paper Restricting random crops to stay inside tiles (blocked=True) does not materially change the training distribution compared to unrestricted random cropping.
    Explicitly noted as a limitation in Section 6; if false, the accuracy comparison is confounded and the recommendation may not transfer.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data." pith.science (2026). https://pith.science/paper/LGWZOFJY

@misc{pith2026250606235,
  author       = {Pith},
  title        = {Pith review of: Optimizing Cloud-to-GPU Throughput for Deep Learning With Earth Observation Data},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/LGWZOFJY}},
  note         = {Machine review of arXiv:2506.06235}
}
read the original abstract

Training deep learning models on petabyte-scale Earth observation (EO) data requires separating compute resources from data storage. However, standard PyTorch data loaders cannot keep modern GPUs utilized when streaming GeoTIFF files directly from cloud storage. In this work, we benchmark GeoTIFF loading throughput from both cloud object storage and local SSD, systematically testing different loader configurations and data parameters. We focus on tile-aligned reads and worker thread pools, using Bayesian optimization to find optimal settings for each storage type. Our optimized configurations increase remote data loading throughput by 20x and local throughput by 4x compared to default settings. On three public EO benchmarks, models trained with optimized remote loading achieve the same accuracy as local training within identical time budgets. We improve validation IoU by 6-15% and maintain 85-95% GPU utilization versus 0-30% with standard configurations. Code is publicly available at https://github.com/microsoft/pytorch-cloud-geotiff-optimization

Figures

Figures reproduced from arXiv: 2506.06235 by the authors.

Figure 1
Figure 1. Sentinel-2 image with internal GeoTIFF block structure overlaid as black grid squares. Colored rectangles show four windowed read patterns and their data loading behavior (colored backgrounds). (A) shows an aligned read that matches block boundaries, efficiently loading only the requested data from a single block. (B) demonstrates a random window read of block size that intersects four blocks, requiring 4× more data… view at source ↗
Figure 2
Figure 2. Training on SSD. Top row: Validation IoU over time shows optimal configurations (dashed lines) achieving marginal improvements over default configurations (solid lines) for Vaihingen (blue, 0.6 vs 0.5), while reaching similar final performance for Potsdam (red, 0.6) and DFC-22 (green, 0.17) datasets. Bottom row: GPU utilization reveals the key advantage of optimal configurations, which maintain consistent high utili… view at source ↗
Figure 3
Figure 3. Training on Azure Cloud Storage. Top row: Validation IoU over time shows optimal configurations (dashed lines) consistently outperforming default ones (solid lines) across Vaihingen (blue, 0.6 vs 0.45), Potsdam (red, 0.6 vs 0.5), and DFC-22 (green, 0.18 vs 0.12) datasets. Bottom row: GPU utilization demonstrates sustained high usage (80–90%) for optimal configurations versus intermittent, low utilization (≤ 30%) for… view at source ↗

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

27 extracted references · 18 canonical work pages

  1. [1]

    Efficient pytorch i/o library for large datasets, many files, many gpus

    Aizman, A., Maltby, G., and Breuel, T. Efficient pytorch i/o library for large datasets, many files, many gpus. https://tinyurl.com/348yfcbu, August 2020. Accessed: 2025-05-12

    work page 2020

  2. [2]

    Optuna: A next-generation hyperparameter optimization framework

    Akiba, T., Sano, S., Yanase, T., Ohta, T., and Koyama, M. Optuna: A next-generation hyperparameter optimization framework. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining , 2019

    work page 2019

  3. [3]

    Geotiff compression optimization guide, 2018

    Alberti, K. Geotiff compression optimization guide, 2018. URL https://kokoalberti.com/articles/geotiff-compression-optimization-guide/. Accessed: 2025-05-19

    work page 2018

  4. [4]

    Algorithms for hyper-parameter optimization

    Bergstra, J., Bardenet, R., Bengio, Y., and K \'e gl, B. Algorithms for hyper-parameter optimization. Advances in neural information processing systems, 24, 2011

    work page 2011

  5. [5]

    Cogeo: Cloud-optimized geospatial ecosystem

    CoGeo Initiative . Cogeo: Cloud-optimized geospatial ecosystem. https://cogeo.org/, 2025. Accessed: 2025-05-12

    work page 2025

  6. [6]

    Dawnbench: An end-to-end deep learning benchmark and competition

    Coleman, C., Narayanan, D., Kang, D., Zhao, T., Zhang, J., Nardi, L., Bailis, P., Olukotun, K., R \'e , C., and Zaharia, M. Dawnbench: An end-to-end deep learning benchmark and competition. Training, 100 0 (101): 0 102, 2017

    work page 2017

  7. [7]

    Consortium, O. G. et al. Zarr storage specification 2.0 community standard, 2023

    work page 2023

  8. [8]

    Sentinel-2: Esa's optical high-resolution mission for gmes operational services

    Drusch, M., Del Bello, U., Carlier, S., Colin, O., Fernandez, V., Gascon, F., Hoersch, B., Isola, C., Laberinti, P., Martimort, P., et al. Sentinel-2: Esa's optical high-resolution mission for gmes operational services. Remote sensing of Environment, 120: 0 25--36, 2012

    work page 2012

Show all 27 references
  1. [9]

    J., Irakulis-Loitxate, I., and Guanter, L

    Gorroño, J., Varon, D. J., Irakulis-Loitxate, I., and Guanter, L. Understanding the potential of sentinel-2 for monitoring methane point emissions. Atmospheric Measurement Techniques, 16 0 (1): 0 89--102, 2023. doi:10.5194/amt-16-89-2023

    2023 doi

  2. [10]

    Cloud native data loaders for machine learning using zarr and xarray, March 2024

    Hamman, J. Cloud native data loaders for machine learning using zarr and xarray, March 2024. URL https://earthmover.io/blog/cloud-native-dataloader/. Accessed: 2025-05-12

    2024

  3. [11]

    C., Boulch, A., Lefevre, S., and Saux, B

    H \"a nsch, R., Persello, C., Vivone, G., Navarro, J. C., Boulch, A., Lefevre, S., and Saux, B. The 2022 ieee grss data fusion contest: Semisupervised learning [technical committees]. IEEE Geoscience and Remote Sensing Magazine, 10 0 (1): 0 334--337, 2022

    2022

  4. [12]

    Deep residual learning for image recognition

    He, K., Zhang, X., Ren, S., and Sun, J. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp.\ 770--778, 2016

    2016

  5. [13]

    Isprs 2d semantic labeling benchmark – vaihingen and potsdam, 2014

    International Society for Photogrammetry and Remote Sensing (ISPRS) . Isprs 2d semantic labeling benchmark – vaihingen and potsdam, 2014. URL https://www.isprs.org/education/benchmarks/UrbanSemLab/semantic-labeling.aspx. Dataset

    2014

  6. [14]

    J., and Leong, W

    Jones, M., Hamman, J. J., and Leong, W. J. Xbatcher-a python package that simplifies feeding xarray data objects to machine learning libraries. In 103rd AMS Annual Meeting. AMS, 2023

    2023

  7. [15]

    s3-connector-for-pytorch

    Labs, A. s3-connector-for-pytorch. https://github.com/awslabs/s3-connector-for-pytorch, 2023. Accessed: 2025-05-12

    2023

  8. [16]

    M., Salman, H., and Madry, A

    Leclerc, G., Ilyas, A., Engstrom, L., Park, S. M., Salman, H., and Madry, A. Ffcv: Accelerating training by removing data bottlenecks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.\ 12011--12020, 2023

    2023

  9. [17]

    and Hutter, F

    Loshchilov, I. and Hutter, F. Decoupled weight decay regularization. arXiv preprint arXiv:1711.05101, 2017

    2017 arXiv

  10. [18]

    Mlperf training benchmark

    Mattson, P., Cheng, C., Diamos, G., Coleman, C., Micikevicius, P., Patterson, D., Tang, H., Wei, G.-Y., Bailis, P., Bittorf, V., et al. Mlperf training benchmark. Proceedings of Machine Learning and Systems, 2: 0 336--349, 2020

    2020

  11. [20]

    Analyzing and mitigating data stalls in dnn training

    Mohan, J., Phanishayee, A., Raniwala, A., and Chidambaram, V. Analyzing and mitigating data stalls in dnn training. arXiv preprint arXiv:2007.06775, 2020 b

    2007 arXiv

  12. [21]

    DALI : Nvidia data loading library

    NVIDIA. DALI : Nvidia data loading library. https://github.com/NVIDIA/DALI, 2023. Accessed: 2025-05-12

    2023

  13. [22]

    Pytorch: An imperative style, high-performance deep learning library

    Paszke, A. Pytorch: An imperative style, high-performance deep learning library. arXiv preprint arXiv:1912.01703, 2019

    1912 arXiv

  14. [23]

    R., Murayama, Y., and Ranagalage, M

    Phiri, D., Simwanda, M., Salekin, S., Nyirenda, V. R., Murayama, Y., and Ranagalage, M. Sentinel-2 data for land cover/use mapping: A review. Remote Sensing, 12 0 (14): 0 2291, 2020. doi:10.3390/rs12142291

    2020 doi

  15. [24]

    U-net: Convolutional networks for biomedical image segmentation

    Ronneberger, O., Fischer, P., and Brox, T. U-net: Convolutional networks for biomedical image segmentation. In Medical image computing and computer-assisted intervention--MICCAI 2015: 18th international conference, Munich, Germany, October 5-9, 2015, proceedings, part III 18, ...

    2015

  16. [25]

    L., Araus, J

    Segarra, J., Buchaillot, M. L., Araus, J. L., and Kefauver, S. C. Remote sensing for precision agriculture: Sentinel-2 improved features and applications. Agronomy, 10 0 (5): 0 641, 2020. doi:10.3390/agronomy10050641

    2020 doi

  17. [26]

    O., McFarland, M., Emanuele, R., Morris, D., and Augspurger, T

    Source, M. O., McFarland, M., Emanuele, R., Morris, D., and Augspurger, T. microsoft/planetarycomputer: October 2022, October 2022. URL https://doi.org/10.5281/zenodo.7261897

    2022 doi

  18. [27]

    Profiling and improving the pytorch dataloader for high-latency storage: A technical report

    Svogor, I., Eichenberger, C., Spanring, M., Neun, M., and Kopp, M. Profiling and improving the pytorch dataloader for high-latency storage: A technical report. arXiv preprint arXiv:2211.04908, 2022

    2022 arXiv

  19. [28]

    Environmental impacts of earth observation data in the constellation and cloud computing era

    Wilkinson, R., Mleczko, M., Brewin, R., Gaston, K., Mueller, M., Shutler, J., Yan, X., and Anderson, K. Environmental impacts of earth observation data in the constellation and cloud computing era. Science of The Total Environment, 909: 0 168584, 2024

    2024

Pith tools

Reviewed August 7, 2026 · model on record in the stance chip above.