Pith. sign in

REVIEW 4 major objections 5 minor 20 references

Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal

T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read A masked autoencoder pre-trained on natural images, fine-tuned with a patch-wise GAN, removes clouds from remote sensing images and beats existing GAN-based methods on the RICE benchmarks.

desk verdict Plausible idea with large reported gains, but the test-time resolution gap between the 224×224 ViT generator and 512×512 RICE images makes the main comparison unverified. read the letter →

arxiv 2501.05265 v1 pith:QOWIBE4Q submitted 2025-01-09 cs.CV eess.IV

Wanli Ma , Oktay Karakus , Paul L. Rosin This is my paper · ORCID
classification cs.CVeess.IV
keywords cloudremovalmaskedautoencoderpatch-GANtransferlearningremotesensingimagereconstruction
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This paper claims that a masked autoencoder trained on natural images can be transferred to the task of removing clouds from satellite imagery. The approach fine-tunes an ImageNet-pretrained MAE as the generator of a generative adversarial network, with a patch-wise discriminator that judges each 16×16 patch separately. On the RICE1 and RICE2 benchmarks the authors report PSNR values of 33.66 and 34.06 and SSIM values of 0.976 and 0.955, above the GAN baselines they compare with. The paper also cautions that comparisons with some state-of-the-art methods rest on unclear train/test split details, so those numbers are less certain.

What carries the argument

The load-bearing component is the masked autoencoder (MAE), a vision-transformer autoencoder that reconstructs a full image from randomly masked patches; here it serves as the generator of a GAN and is fine-tuned from an ImageNet-pretrained checkpoint. Input images are cropped to 224×224 and divided into 196 patches of 16×16 pixels, and the decoder outputs cloud-free patches. The second component is a patch-wise discriminator: a fully connected network that flattens each 16×16 RGB patch into a 768-dimensional vector and outputs a real/fake score per patch. The training objective is the sum of MSE between the prediction and the cloud-free ground truth and a patch-based GAN loss, with layer-wise learning rate decay to keep early general features intact while adapting later layers to the task.

What would settle it

Rerun Conditional GAN, Cycle GAN, and SpA GAN on the same RICE1 and RICE2 splits, crops, and metric code used by the proposed method, then compare PSNR and SSIM. If the reported margins shrink or disappear, the central claim of improvement would be shown to rest on protocol mismatch instead of the model itself.

Watch

Extended reading notes

Core claim

The central claim is that reconstructive pretraining transfers to cloud removal. An MAE, whose encoder is a vision transformer operating on visible patches and whose decoder fills masked ones, is fine-tuned with a patch-wise GAN loss so that the cloud-obscured areas are treated as masks to reconstruct. The paper reports performance on RICE1 and RICE2: PSNR 33.659 and 34.056, SSIM 0.976 and 0.955, exceeding the Conditional GAN, Cycle GAN, and SpA GAN baselines, and SSIM scores competitive with or above published state-of-the-art methods such as CMNet.

Load-bearing premise

The claimed superiority over other GAN-based methods assumes that the baseline PSNR and SSIM values taken from [19] were measured under the same train/test splits, preprocessing, and metric implementation as the proposed method's numbers.

Editorial extensions

If this is right

  • The method's reported PSNR and SSIM on RICE1 and RICE2 beat the three GAN baselines it is directly compared with, indicating a transferable advantage from reconstructive pretraining.
  • The patch-wise discriminator, instead of an image-level one, allows the adversarial loss to act on local 16×16 regions, which is how the paper handles the spatial complexity of remote sensing scenes.
  • Because the pre-trained MAE checkpoint is public and the fine-tuning is light enough for a single NVIDIA 3060 GPU, the approach can be reproduced without large compute budgets.
  • The SSIM values on RICE1 (0.976) and RICE2 (0.955) place the method at or above the published state-of-the-art values compared in Table II, though those comparisons are qualified by unclear data splits in prior work.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the transfer works as claimed, MAE-style masked reconstruction could become a generic pretraining backbone for other structured-occlusion tasks in remote sensing, such as shadow or haze removal, because the encoder already learns to reason about missing patches.
  • The paper's own caveat about unclear train/test splits in the state-of-the-art comparison suggests the SSIM rankings in Table II could shift when all methods are evaluated under one shared protocol.
  • A natural testable extension is to swap the fully connected patch discriminator for a convolutional or transformer-based discriminator and measure whether local realism and PSNR improve further.
  • Because training crops are 224×224 while RICE images are 512×512, full-resolution behavior is untested; evaluating on full images could show whether the patch-level realism generalizes beyond the crop size.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 5 minor

Summary. The paper proposes a cloud-removal method for remote sensing images that fine-tunes a pre-trained Masked Autoencoder (MAE) ViT-large model with a patch-wise GAN discriminator. The generator is initialized from an ImageNet-pretrained MAE with GAN loss, then fine-tuned on the RICE1/RICE2 datasets using a combination of MSE and patch-level GAN losses with layer-wise learning-rate decay. Experiments report PSNR/SSIM values in Table I showing large margins over Conditional GAN, Cycle GAN, and SpA GAN baselines taken from reference [19], and SSIM comparisons in Table II against several state-of-the-art methods, while acknowledging that the authors of those methods did not disclose their train/test splits. The paper claims significant improvements in cloud-removal performance and competitive results against state-of-the-art methods.

Significance. If the quantitative claims are reproducible and the comparisons are fair, the paper would make a useful contribution by demonstrating how a natural-image reconstruction model can be transferred to cloud removal with a patch-wise GAN objective. The idea is simple and potentially of interest to the remote sensing community, and the paper is honest about the limitations of the Table II comparison. However, the central evidence is currently not self-contained: the test-time input resolution is unspecified, the Table I baselines are copied from another paper without verification of protocol, and no variance or multiple-run information is provided. The multi-spectral language in the abstract and introduction is also not supported by the RGB experiments. These gaps prevent the paper from being accepted in its current form.

major comments (4)
  1. [Section III-A and Section IV-A] The generator is described as a ViT that operates on 196 patches of 16×16 pixels, i.e., a fixed 224×224 input, and during training images are randomly cropped to 224×224. RICE1 and RICE2 images are 512×512. The paper never states how 512×512 test images are fed to the model: whether they are resized to 224×224, center-cropped, or tiled and stitched. This directly affects the PSNR/SSIM values in Table I and their comparability with baselines from [19]. If test images are resized, the metrics are computed at reduced resolution and compared with baselines evaluated at native resolution; if center-cropped, only a subregion is evaluated; if tiled, the stitching procedure is missing. Please specify the exact test-time protocol and, ideally, rerun the baselines under the same protocol.
  2. [Table I and Section IV-C] The baseline PSNR and SSIM values are taken directly from [19] with no verification that [19] used the same train/test split, preprocessing, or metric calculation on RICE1 and RICE2. The paper's own caveat about undisclosed data splits for the methods in Table II applies equally to the GAN baselines in Table I. No standard deviations or multiple-run results are reported for the proposed method, so the margins over SpA GAN (PSNR 33.659 vs 30.232 on RICE1, 34.056 vs 28.368 on RICE2) may be artifacts of protocol mismatch or evaluation variance. Please report the proposed method's results on the exact split used by [19], and provide error bars or at least confirm that the reported numbers are stable across runs.
  3. [Abstract and Section III-B] The paper repeatedly describes the input as 'multi-spectral remote sensing images,' but the discriminator is described as flattening 'RGB channels' (Section III-B, 16×16×3 = 768 values per patch) and the RICE dataset used in the experiments is a color-image dataset. No multi-spectral experiment is performed, so the claim that the method extracts information from multi-spectral imagery is unsupported. Please either restrict the claims to RGB inputs or add an experiment on a multi-spectral dataset (e.g., Sentinel-2 or Landsat imagery with spectral bands beyond RGB).
  4. [Table II and Section V (Conclusion)] The paper explicitly states that the authors of CMNet [20] and other methods 'did not disclose critical details about their dataset configurations, such as the specific splits used for training, testing, and validation.' Given this acknowledged protocol mismatch, the conclusion that the proposed method 'demonstrates competitive results' is stronger than the evidence supports. Please either obtain the exact evaluation protocols and splits from the authors of [20] and rerun the comparison, or soften the conclusion to state that the proposed method achieves favorable SSIM values on the paper's own splits under a best-effort comparison.
minor comments (5)
  1. [Equation (1)] There is a typo in Equation (1): 'gks' should be 'g_k', and 'weight and height' should be 'width and height'. The summation indexing is also confusing; please clarify that the sum runs over all C×W×H pixels.
  2. [Section III-C] The layer-wise learning rate decay schedule is described qualitatively but no concrete values (base learning rate, decay factor, number of layers) are given. For reproducibility, please include the exact schedule used in the experiments.
  3. [Table I] Reporting only single PSNR/SSIM values without standard deviations or the number of test images makes it impossible to assess the significance of the reported margins. Please add error bars or variance information.
  4. [Section IV-B] The paper states the experiments ran on an NVIDIA 3060 GPU without giving training time or model size; providing these would help readers assess the practical cost of the method.
  5. [Section IV-A] The sentence 'Following [19], we used 1/5 training data for validation during the training phase' is ambiguous: it is unclear whether the validation split is taken from the training set (leaving fewer than 400/588 training pairs) or whether an additional validation set is used. Please clarify the exact split sizes.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity found; the results are measured on external benchmarks and are not derived from the paper's assumptions.

full rationale

The paper makes an empirical claim: fine-tuning an ImageNet-pretrained MAE with a patch-wise GAN improves cloud removal on RICE1/RICE2, as measured by PSNR and SSIM. There is no derivation chain in which an output quantity is defined in terms of the quantity it is claimed to predict, and no fitted parameter is renamed as a prediction. The generator, discriminator, and loss functions (Eqs. 1-7) are standard supervised objectives; the reported Table I and Table II numbers are measured model outputs compared against external baselines. The MAE pretrained weights come from an external source, and the RICE dataset is an external benchmark, so the central claim is not supported by a self-citation chain. Potential concerns such as the unspecified test-time resolution for 512x512 images and the possibility that baseline PSNR/SSIM values from [19] were computed under different protocols are evaluation-validity risks, not circularity: they concern whether the comparison is fair, not whether the result is equivalent to its inputs by construction. No load-bearing self-citation appears, and no uniqueness theorem or ansatz is imported from the authors' prior work. Therefore the appropriate circularity score is 0.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's quantitative claim depends on four unverified assumptions: ImageNet transfer, comparable baseline splits, accurate external numbers, and the sufficiency of RGB input. It also hides at least three training hyperparameters. No new physical or model entities are invented.

free parameters (3)
  • Layer-wise learning rate decay schedule
    The paper says a layer-wise decay is used but gives no numerical values; the exact schedule affects fine-tuning outcomes.
  • Generator loss weighting (MSE vs GAN)
    Equation (1) and the GAN loss are combined, but the relative weight between them is not specified, leaving a hidden tuning knob.
  • Training epochs and batch size
    No training duration or batch size is stated, so the reported metrics cannot be reproduced exactly.
assumptions (4)
  • domain assumption ImageNet-pretrained MAE features transfer to cloud removal in remote sensing imagery.
    The entire approach rests on fine-tuning rather than training from scratch; the paper provides no analysis or ablation that quantifies the contribution of the pretrained weights versus the GAN training.
  • domain assumption The RICE dataset splits used in this paper match the splits used for the baseline results cited from [19].
    The comparison tables assume that the numbers from [19] were produced under equivalent train/test conditions; the paper does not verify this.
  • domain assumption The reported baseline numbers from [19] are accurate and were computed with the same metrics and preprocessing.
    Table I reuses values from another paper, so the claimed superiority depends on those external numbers being correct and comparable.
  • ad hoc to paper The method's RGB input is sufficient to represent the remote sensing imagery in the RICE datasets.
    The paper frames the task as multi-spectral while the implementation only uses RGB, an unexamined simplification.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal." pith.science (2026). https://pith.science/paper/QOWIBE4Q

@misc{pith2026250105265,
  author       = {Pith},
  title        = {Pith review of: Patch-GAN Transfer Learning with Reconstructive Models for Cloud Removal},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/QOWIBE4Q}},
  note         = {Machine review of arXiv:2501.05265}
}
read the original abstract

Cloud removal plays a crucial role in enhancing remote sensing image analysis, yet accurately reconstructing cloud-obscured regions remains a significant challenge. Recent advancements in generative models have made the generation of realistic images increasingly accessible, offering new opportunities for this task. Given the conceptual alignment between image generation and cloud removal tasks, generative models present a promising approach for addressing cloud removal in remote sensing. In this work, we propose a deep transfer learning approach built on a generative adversarial network (GAN) framework to explore the potential of the novel masked autoencoder (MAE) image reconstruction model in cloud removal. Due to the complexity of remote sensing imagery, we further propose using a patch-wise discriminator to determine whether each patch of the image is real or not. The proposed reconstructive transfer learning approach demonstrates significant improvements in cloud removal performance compared to other GAN-based methods. Additionally, whilst direct comparisons with some of the state-of-the-art cloud removal techniques are limited due to unclear details regarding their train/test data splits, the proposed model achieves competitive results based on available benchmarks.

Figures

Figures reproduced from arXiv: 2501.05265 by the authors.

Figure 1
Figure 1. The overall framework of the proposed deep transfer learning. [PITH_FULL_IMAGE:figures/full_fig_p003_1.png] view at source ↗
Figure 2
Figure 2. Dataset samples of RICE dataset. The top row displays cloudy images, [PITH_FULL_IMAGE:figures/full_fig_p003_2.png] view at source ↗
Figure 3
Figure 3. Visual results on the RICE1 (top) and RICE2 (bottom) datasets. From [PITH_FULL_IMAGE:figures/full_fig_p004_3.png] view at source ↗

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

20 extracted references · 17 canonical work pages

  1. [19]

    Cloud removal for remote sensing imagery via spa- tial attention generative adversarial network,

    H. Pan, “Cloud removal for remote sensing imagery via spa- tial attention generative adversarial network,” arXiv preprint arXiv:2009.13015, 2020

    arXiv 2009

  2. [20]

    Cascaded memory network for opti- cal remote sensing imagery cloud removal,

    J. Liu, B. Pan, and Z. Shi, “Cascaded memory network for opti- cal remote sensing imagery cloud removal,” IEEE Transactions on Geoscience and Remote Sensing , vol. 62, pp. 1–11, 2024

    work page 2024

  3. [1]

    Thick cloud removal under land cover changes using multisource satellite imagery and a spatiotemporal attention network,

    H. Liu, B. Huang, and J. Cai, “Thick cloud removal under land cover changes using multisource satellite imagery and a spatiotemporal attention network,” IEEE Transactions on Geoscience and Remote Sensing , vol. 61, pp. 1–18, 2023

    work page 2023

  4. [2]

    A spatiotemporal fusion based cloud removal method for remote sensing images with land cover changes,

    H. Shen, J. Wu, Q. Cheng, M. Aihemaiti, C. Zhang, and Z. Li, “A spatiotemporal fusion based cloud removal method for remote sensing images with land cover changes,” IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, vol. 12, no. 3, pp. 862–874, 2019

    work page 2019

  5. [3]

    Joint cloud removal and classification of Sentinel-2 image time series for agricultural land cover mapping in northern Benin,

    S. B. N. Sar `e, R. Gaetano, R. Interdonato, Y . Hountondji, D. Ienco, and C. Dantas, “Joint cloud removal and classification of Sentinel-2 image time series for agricultural land cover mapping in northern Benin,” in IGARSS 2024-2024 IEEE Inter- national Geoscience and Remote Sensing Symposium . IEEE, 2024, pp. 4824–4827

    work page 2024

  6. [4]

    An effective thin cloud removal procedure for visible remote sensing im- ages,

    H. Shen, H. Li, Y . Qian, L. Zhang, and Q. Yuan, “An effective thin cloud removal procedure for visible remote sensing im- ages,” ISPRS Journal of Photogrammetry and Remote Sensing , vol. 96, pp. 224–235, 2014

    work page 2014

  7. [5]

    Masked autoencoders are scalable vision learners,

    K. He, X. Chen, S. Xie, Y . Li, P. Doll ´ar, and R. Girshick, “Masked autoencoders are scalable vision learners,” in Pro- ceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2022, pp. 16 000–16 009

    work page 2022

  8. [6]

    Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion,

    A. Meraner, P. Ebel, X. X. Zhu, and M. Schmitt, “Cloud removal in Sentinel-2 imagery using a deep residual neural network and SAR-optical data fusion,” ISPRS Journal of Photogrammetry and Remote Sensing , vol. 166, pp. 333–346, 2020

    work page 2020

Show all 20 references
  1. [7]

    Blind cloud and cloud shadow removal of multitemporal images based on total variation regularized low-rank sparsity decomposition,

    Y . Chen, W. He, N. Yokoya, and T.-Z. Huang, “Blind cloud and cloud shadow removal of multitemporal images based on total variation regularized low-rank sparsity decomposition,” ISPRS Journal of Photogrammetry and Remote Sensing , vol. 157, pp. 93–107, 2019

    2019

  2. [8]

    Thick clouds removing from multitemporal Landsat images using spatiotemporal neural networks,

    Y . Chen, Q. Weng, L. Tang, X. Zhang, M. Bilal, and Q. Li, “Thick clouds removing from multitemporal Landsat images using spatiotemporal neural networks,” IEEE Transactions on Geoscience and Remote Sensing , vol. 60, pp. 1–14, 2020

    2020

  3. [9]

    Thin cloud removal from single satellite images,

    J. Liu, X. Wang, M. Chen, S. Liu, X. Zhou, Z. Shao, and P. Liu, “Thin cloud removal from single satellite images,”Optics Express, vol. 22, no. 1, pp. 618–632, 2014

    2014

  4. [10]

    Cloud removal methodology from modis snow cover product,

    A. Gafurov and A. B ´ardossy, “Cloud removal methodology from modis snow cover product,” Hydrology and Earth System Sciences, vol. 13, no. 7, pp. 1361–1373, 2009

    2009

  5. [11]

    A modified homomorphism filtering algorithm for cloud removal,

    X. Wang, M. Li, and H. Tang, “A modified homomorphism filtering algorithm for cloud removal,” in 2010 International Conference on Computational Intelligence and Software Engi- neering, 2010, pp. 1–4

    2010

  6. [12]

    Cloud-GAN: Cloud removal for Sentinel-2 imagery using a cyclic consistent generative ad- versarial networks,

    P. Singh and N. Komodakis, “Cloud-GAN: Cloud removal for Sentinel-2 imagery using a cyclic consistent generative ad- versarial networks,” in IGARSS 2018-2018 IEEE International Geoscience and Remote Sensing Symposium . IEEE, 2018, pp. 1772–1775

    2018

  7. [13]

    Thin cloud removal in optical remote sensing images based on generative adversarial networks and physical model of cloud distortion,

    J. Li, Z. Wu, Z. Hu, J. Zhang, M. Li, L. Mo, and M. Molinier, “Thin cloud removal in optical remote sensing images based on generative adversarial networks and physical model of cloud distortion,” ISPRS Journal of Photogrammetry and Remote Sensing, vol. 166, pp. 373–389, 2020

    2020

  8. [14]

    An image is worth 16 ×16 words: Trans- formers for image recognition at scale,

    A. Dosovitskiy, “An image is worth 16 ×16 words: Trans- formers for image recognition at scale,” arXiv preprint arXiv:2010.11929, 2020

    2010 arXiv

  9. [15]

    A survey on deep transfer learning,

    C. Tan, F. Sun, T. Kong, W. Zhang, C. Yang, and C. Liu, “A survey on deep transfer learning,” in Artificial Neural Networks and Machine Learning–ICANN 2018: 27th International Con- ference on Artificial Neural Networks, Rhodes, Greece, October 4-7, 2018, Proceedings, Part III ...

    2018

  10. [16]

    Convolutional neural network-driven improvements in global cloud detection for Landsat 8 and transfer learning on Sentinel-2 imagery,

    S. Pang, L. Sun, Y . Tian, Y . Ma, and J. Wei, “Convolutional neural network-driven improvements in global cloud detection for Landsat 8 and transfer learning on Sentinel-2 imagery,” Remote Sensing, vol. 15, no. 6, p. 1706, 2023

    2023

  11. [17]

    Cloud removal on satellite image using transfer learning based generative adversar- ial network,

    S. Ahn, S. Kim, J. Do, J. Park, and J. Kang, “Cloud removal on satellite image using transfer learning based generative adversar- ial network,” in 2020 International Conference on Information and Communication Technology Convergence (ICTC) . IEEE, 2020, pp. 203–205

    2020

  12. [18]

    A remote sensing image dataset for cloud removal,

    D. Lin, G. Xu, X. Wang, Y . Wang, X. Sun, and K. Fu, “A remote sensing image dataset for cloud removal,” arXiv preprint arXiv:1901.00600, 2019

    1901 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.