Pith. sign in

REVIEW 3 major objections 4 minor 36 references

SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection

T0 review · 3 major / 4 minor · reviewed 2026-08-10 · deepseek-v4-flash

Pith's one-line read SpecTf shows that per-pixel spectra alone—no spatial or temporal context—can detect clouds at ROC AUC 0.982, beating the operational EMIT mask (0.606) and matching larger models.

desk verdict A compact pixelwise spectral transformer that matches much larger ML baselines and transfers zero-shot across instruments; the label-selection bias is real but not fatal, and the paper deserves peer review. read the letter →

arxiv 2501.04916 v1 pith:JNZKHYQ4 submitted 2025-01-09 cs.LG

classification cs.LG
keywords cloudmaskingimagingspectroscopytransformersdeeplearningEMITAVIRIS-NGattentioninterpretabilityper-pixelclassification
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

SpecTf is a compact transformer for imaging spectroscopy that detects clouds pixel by pixel using only each pixel's top-of-atmosphere reflectance spectrum, with no spatial or temporal context. The paper claims that on held-out EMIT scenes this 20,000-parameter model reaches a ROC AUC of 0.982, far above the 0.606 of the operational EMIT L2A threshold mask and nearly identical to much larger machine-learning baselines (GBT within 0.005 AUC, ANN within 0.004 F1). What would make this matter is that it removes the need for spatial structure or repeated observations, so cloud screening can be done for any single scene, and because the input is wavelength-position pairs rather than a fixed channel grid, the same trained model can be applied to a different instrument with different spectral sampling. The paper also argues that the self-attention weights are directly interpretable, concentrating on known water-vapor, O2, and CO2 absorption features. If these claims hold, a single lightweight per-pixel model could replace coarse threshold baselines across current and planned VSWIR missions.

What carries the argument

The central object is the Spectroscopic Transformer (SpecTf), a single-encoder transformer that treats a pixel's spectrum as a length-n sequence of (wavelength, reflectance) pairs. Its defining choice is concatenating the centered absolute wavelength to each reflectance before a learned embedding, instead of using a fixed additive positional encoding, so that the model sees each channel's physical identity; a single multi-head self-attention layer then compares every wavelength with every other, and a max-pool over the sequence aggregates the most informative spectral signal before the classification head. The variable-length, wavelength-labeled input is what lets the same weights run on EMIT's 268 bands and AVIRIS-NG's 425 bands without retraining. The attention-weight sum over queries, which the authors call the attention spectrum, is the interpretability mechanism: it assigns each wavelength an importance score for a given prediction.

What would settle it

A direct test would be to take a set of scenes with independently verified cloud labels from lidar or radar cloud retrievals, or from radiative-transfer simulation, and compare SpecTf's mask against those labels. If SpecTf's ROC AUC on such labels drops toward the baseline's, or its attention weights no longer track water-vapor and oxygen absorption features, the reported generalization would be an artifact of the annotation procedure rather than a learned physical rule.

Watch

Extended reading notes

Core claim

The central claim is that a transformer trained on spectra as sequences, not on images or on independent band values, can learn the physical signatures of clouds well enough to outperform the operational threshold-based cloud mask and match gradient-boosted and neural-network references with two orders of magnitude fewer parameters. SpecTf encodes each band as a (wavelength, reflectance) pair, runs one multi-head self-attention layer across the 268-band sequence, and pools with a maximum over wavelengths before classification. Applied to held-out EMIT scenes it reports ROC AUC 0.982 versus 0.606 for the L2A baseline, with true-positive rate 0.944 at false-positive rate 0.039; applied zero-shot to an AVIRIS-NG flightline it detects the clouds present in that scene, with some false positives over agricultural and urban areas. The authors take the attention spectra as evidence that the model has learned real atmospheric absorption features, especially the 1300–1500 nm water-vapor band, rather than sensor-specific artifacts.

Load-bearing premise

The load-bearing premise is that the human 'clear' and 'cloud' labels are correct enough to train and evaluate on; the methods section concedes that labelers could not consistently separate thin cirrus from opaque cloud or cloud shadow, so classes were merged, and a second batch of scenes was chosen specifically to fix the model's mistakes.

Editorial extensions

If this is right

  • A single-scene cloud mask no longer needs spatial structure or repeat-pass observations, so the approach applies to scenes with fragmented or infrequent temporal coverage.
  • The same trained model can be run on a different imaging spectrometer with a different band count and spacing, as demonstrated by the EMIT-to-AVIRIS-NG transfer.
  • Attention weights provide a per-pixel map of which wavelengths drove the prediction, allowing cloud masks to be audited against known gas-absorption features without a separate explanation model.
  • At about 20,000 learned parameters, SpecTf achieves the reported accuracy with far less capacity than the ANN reference, reducing the training and deployment burden for data-driven spectral screening.
  • The wavelength-labeled sequence design is not tied to cloud detection; the same architecture can be pointed at other spectral classification and retrieval tasks, as the authors state is planned.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • The paper does not test whether SpecTf's smooth wavelength embedding generalizes to larger calibration shifts than the EMIT-to-AVIRIS-NG difference; a controlled experiment perturbing channel centers or response functions would clarify the limits of the instrument-agnostic claim.
  • Because the labels merged opaque clouds with optically clear cirrus, the binary model cannot answer the next natural question of how much a pixel is obscured; a regression version predicting optical depth would need simulated labels, which the paper acknowledges.
  • The lack of vertical streaking compared with the GBT and ANN references is attributed to the sequential representation; a testable extension is to compare crosstrack false-positive rates across many scenes rather than the two qualitative examples shown.
  • A systematic multi-instrument evaluation with several AVIRIS-NG flightlines, different seasons, and different viewing geometries would be the natural next step, since the paper's zero-shot demonstration is a single scene.
Share X Bluesky LinkedIn Reddit HN

Signed reviews

No signed human review yet.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

3 major / 4 minor

Summary. The paper introduces SpecTf, a pixelwise transformer that classifies cloud versus clear pixels from VSWIR reflectance spectra alone, using wavelength values as positional encodings and a maxpool classification head. On held-out EMIT scenes, the authors report that SpecTf reaches ROC AUC 0.982 compared with 0.606 for the operational EMIT L2A threshold mask, matches GBT and ANN classifiers within 0.005 AUC and 0.004 F1 while using about 20,000 learned parameters, produces spatially coherent masks without the vertical streaking seen in the reference ML models, and transfers zero-shot to an AVIRIS-NG scene. The paper also presents attention-weight spectra as an interpretability analysis and makes both the annotated data and model code publicly available.

Significance. If the reported numbers are robust, the work is a useful contribution: it shows that a compact sequence model can outperform a coarse operational threshold baseline on a relevant Earth-observation task, and the cross-instrument zero-shot result is a valuable proof of concept for instrument-agnostic spectroscopy models. The paper is commendably reproducible in intent: data links, code links, trained weights, and a detailed architecture description are provided. The scene-level train/validation split is appropriate for avoiding pixel-level leakage. The main risk is that the evaluation labels and the validation scene selection may not represent the deployment distribution, so the headline Table 1 metrics currently carry unknown selection bias; this is testable and should be fixable within the manuscript's scope.

major comments (3)
  1. [§4.1, Table 1] The evaluation set is not a random sample of EMIT scenes. Section 4.1 states that after an initial stratified batch of 221 scenes, 'another 313 scenes were selected to correct false predictions' made by a preliminary model. The validation set of 69 scenes is drawn from the pooled 534 scenes, so the reported AUC 0.982 and the F-scores may be driven by corrective-selection scenes that are enriched for exactly the failure modes the final model was subsequently tuned against. Please report the validation metrics separately for the original stratified batch and the corrective second batch, and, if possible, evaluate the final model on a fresh set of randomly selected EMIT scenes that were not used in any stage of model development or architecture search. Without this breakdown, the generalization claim in Sections 2.2 and 3 is not yet supported.
  2. [§4.1, Table 1] The ground-truth labels exclude precisely the hard cases that matter for cloud screening. Only pixels that labelers could confidently identify were labeled; ambiguous boundaries, thin cirrus, cloud shadows, and mixed pixels were excluded, and the 'clear'/'cloud shadow' and 'cloud'/'optically clear cirrus' classes were merged because labelers could not annotate them consistently. Consequently the TPR/FPR values in Table 1 are measured on an easy-to-label subset of pixels, not on the full distribution that a deployed cloud mask would encounter. Please quantify the fraction of pixels excluded per scene, report a sensitivity analysis on the excluded or ambiguously labeled pixels where possible, and discuss how the confident-only labeling protocol affects the comparison with the L2A baseline, which does make a prediction on every pixel.
  3. [§2.2, Table 1] No uncertainty quantification is reported for any metric. The differences between SpecTf, GBT, and ANN are small (0.005 in ROC AUC, 0.004 in F1), and the text calls GBT's advantage 'insignificant' without any statistical test. Because the validation set is a single split, the reported metric ordering could easily change with another scene-level split or another training run. Please provide bootstrap confidence intervals over scenes for ROC AUC and F-scores, and ideally repeated training runs with different seeds, so that the 'performs comparably' claim and the baseline comparison have a well-defined error bar.
minor comments (4)
  1. [§4.3, Eqs. (12)-(13)] The attention notation is ambiguous: Eq. (12) writes Wi = softmax(QiK^T / sqrt(dk)) with K as a matrix while Wi is defined as a vector, and Eq. (13) uses V both as the full value matrix and as the value vectors Vj. Please clarify whether the softmax is over the key index j and align the subscripts consistently.
  2. [§2.2] The claim that SpecTf's posterior is 'better calibrated' because its best binary threshold is 0.52 rather than 0.96 or 0.98 is suggestive but not a calibration test. A reliability diagram or expected calibration error would be needed to support this statement.
  3. [§3] The sentence 'A pixelwise model that makes an independent determination with each spectrum will always be superior to a spatial model if spatial features are unnecessary' is too strong; the paper demonstrates one empirical case, not a general theorem. A more hedged formulation would be more appropriate.
  4. [§2.3, Fig. 3] The mean attention spectra show peaks at known absorption features, which is good evidence of physically relevant feature use, but attention weights only indicate importance and not the sign or causal role of a band. The paper already acknowledges the directionality limitation in §4.4; please make the same caveat visible in the Results section where the interpretability claim is first presented.

Circularity Check

0 steps flagged · score 0.0 of 10

No significant circularity: SpecTf's performance claims are empirical and anchored to external baselines and held-out human labels.

full rationale

The central derivation chain is empirical rather than definitional. SpecTf is trained on human-labeled EMIT spectra and evaluated on held-out scenes at the scene level, compared against an operational threshold baseline ([17]) and two reference ML models trained on the same data. The cited baseline is a concrete algorithm from the same institution, but it is used as a benchmark, not as justification for SpecTf's design or results; comparing against it is therefore not circular. The interpretability claim uses the model's own attention weights, but this is an explanatory analysis and is checked against known atmospheric absorption features, so it does not constitute a renamed input or a fitted prediction. Two caveats are data-validity concerns rather than circularity: the best-F1 binary thresholds used for Table 1 are selected on the same validation set, and the second labeling batch was selected to correct a preliminary model's false predictions, which could bias point estimates. However, the headline ROC AUC is threshold-independent and is computed on scenes held out from training by a stratified scene-level split. No equation in the paper reduces to its own input by construction, and no claimed result is merely a refit of its training target. Thus the paper is not circular; the noted caveats belong to evaluation robustness, not circular reasoning.

Assumptions & free parameters 3 free parameters · 4 assumptions · 0 invented entities

The paper's central claim is empirical, so its free parameters are the fitted model weights, the decision threshold, and architecture hyperparameters. No new physical entities are postulated. The main unproved assumptions are label quality, reflectance normalization, attention-as-explanation, and wavelength-center transfer across instruments.

free parameters (3)
  • SpecTf learned weights = ~20,000 parameters
    All model weights are fitted to 3.08 million labeled training spectra; the central performance comparison depends on this fitted model.
  • Decision threshold for SpecTf = 0.52 cloud probability
    Chosen as the best-F1 threshold on the validation set; Table 1 metrics use it. GBT and ANN use thresholds 0.96 and 0.98.
  • Architecture hyperparameters = d_model=64, heads=8, feedforward size 64, dropout 0.1, learning rate 1e-4, batch size 256, 30 epochs
    Selected by Bayesian search over roughly 2,000 trained models; reported performance depends on these choices, though the paper does not quantify sensitivity.
assumptions (4)
  • domain assumption Human-annotated pixel labels are an accurate ground truth for clear versus cloud pixels.
    Section 4.1: labels come from Labelbox and MMGIS; classes were merged because labelers could not distinguish thin cirrus from opaque clouds or cloud shadows. All reported metrics inherit this label noise.
  • domain assumption TOA reflectance computed from L1B radiance and observation geometry is a sufficient normalized input for cloud detection.
    Sections 2.1 and 4.1: the input is a normalized reflectance estimate; there is no atmospheric correction. The method assumes this normalization is adequate across the global training set.
  • domain assumption Attention weights can be read as feature-importance scores for the predicted class.
    Section 2.3 and Equation 23: the paper interprets summed attention as a measure of wavelength relevance. Attention is a known imperfect attribution proxy, and the authors note it does not give directionality.
  • domain assumption Embedding wavelength centers is sufficient for cross-instrument transfer; spectral response functions can be ignored.
    Section 4.5: the zero-shot AVIRIS-NG result uses only wavelength centers. The Discussion acknowledges that real response functions differ between instruments and might need to be included in future work.

how reviews work

0 comments
Cite this review

Pith. "Pith review of SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection." pith.science (2026). https://pith.science/paper/JNZKHYQ4

@misc{pith2026250104916,
  author       = {Pith},
  title        = {Pith review of: SpecTf: Transformers Enable Data-Driven Imaging Spectroscopy Cloud Detection},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/JNZKHYQ4}},
  note         = {Machine review of arXiv:2501.04916}
}
read the original abstract

Current and upcoming generations of visible-shortwave infrared (VSWIR) imaging spectrometers promise unprecedented capacity to quantify Earth System processes across the globe. However, reliable cloud screening remains a fundamental challenge for these instruments, where traditional spatial and temporal approaches are limited by cloud variability and limited temporal coverage. The Spectroscopic Transformer (SpecTf) addresses these challenges with a spectroscopy-specific deep learning architecture that performs cloud detection using only spectral information (no spatial or temporal data are required). By treating spectral measurements as sequences rather than image channels, SpecTf learns fundamental physical relationships without relying on spatial context. Our experiments demonstrate that SpecTf significantly outperforms the current baseline approach implemented for the EMIT instrument, and performs comparably with other machine learning methods with orders of magnitude fewer learned parameters. Critically, we demonstrate SpecTf's inherent interpretability through its attention mechanism, revealing physically meaningful spectral features the model has learned. Finally, we present SpecTf's potential for cross-instrument generalization by applying it to a different instrument on a different platform without modifications, opening the door to instrument agnostic data driven algorithms for future imaging spectroscopy tasks.

Discussion (0). Continue with ORCID to comment.

Reference graph

Works this paper leans on

36 extracted references · 35 canonical work pages

  1. [1]

    Green, R. O. et al. The earth surface mineral dust source investigation: An earth science imaging spectroscopy mission. 2020 IEEE Aerospace Conference 1–15 (2020)

  2. [2]

    Cawse-Nicholson, K. et al. Nasa’s surface biology and geology designated observable: A perspective on surface imaging algorithms. Remote Sensing of Environment 257, 112349 (2021). URL https://www.sciencedirect.com/science/ article/pii/S0034425721000675. 20

  3. [3]

    Nieke, J. et al. The copernicus hyperspectral imaging mission for the environment (CHIME): an overview of its mission, system and planning status. Sensors, Systems, and Next-Generation Satellites XXVII 12729, 1272909 (2023). URL https://doi.org/10.1117/12.2679977

  4. [4]

    & Woodcock, C

    Zhu, Z. & Woodcock, C. E. Object-based cloud and cloud shadow detection in landsat imagery. Remote Sensing of Environment 118, 83–94 (2012). URL https://www.sciencedirect.com/science/article/pii/S0034425711003853

  5. [5]

    Qiu, S., Zhu, Z. & He, B. Fmask 4.0: Improved cloud and cloud shadow detec- tion in landsats 4–8 and sentinel-2 imagery. Remote Sensing of Environment 231, 111205 (2019). URL https://www.sciencedirect.com/science/article/pii/ S0034425719302172

  6. [6]

    Yang, J. et al. Cdnet: Cnn-based cloud detection for remote sensing imagery. IEEE Transactions on Geoscience and Remote Sensing 57, 6195–6211 (2019)

  7. [7]

    Hagolle, O., Huc, M., Pascual, D. V. & Dedieu, G. A multi-temporal method for cloud detection, applied to formosat-2, venµs, landsat and sentinel-2 images. Remote Sensing of Environment 114, 1747–1755 (2010)

  8. [8]

    & Woodcock, C

    Zhu, Z. & Woodcock, C. E. Automated cloud, cloud shadow, and snow detection in multitemporal landsat data: An algorithm designed specifically for monitoring land cover change. Remote Sensing of Environment 152, 217–234 (2014). URL https://www.sciencedirect.com/science/article/pii/S0034425714002259

Show all 36 references
  1. [9]

    & Helmer, E

    Zhu, X. & Helmer, E. H. An automatic method for screening clouds and cloud shadows in optical satellite image time series in cloudy regions. Remote Sens- ing of Environment 214, 135–153 (2018). URL https://www.sciencedirect.com/ science/article/pii/S0034425718302530

  2. [10]

    Green, R. O. et al. Imaging spectroscopy and the airborne visible/infrared imag- ing spectrometer (aviris). Remote Sensing of Environment 65, 227–248 (1998). URL https://www.sciencedirect.com/science/article/pii/S0034425798000649

  3. [11]

    & Goetz, A

    Gao, B.-C. & Goetz, A. F. H. Cloud area determination from aviris data using water vapor channels near 1 µm. Journal of Geophysical Research: Atmospheres 96, 2857–2864 (1991). URL https://agupubs.onlinelibrary.wiley.com/doi/abs/ 10.1029/90JD02394

  4. [12]

    Sun, L. et al. Satellite data cloud detection using deep learning supported by hyperspectral data. International Journal of Remote Sensing 41, 1349–1371 (2020)

  5. [13]

    Giuffrida, G. et al. Cloudscout: A deep neural network for on-board cloud detection on hyperspectral images. Remote Sensing 12 (2020). URL https: //www.mdpi.com/2072-4292/12/14/2205. 21

  6. [14]

    Zhai, H., Zhang, H., Zhang, L. & Li, P. Cloud/shadow detection based on spectral indices for multi/hyperspectral optical remote sensing imagery.ISPRS Journal of Photogrammetry and Remote Sensing 144, 235–253 (2018). URL https://www. sciencedirect.com/science/article/pii/S09242...

  7. [15]

    Thompson, D. R. et al. Rapid spectral cloud screening onboard aircraft and spacecraft. IEEE Transactions on Geoscience and Remote Sensing 52, 6779–6792 (2014)

  8. [16]

    Sandford, M. W. et al. Global cloud property models for real-time triage on board visible–shortwave infrared spectrometers. Atmospheric Measurement Techniques 13, 7047–7057 (2020). URL https://amt.copernicus.org/articles/13/7047/2020/

  9. [17]

    Thompson, D. R. et al. Emit l2a algorithm: Surface reflectance and scence content masks, theoretical basis. Jet Propulsion Laboratory (2024)

  10. [18]

    & Wiscombe, W

    Gao, B.-C., Yang, P., Han, W., Li, R.-R. & Wiscombe, W. J. An algorithm using visible and 1.38-/spl mu/m channels to retrieve cirrus cloud reflectances from aircraft and satellite data. IEEE Transactions on Geoscience and Remote Sensing 40, 1659–1668 (2002)

  11. [19]

    Lundberg, S. M. & Lee, S.-I. Guyon, I. et al. (eds) A unified approach to inter- preting model predictions. (eds Guyon, I. et al.) Advances in Neural Information Processing Systems, Vol. 30 (Curran Associates, Inc., 2017)

  12. [20]

    Qi, Z., Khorram, S. & Li, F. Visualizing deep networks by optimizing with integrated gradients. CVPR workshops 2, 1–4 (2019)

  13. [21]

    Gao, B.-C., Goetz, A. F. & Wiscombe, W. J. Cirrus cloud detection from airborne imaging spectrometer data using the 1.38 µm water vapor band. Geophysical Research Letters 20, 301–304 (1993)

  14. [22]

    Taylor, T. E. et al. Comparison of cloud-screening methods applied to gosat near-infrared spectra. IEEE Transactions on Geoscience and Remote Sensing 50, 295–309 (2011)

  15. [23]

    J., Brown, C

    Pasquarella, V. J., Brown, C. F., Czerwinski, W. & Rucklidge, W. J. Compre- hensive quality assessment of optical satellite imagery using weakly supervised video learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition 2125–2135 (2023)

  16. [24]

    Fast cloud segmentation using convolutional neural networks

    Dr¨ onner, J.et al. Fast cloud segmentation using convolutional neural networks. Remote Sensing 10, 1782 (2018)

  17. [25]

    & Yao, Y

    Zhang, B., Zhang, Y., Li, Y., Wan, Y. & Yao, Y. Cloudvit: A lightweight vision transformer network for remote sensing cloud detection. IEEE Geoscience and Remote Sensing Letters 20, 1–5 (2022). 22

  18. [26]

    Hong, D. et al. Spectralformer: Rethinking hyperspectral image classification with transformers. IEEE Transactions on Geoscience and Remote Sensing 60, 1–15 (2021)

  19. [27]

    & Lin, Z

    He, X., Chen, Y. & Lin, Z. Spatial-spectral transformer for hyperspectral image classification. Remote Sensing 13, 498 (2021)

  20. [28]

    Calef, F. J. & Soliman, T. K. Nasa-ammos mmgis open source software. GitHub (2024). URL https://github.com/NASA-AMMOS/MMGIS

  21. [29]

    & Guestrin, C

    Chen, T. & Guestrin, C. Xgboost: A scalable tree boosting system. Proceedings of the 22nd acm sigkdd international conference on knowledge discovery and data mining 785–794 (2016)

  22. [30]

    Paszke, A. et al. Pytorch: An imperative style, high-performance deep learning library. Advances in neural information processing systems 32 (2019)

  23. [31]

    & Sun, J

    He, K., Zhang, X., Ren, S. & Sun, J. Deep residual learning for image recognition. Proceedings of the IEEE conference on computer vision and pattern recognition 770–778 (2016)

  24. [32]

    Defazio, A. et al. The road less scheduled (2024). 2405.15682

  25. [33]

    Vaswani, A. et al. Guyon, I. et al. (eds) Attention is all you need . (eds Guyon, I. et al. ) Advances in Neural Information Processing Systems , Vol. 30 (Curran Associates, Inc., 2017). URL https://proceedings.neurips.cc/paper files/paper/ 2017/file/3f5ee243547dee91fbd053c1c4...

  26. [34]

    Dosovitskiy, A. et al. An image is worth 16x16 words: Transformers for image recognition at scale. International Conference on Learning Representations (2021)

  27. [35]

    Kenton, J. D. M.-W. C. & Toutanova, L. K. Bert: Pre-training of deep bidirec- tional transformers for language understanding. Proceedings of naacL-HLT 1, 2 (2019)

  28. [36]

    & Abdelzaher, T

    Liu, S., Le, F., Chakraborty, S. & Abdelzaher, T. On exploring attention- based explanation for transformer models in text classification. 2021 IEEE International Conference on Big Data (Big Data) 1193–1203 (2021). 23

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.