REVIEW 4 major objections 5 minor 50 references
Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach
T0 review · 4 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash
Pith's one-line read Training a multi-view Earth-observation model on every non-empty combination of available views, fused with a dynamic merge function that ignores missing views, makes predictions robust to missing sensors and can slightly improve…
desk verdict Solid, incremental robustness trick for multi-view EO models; the abstract oversells full-view gains, but the moderate-missingness claim holds. read the letter →
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
The reading
What carries the argument
The central object is the CoM (Combinations of Missing views) augmentation: from the power set of the view set minus the empty set, each training sample generates 2^m - 1 prediction tasks, one for each combination of available views. The complementary mechanism is the dynamic merge function, a fusion that maps any number of encoded view representations to the same dimension d; the paper uses average, gated softmax-weighted fusion, cross-attention with a learnable fusion token, and LSTM memory fusion. Together they allow the model to ignore missing views at inference without any imputation or forced view alignment.
What would settle it
Run the same CoM models on the same datasets but simulate missingness from cloud masks or actual sensor outages (such as the Sentinel-1B failure window) instead of random whole-view dropping, and test both full-view-trained and partially-missing-trained scenarios; if the robustness advantage of FCoM methods over imputation baselines disappears or reverses under these realistic masks, the central claim is not general. A controlled comparison that applies missingness to random time steps within a view rather than the whole view, at matched overall missing fractions, would also settle whether the benefit comes from ignoring views or from the augmentation effect.
Extended reading notes
Core claim
The paper introduces Combinations of Missing views (CoM), a data-augmentation method that, at the feature level, exposes the model to every non-empty subset of views during training, and combines it with a dynamic merge function that produces a fixed-size fused representation regardless of the number of available views, so missing views are ignored rather than imputed. Across the four datasets, the methods FCoM-av, FCoM-ga, FCoM-cr, and FCoM-me improve robustness to moderate missingness and, in some cases, exceed the full-view performance of the baselines. The authors conclude that integrating CoM with a merge function that ignores the missing views enhances both the predictions and the robustness of multi-view learning models, delivering a single adaptive solution that operates with any combination of available views.
Load-bearing premise
The evaluation assumes that randomly dropping entire views at inference, from a model trained on full-view data, faithfully represents real Earth-observation missingness, which is often partial within a view, correlated with cloud cover or season, and may already be present in training data.
Editorial extensions
If this is right
- A single model can serve any view-availability scenario, eliminating the need for separate missing-data imputation pipelines.
- Training cost stays moderate because the encoders are forwarded once per sample, while the fusion and prediction head process all 2^m - 1 combinations.
- Classification tasks gain a small full-view performance boost from the augmentation effect, while regression tasks gain robustness mainly under moderate missingness.
- Prediction time shrinks as more views are missing, since the model only processes the available data rather than inserting fake values.
Reading between the lines
- The simulated missingness drops entire views wholesale, whereas real EO failures often corrupt patches, orbits, or time steps within a view; a natural test is whether CoM retains its advantage when missingness is partial within a view and correlated with cloud cover or season.
- Because CoM operates at the feature level, it can be layered on top of pre-trained encoders, suggesting a cheap way to retrofit existing multi-view models for robustness without retraining from scratch.
- The paper's observation that simple average fusion is a strong baseline hints that most of the benefit comes from the CoM augmentation itself, and that complex fusions add robustness mainly in extreme missingness; a cleaner ablation would isolate these two contributions.
- The relative-robustness metric PRS favors a simpler ensemble baseline (ESensI-av), while absolute performance favors CoM methods; a combined metric that mixes both level and robustness would make the practical choice clearer for a practitioner.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper proposes a data-augmentation strategy called Combinations of Missing views (CoM) for multi-view learning (MVL) in the Earth Observation (EO) domain. Instead of imputing missing views, the method trains a feature-level fusion model on all non-empty subsets of views, and uses dynamic merge functions (average, gated fusion, cross-attention, memory fusion) whose output dimension is independent of the number of available views. The authors evaluate on four EO datasets (CropHarvest binary and multi-class classification, LFMC and PM25 regression), comparing against several baselines under simulated missing-view scenarios at inference. They report improved robustness under moderate missingness and, in some cases, improved full-view predictive performance.
Significance. If the results hold, the proposed CoM plus dynamic fusion provides a practical single-model solution that can adapt to arbitrary combinations of available views in EO time series, a setting where sensor failures and cloud cover are common. The paper's strengths include a broad empirical comparison across four datasets, both classification and regression tasks, ablations of augmentation levels and fusion architectures (Table 3 and Appendix B.2), and an execution-time comparison. However, the central novelty relative to the authors' prior work is incremental, and the strength of the claims about full-view improvements and the evaluation protocol need qualification. The moderate-missingness robustness claims are reasonably supported for CropH and LFMC by Tables 1 and 2, but the abstract overstates full-view gains and the evaluation does not cover partial within-view missingness.
major comments (4)
- [Abstract / Table 2] The abstract states that the methods 'improve the predictive performance when all views are present' without qualification. Table 2 contradicts this for PM25, where all four FCoM variants (best R2=0.660) are below ITempD-co (R2=0.866), and for LFMC only FCoM-ga (0.700) exceeds ITempD-co (0.691). The introduction correctly says 'in some cases', and the abstract should be reworded to match the evidence.
- [Table 2] The ITempD-co row contains suspicious duplicated values: in LFMC, 'Missing Radar' and 'Missing Optical' are both 0.036±0.100, and 'Only available Optical' and 'Only available Radar' are both -0.036±0.06; in PM25, 'Missing Dynamic' and 'Only available Condition' are both -0.124±0.14, and 'Missing Condition' and 'Only available Dynamic' are effectively identical (0.074±0.035 vs 0.073±0.035). These coincidences are unlikely to be genuine and suggest a data handling or reporting error. Since Table 2 underpins the claim that ITempD-co has poor regression robustness, the authors should verify the baseline numbers and correct the table or explain the duplication.
- [Section 3.3, Eqs. (4)-(5)] The gated fusion is described as a dynamic merge function that 'ignores' missing views, but the implementation zero-imputes missing features and computes the softmax weights over the full view set before masking the missing views' weights to zero. Because the softmax normalization includes the logits of the missing views, the fused vector's scale depends on the number of available views even when the missing features are zero. Thus the missing views are not fully ignored; they affect the normalization and hence the available views' effective weights. The authors should renormalize over the available views only, or explicitly state that the scale dependence on m(i) is intended and characterize its effect.
- [Abstract / Sections 4.2-4.3 and 5] The evaluation protocol, as stated in Sec 4.2, simulates missingness only by dropping entire views at inference from models trained on full-view data. The abstract's claim of 'a single adaptive solution to operate effectively with any combination of available views' is therefore only supported for whole-view dropout, not for the partial within-view missingness (e.g., cloud-corrupted optical time series with some valid dates) or train-time missingness that commonly occur in EO. Although Sec 5 acknowledges this limitation, the abstract and the final conclusions present the result without that qualifier. Either add experiments with partial missingness or narrow the claims accordingly.
minor comments (5)
- [General / Tables 1-2] The paper does not report paired significance tests or effect sizes. Many of the 'best' results in Table 1 overlap with baselines within one standard deviation (e.g., CropH-b full-view, FCoM-ga 0.839±0.005 vs FCoMl-co 0.832±0.007). Consider adding a paired test such as Wilcoxon signed-rank across folds, or at least discussing significance.
- [Figure 2] The performance curves in Figure 2 and Figure 7 are plotted without error bars or confidence intervals, making it difficult to assess the reliability of differences between methods.
- [Section 4.3] The selection of the two top views is based on individually trained models. Please clarify that this selection is performed within each cross-validation fold (or otherwise state why it does not leak test information).
- [Table 1 / Table 2] The table headers read '(4/4) No' and '(6/6) No'; the word 'Missing' is missing. Please correct.
- [Section 4.2 / Data and code availability] The paper states the code 'will be released' and provides a GitHub URL. Since reproducibility is a key strength, please make the code available at review time or include complete pseudocode for the masking step in the gated fusion.
Circularity Check
No circular derivation: CoM is an explicit augmentation technique, dynamic fusion is a design choice, and robustness claims rest on held-out comparisons with external datasets.
full rationale
The paper's central claim is that training a multi-view model on all combinations of missing views (CoM) with a dynamic merge function improves robustness to missing views. This is a data-augmentation method, not a fitted input renamed as a prediction. The training objective (Eq. 9) weights every subset V(j) of views equally, and the evaluation in Sec. 4.2-4.3 measures held-out validation-fold performance after simulating missing views; no test-set parameter is fitted. The dynamic merge functions (average, gated, cross-attention, memory) are defined to operate on available views only, so their missing-view invariance is a stated design property rather than a hidden equivalence; the claimed accuracy and robustness gains are empirical and compared against external baselines (ITempD-co, ISensD-co, FSensD-cr, FCoMl-co, FEmbr-sa, ESensI-av) on four public datasets (CropHarvest, LFMC, PM25). The paper's use of the authors' prior work (Mena et al. 2024a, 2024b) is contextual and comparative, not as a uniqueness theorem or as the sole justification of the central claim. The acknowledged limitation in Sec. 5 — 'we assess the effect of missing views only at inference, assuming a full-view training dataset' — is an external-validity concern about whether simulated whole-view dropout matches real EO missingness, not circularity. No equation reduces to its input by construction, and no fitted parameter is presented as a prediction. The score of 1 reflects the presence of many self-citations that are nonetheless not load-bearing; the derivation itself is self-contained empirical comparison.
Assumptions & free parameters
free parameters (4)
- encoder architecture =
2-layer CNN/MLP, 128 units, 20% dropout
- fusion hyperparameters =
8 heads, 1 layer, 40% dropout for cross-attention; 2 Bi-LSTM layers, 40% dropout for memory
- CoM loss weighting =
equal weight 1/|T| for every view subset
- early stopping patience =
5 epochs
assumptions (5)
- domain assumption Validation missingness is simulated by dropping entire views at inference only, with full-view training data.
- ad hoc to paper Dynamic merge functions with fixed output dimension can represent fused information regardless of the number of available views.
- standard math Zero-imputing missing views inside gated fusion is equivalent to ignoring them.
- domain assumption The four public datasets are representative of EO multi-view tasks with missing views.
- domain assumption Early stopping on full-view validation does not unfairly disadvantage methods trained with missing-view augmentation.
Cite this review
Pith. "Pith review of Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach." pith.science (2026). https://pith.science/paper/ZIEKOY7F
@misc{pith2026250101132,
author = {Pith},
title = {Pith review of: Missing Data as Augmentation in the Earth Observation Domain: A Multi-View Learning Approach},
year = {2026},
howpublished = {\url{https://pith.science/paper/ZIEKOY7F}},
note = {Machine review of arXiv:2501.01132}
}
read the original abstract
Multi-view learning (MVL) leverages multiple sources or views of data to enhance machine learning model performance and robustness. This approach has been successfully used in the Earth Observation (EO) domain, where views have a heterogeneous nature and can be affected by missing data. Despite the negative effect that missing data has on model predictions, the ML literature has used it as an augmentation technique to improve model generalization, like masking the input data. Inspired by this, we introduce novel methods for EO applications tailored to MVL with missing views. Our methods integrate the combination of a set to simulate all combinations of missing views as different training samples. Instead of replacing missing data with a numerical value, we use dynamic merge functions, like average, and more complex ones like Transformer. This allows the MVL model to entirely ignore the missing views, enhancing its predictive robustness. We experiment on four EO datasets with temporal and static views, including state-of-the-art methods from the EO domain. The results indicate that our methods improve model robustness under conditions of moderate missingness, and improve the predictive performance when all views are present. The proposed methods offer a single adaptive solution to operate effectively with any combination of available views.
Figures
Figures from the paper (5 more)
Reference graph
Works this paper leans on
-
[1]
Astruc, G., Gonthier, N., Mallet, C., and Landrieu, L. (2025). Omnisat: Self-supervised modality fusion for earth observation. In European Conference on Computer Vision , pages 409--427. Springer
work page 2025
-
[2]
Audebert, N., Le Saux, B., and Lefèvre, S. (2018). Beyond RGB : Very high resolution urban remote sensing with multimodal deep networks. ISPRS Journal of Photogrammetry and Remote Sensing , 140:20--32
work page 2018
-
[3]
Bouthillier, X., Konda, K., Vincent, P., and Memisevic, R. (2015). Dropout as data augmentation. arXiv preprint arXiv:1506.08700
arXiv 2015
-
[4]
Bugue \ n o, M. and Mendoza, M. (2020). Learning to combine classifiers outputs with the Transformer for text classification. Intelligent Data Analysis , 24(S1):15--41
work page 2020
-
[5]
Camps-Valls, G., Tuia, D., Zhu, X. X., and Reichstein, M. (2021). Deep learning for the Earth Sciences : A comprehensive approach to remote sensing, climate science and geosciences . John Wiley & Sons, New York
work page 2021
-
[6]
Cao, W., Wang, D., Li, J., Zhou, H., Li, L., and Li, Y. (2018). BRITS : Bidirectional recurrent imputation for time series. Advances in Neural Information Processing Systems (NIPS) , 31
work page 2018
-
[7]
Che, X., Zhang, H. K., Li, Z. B., Wang, Y., Sun, Q., Luo, D., and Wang, H. (2024). Linearly interpolating missing values in time series helps little for land cover classification using recurrent or attention networks. ISPRS Journal of Photogrammetry and Remote Sensing , 212:73--95
work page 2024
-
[8]
Chen, S. (2017). PM2.5 Data of Five Chinese Cities . UCI Machine Learning Repository
work page 2017
Show all 50 references
-
[9]
Chen, Y., Zhao, M., and Bruzzone, L. (2024). A novel approach to incomplete multimodal learning for remote sensing data fusion. IEEE Transactions on Geoscience and Remote Sensing
2024
-
[10]
and Lee, J.-S
Choi, J.-H. and Lee, J.-S. (2019). EmbraceNet : A robust deep learning architecture for multimodal classification. Information Fusion , 51:259--270
2019
-
[11]
Cong, Y., Khanna, S., Meng, C., Liu, P., Rozi, E., He, Y., Burke, M., Lobell, D., and Ermon, S. (2022). Satmae: Pre-training transformers for temporal and multi-spectral satellite imagery. Advances in Neural Information Processing Systems , 35:197--211
2022
-
[12]
Devlin, J., Chang, M.-W., Lee, K., and Toutanova, K. (2018). BERT : Pre-training of deep bidirectional transformers for language understanding. In Proceedings of the Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Techn...
2018
-
[13]
Du, W., C \^o t \'e , D., and Liu, Y. (2023). SAITS : Self -attention-based imputation for time series. Expert Systems with Applications , 219:119619
2023
-
[14]
Ebel, P., Garnot, V. S. F., Schmitt, M., Wegner, J. D., and Zhu, X. X. (2023). UnCRtainTS : Uncertainty quantification for cloud removal in optical satellite time series. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 2085--2095
2023
-
[15]
and Schmitt, M
Ekim, B. and Schmitt, M. (2024). Deep occlusion framework for multimodal earth observation data. IEEE Geoscience and Remote Sensing Letters
2024
-
[16]
Fasnacht, L., Renard, P., and Brunner, P. (2020). Robust input layer for neural networks for hyperspectral classification of data with missing bands. Applied Computing and Geosciences , 8:100034
2020
-
[17]
P., Almeida, C
Ferrari, F., Ferreira, M. P., Almeida, C. A., and Feitosa, R. Q. (2023). Fusing Sentinel-1 and Sentinel-2 images for deforestation detection in the Brazilian Amazon under diverse cloud conditions. IEEE Geoscience and Remote Sensing Letters , 20:1--5
2023
-
[18]
Francis, A. (2024). Sensor independent cloud and shadow masking with partial labels and multimodal inputs. IEEE Transactions on Geoscience and Remote Sensing
2024
-
[19]
Gawlikowski, J., Saha, S., Niebling, J., and Zhu, X. X. (2023). Handling unexpected inputs: Incorporating source-wise out-of-distribution detection into SAR -optical data fusion for scene classification. EURASIP Journal on Advances in Signal Processing , 2023(1):47
2023
-
[20]
Ghamisi, P., H \"o fle, B., and Zhu, X. X. (2016). Hyperspectral and LiDAR data fusion using extinction profiles and deep convolutional neural network. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 10(6):3011--3024
2016
-
[21]
Guo, X., Lao, J., Dang, B., Zhang, Y., Yu, L., Ru, L., Zhong, L., Huang, Z., Wu, K., Hu, D., et al. (2024). Skysense: A multi-modal remote sensing foundation model towards universal interpretation for Earth observation imagery. In Proceedings of the IEEE/CVF Conference on Comp...
2024
-
[22]
M., Paoletti, M
Haut, J. M., Paoletti, M. E., Plaza, J., Plaza, A., and Li, J. (2019). Hyperspectral image classification using random occlusion data augmentation. IEEE Geoscience and Remote Sensing Letters , 16(11):1751--1755
2019
-
[23]
He, K., Chen, X., Xie, S., Li, Y., Doll \'a r, P., and Girshick, R. (2022). Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 16000--16009
2022
-
[24]
Heinrich, R., Scholz, C., Vogt, S., and Lehna, M. (2023). Targeted adversarial attacks on wind power forecasts. Machine Learning
2023
-
[25]
Hong, D., Gao, L., Yokoya, N., Yao, J., Chanussot, J., Du, Q., and Zhang, B. (2021). More diverse means better: Multimodal deep learning meets remote-sensing imagery classification. IEEE Transactions on Geoscience and Remote Sensing , 59(5):4340--4354
2021
-
[26]
Inglada, J., Vincent, A., Arias, M., and Marais-Sicre, C. (2016). Improved early crop type identification by joint use of high temporal resolution SAR and optical image time series. Remote Sensing , 8(5):362
2016
-
[27]
Kussul, N., Lavreniuk, M., Skakun, S., and Shelestov, A. (2017). Deep learning classification of land cover and crop types using remote sensing data. IEEE Geoscience and Remote Sensing Letters , 14(5):778--782
2017
-
[28]
Lee, J., Lee, Y., Kim, J., Kosiorek, A., Choi, S., and Teh, Y. W. (2019). Set Transformer : A framework for attention-based permutation-invariant neural networks. In International Conference on Machine Learning (ICML) , pages 3744--3753. PMLR
2019
-
[29]
Ma, M., Ren, J., Zhao, L., Testuggine, D., and Peng, X. (2022). Are multimodal transformers robust to missing modality? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 18177--18186
2022
-
[30]
Mena, F., Arenas, D., Charfuelan, M., Nuske, M., and Dengel, A. (2024a). Impact assessment of missing data in model predictions for earth observation applications. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS) , pages 967--971
2024
-
[31]
Mena, F., Arenas, D., and Dengel, A. (2024b). Increasing the robustness of model predictions to missing sensors in earth observation. arXiv preprint arXiv:2407.15512
2024 arXiv
-
[32]
Mena, F., Arenas, D., Nuske, M., and Dengel, A. (2023). A comparative assessment of multi-view fusion learning for crop classification. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium ( IGARSS ) , pages 5631--5634. IEEE
2023
-
[33]
Mena, F., Arenas, D., Nuske, M., and Dengel, A. (2024c). Common practices and taxonomy in deep multi-view fusion for remote sensing applications. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , pages 4797 -- 4818
2024
-
[34]
Mena, F., Pathak, D., Najjar, H., Sanchez, C., Helber, P., Bischke, B., Habelitz, P., Miranda, M., Siddamsetty, J., Nuske, M., et al. (2024d). Adaptive fusion of multi-view remote sensing data for optimal sub-field crop yield prediction. arXiv preprint arXiv:2401.11844
2024 arXiv
-
[35]
Najjar, H., Nuske, M., and Dengel, A. (2024). Data-centric machine learning for earth observation: Necessary and sufficient features. arXiv preprint arXiv:2408.11384
2024 arXiv
-
[36]
Ofori-Ampofo, S., Pelletier, C., and Lang, S. (2021). Crop type mapping from optical and radar time series using attention-based deep learning. Remote Sensing , 13(22)
2021
-
[37]
P., Flefil, J
Rao, K., Williams, A. P., Flefil, J. F., and Konings, A. G. (2020). SAR -enhanced mapping of live fuel moisture content. Remote Sensing of Environment , 245:111797
2020
-
[38]
Rolf, E., Klemmer, K., Robinson, C., and Kerner, H. (2024). Mission critical-- Satellite data is a distinct modality in machine learning. arXiv preprint arXiv:2402.01444
2024 arXiv
-
[39]
Sainte Fare Garnot, V., Landrieu, L., and Chehata, N. (2022). Multi-modal temporal attention models for crop mapping from satellite time series. ISPRS Journal of Photogrammetry and Remote Sensing , 187:294--305
2022
-
[40]
Shen, H., Li, X., Cheng, Q., Zeng, C., Yang, G., Li, H., and Zhang, L. (2015). Missing information reconstruction of remote sensing data: A technical review. IEEE Geoscience and Remote Sensing Magazine , 3(3):61--85
2015
-
[41]
Srivastava, N., Hinton, G., Krizhevsky, A., Sutskever, I., and Salakhutdinov, R. (2014). Dropout: a simple way to prevent neural networks from overfitting. Journal of Machine Learning Research , 15(1):1929--1958
2014
-
[42]
L., and Kerner, H
Tseng, G., Zvonkov, I., Nakalembe, C. L., and Kerner, H. (2021). CropHarvest : A global dataset for crop-type classification. Proceedings of NIPS Datasets and Benchmarks Track
2021
-
[43]
Tseng, G., Zvonkov, I., Purohit, M., Rolnick, D., and Kerner, H. (2023). Lightweight, pre-trained transformers for remote sensing timeseries. arXiv preprint arXiv:2304.14065
2023 arXiv
-
[44]
N., Kaiser, ., and Polosukhin, I
Vaswani, A., Shazeer, N., Parmar, N., Uszkoreit, J., Jones, L., Gomez, A. N., Kaiser, ., and Polosukhin, I. (2017). Attention is all you need. Advances in Neural Information Processing Systems (NIPS) , 30
2017
-
[45]
Wang, C., Liu, X., Pei, J., Huang, Y., Zhang, Y., and Yang, J. (2021). Multiview attention CNN - LSTM network for SAR automatic target recognition. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing , 14:12504--12513
2021
-
[46]
M., and Zhu, X
Wang, Y., Albrecht, C. M., and Zhu, X. X. (2022). Self-supervised vision transformers for joint SAR -optical representation learning. In Proceedings of the IEEE International Geoscience and Remote Sensing Symposium (IGARSS) , pages 139--142
2022
-
[47]
Yan, X., Hu, S., Mao, Y., Ye, Y., and Yu, H. (2021). Deep multi-view learning methods: A review. Neurocomputing , 448:106--129
2021
-
[48]
Yuan, Y., Lin, L., Liu, Q., Hang, R., and Zhou, Z.-G. (2022). SITS-Former : A pre-trained spatio-spectral-temporal representation model for Sentinel-2 time series classification. International Journal of Applied Earth Observation and Geoinformation , 106:102651
2022
-
[49]
Zhang, P., Du, P., Lin, C., Wang, X., Li, E., Xue, Z., and Bai, X. (2020). A hybrid attention-aware fusion network ( HAFNet ) for building extraction from high-resolution imagery and LiDAR data. Remote Sensing , 12(22)
2020
-
[50]
Zheng, Z., Ma, A., Zhang, L., and Zhong, Y. (2021). Deep multisensor learning for missing-modality all-weather mapping. ISPRS Journal of Photogrammetry and Remote Sensing , 174:254--264
2021
Reviewed August 10, 2026 · model on record in the stance chip above.
Discussion (0). Continue with ORCID to comment.