{"id":"bd0c94d9-6277-48b3-8841-e770fc2184b4","arxiv_id":"2412.06451","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"Three new Earth observation benchmark datasets with reference uncertainty values for regression, segmentation, and classification are introduced to evaluate machine learning uncertainty quantification methods.","lead":"This paper introduces three benchmark datasets for testing how well machine learning models can estimate their own uncertainty on Earth observation tasks: biomass regression, building footprint segmentation, and urban scene classification. Each dataset includes a reference measure of the irreducible data uncertainty, allowing researchers to compare different uncertainty quantification methods on a common ground.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"SegmentationUQ's reference aleatoric entropy is computed with a learned U-Net rather than the known rendering model, and the paper's own limitations section concedes this conflates aleatoric and epistemic uncertainty, so the benchmark's central 'ground-truth uncertainty' claim is unsubstantiated.","rationale":"The reader's weakest assumption correctly identified the SegmentationUQ reference as the load-bearing point: the paper assumes the trained U-Net's epistemic uncertainty is negligible, while Section V explicitly concedes that an inaccurate model conflates aleatoric and epistemic uncertainties. My stress-test confirms this is the single most important threat to the central claim. The RegressionUQ reference is defensible because the allometric equation is declared the ground-truth model, so the reference uncertainty is well-defined relative to that model even if the model itself is imperfect. The ClassificationUQ label votes are a direct empirical uncertainty signal. SegmentationUQ, however, claims to provide a reference aleatoric entropy for a task where no perfect model exists, and the reference is generated by the authors' own network. This makes the benchmark circular in the worst case: a UQ method that happens to mimic the reference U-Net's errors will score well, and the advertised 'ground truth' is not ground truth but one model's predictive uncertainty. The paper's own limitation statement in Section V is an in-scope admission that should be weighed heavily. That said, the concern is testable and potentially fixable, so conditional acceptance remains appropriate rather than rejection. The reader's verdict already conditions on validation of the reference uncertainty, so my read does not move the verdict; it sharpens the required validation.","tokens_in":18388,"tokens_out":4110,"duration_ms":47589,"concrete_test":"Compute the SegmentationUQ reference entropy three ways: (1) with the published U-Net; (2) with one or more independently trained U-Nets using different architectures and random seeds on the same clean renders; and (3) where feasible, directly from the known generative model by Monte Carlo sampling over the noise distribution and the LoD2 ground-truth masks. Compare the per-patch aleatoric entropy values across methods for the same noise types and levels in Table VI. If the references differ by more than, say, 10% relative RMSE or if the ranking of noise levels changes, the reference is model-dependent and does not isolate aleatoric uncertainty; if they agree closely, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim—that the datasets provide reference aleatoric uncertainty for benchmarking—is only as strong as the SegmentationUQ reference, because RegressionUQ defines its reference against an explicit allometric equation and ClassificationUQ uses empirical expert votes. In Section III-B3, the reference aleatoric entropy is produced by feeding 50 noisy renders through a single trained U-Net, estimating the mean and variance of each logit, and computing entropy from the resulting Gaussian. The authors justify this by writing 'We believe the epistemic uncertainty of such model is minimized'—a belief, not a demonstrated property. Section V then states the limitation: 'An inaccurate model causes the conflation of aleatoric and epistemic uncertainties, which was observed in both the experiments.' This is not a peripheral caveat; it directly undermines the dataset's advertised purpose. Any UQ method benchmarked against this reference is effectively being scored by how well it reproduces the specific U-Net's sensitivity to noise, not by how well it captures irreducible data uncertainty. Because the rendering pipeline provides true LoD2 masks and known noise distributions, a physical reference is available in principle; substituting a learned surrogate is the weakest point in the paper.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes three Earth observation (EO) benchmark datasets for uncertainty quantification (UQ): RegressionUQ, a simulated single-tree biomass regression dataset in which the ground-truth label and reference aleatoric uncertainty are derived from the Chave allometric equation; SegmentationUQ, a rendered building segmentation dataset with simulated Gaussian, Poisson, and viewpoint noise, where a reference aleatoric entropy is computed by propagating noisy images through a trained U-Net; and ClassificationUQ, an extension of So2Sat LCZ42 with ten expert votes per image patch, converted into distributional labels. The authors describe data generation, reference uncertainty calculation, and demonstration experiments: two UQ methods on RegressionUQ, two UQ methods on SegmentationUQ, and a distributional-label training approach on ClassificationUQ. The central claim is that these are the first EO-specific benchmarks providing reference aleatoric uncertainty.","tokens_in":18526,"tokens_out":4188,"duration_ms":45418,"significance":"If the reference uncertainties are valid, the three datasets fill a genuine gap: existing EO benchmarks provide labels but no ground-truth uncertainty, making it impossible to score UQ methods objectively. The regression reference is anchored in an explicit physical equation and propagated via Monte Carlo simulation, which is a reproducible, parameter-free approach; the classification reference is anchored in external human votes, giving an independent measure of label ambiguity. The segmentation reference is the weakest link because it substitutes a learned surrogate for a physical ground-truth model. The paper also ships datasets and code, which supports community adoption. However, the validity of the segmentation benchmark is load-bearing for the paper's headline claim, and the current evidence for it is an unexamined belief rather than a verified property.","major_comments":[{"comment":"The SegmentationUQ reference aleatoric entropy rests entirely on a single trained U-Net used as a ground-truth model. In Section III-B3 the authors write 'We believe the epistemic uncertainty of such model is minimized', and in Section V they concede that 'an inaccurate model causes the conflation of aleatoric and epistemic uncertainties, which was observed in both the experiments'. This is a direct admission that the reference may not isolate aleatoric uncertainty. Because the rendering pipeline knows the true noise models and the true LoD2 masks, a physically grounded reference is in principle available; the paper should either construct such a reference or validate that the U-Net-based reference is stable under changes in network architecture, random seed, and training set size. Without this, a UQ method benchmarked on SegmentationUQ is being scored on how well it reproduces the specific U-Net's noise sensitivity, not on how well it captures irreducible data uncertainty.","section":"III-B3 and V"},{"comment":"The definition of the RegressionUQ reference is ambiguous. The text says that because a neural network is trained on pooled noisy data, 'a realistic reference aleatoric uncertainty from a neural network shall also be calculated from the pooled distribution', but the implementation uses '800 neighboring points' per input and a smoothing curve. It is unclear whether the reference is the local conditional variance Var(f(ρ,D,H) | D0,H0) for a clean input, or the variance of the mixture of outputs over the noisy training distribution, or something else. The estimand matters because each test sample receives a single reference value that is compared against the model's prediction. Please state the exact quantity being estimated, how the neighboring points are selected, and how the smoothing curve is fitted, and provide a Monte Carlo error estimate for the reference.","section":"II-B3"},{"comment":"The abstract presents ClassificationUQ as one of three benchmarks for UQ, but the demonstration in Section IV-C is a training-time comparison of distributional-label learning against one-hot learning (accuracy, cross-entropy, ECE); no UQ method is benchmarked against a reference uncertainty. While the vote counts do permit users to compute a reference label distribution, the paper does not provide a precomputed reference uncertainty (e.g., entropy per patch) nor an example of benchmarking a model's predictive uncertainty against it. To substantiate the 'benchmark' claim for the classification dataset, either provide such a reference and a benchmarking demonstration, or explicitly scope ClassificationUQ as a label-uncertainty resource rather than a UQ benchmark.","section":"IV-C and Abstract"}],"minor_comments":[{"comment":"Figure 2, cited in the RegressionUQ section, is labeled as 'the training/test split in the SegmentationUQ dataset', but it plots tree diameter and height samples from the regression dataset; the caption or the figure label should be corrected.","section":"II-B2"},{"comment":"The sentence '20, 0002 noisy input points were simulated' appears to be a typo; it should read '20,000 noisy input points'.","section":"II-B3"},{"comment":"Section I-A contains several typos: 'segmntation', 'ans test', and 'infared' (the last in Section I-B2).","section":"I-A"},{"comment":"In Section II-A, 'biomass datasets generated from allometric equations are still scars' should be 'scarce'.","section":"II-A"},{"comment":"In Table V, step 4, the number N of Monte Carlo samples drawn from the Gaussian logit distribution is never specified; please state the value and justify its sufficiency for low-variance entropy estimates.","section":"III-B3"},{"comment":"Section IV-B says 'The dataset were created'; this should be 'The dataset was created'.","section":"IV-B"}],"recommendation":"major_revision","confidential_remarks":"The authors are unusually transparent about the SegmentationUQ limitation in Section V, which is commendable. The central risk is that the segmentation reference is a learned surrogate whose epistemic uncertainty is assumed away; this is a fixable issue if the authors add validation against a physics-based reference or demonstrate invariance across model choices. The RegressionUQ estimand ambiguity is also worth clarifying. The paper fits the scope of IEEE GRSM as a dataset/resource paper, but the acceptance decision should hinge on whether the segmentation reference can be made trustworthy."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Three benchmark datasets for UQ in Earth observation; the regression one is credible, the segmentation one has a load-bearing weakness, and the classification one is mostly a repackaging of So2Sat votes.\n\nWhat is actually new: the paper documents a real gap—no existing EO dataset is built for benchmarking UQ methods—and it offers concrete artifacts. RegressionUQ is the strongest part. The allometric equation is an external anchor, the MC propagation of input noise is well specified, and the demo experiments are transparent, including the negative R2 at low noise. That alone is a useful resource. SegmentationUQ, the rendered Berlin dataset, is also a serious effort: 6 million image patches from LoD2 models with controlled Gaussian, Poisson, and viewpoint noise. The honest reporting of baseline behaviour is a plus.\n\nThe soft spot is the segmentation reference. The paper states in Section III-B3 that a single trained U-Net serves as the ground-truth model and \"we believe the epistemic uncertainty of such model is minimized.\" That is a belief, not a result, and Section V concedes the consequence: an inaccurate model conflates aleatoric and epistemic uncertainty, which the authors say they observed in both experiments. So the benchmark's advertised reference entropy is really this particular U-Net's noise sensitivity, not irreducible data uncertainty. The rendering pipeline itself provides the true noise distributions and LoD2 masks, so a physical reference is available in principle; substituting a learned surrogate is the weakest decision in the paper. That said, the dataset can still be useful as a consistent reference for ranking UQ methods relative to each other, as long as the claim is softened accordingly.\n\nTwo smaller issues. The evaluation sections are thin—R2/RMSE over five noise levels, no error bars, and the entropy values are averages over 50 renders per patch. And the release artifacts are hard to verify: the GitLab URL is provided but there is no commit hash, DOI, or license details in the text, which matters for a dataset paper. ClassificationUQ reuses the So2Sat LCZ42 expert votes and the results from reference [22], so its novelty is packaging rather than new data.\n\nWho should read it: anyone designing or comparing UQ methods for EO tasks, and the benchmark community evaluating what makes a valid \"ground truth\" for uncertainty. It deserves a serious referee, but with a major-revision request: validate the segmentation reference against an independent standard (or at least characterise the model's epistemic error), add reproducibility metadata, and temper the claims about reference aleatoric uncertainty. I would engage with the paper, but I would cite it with caution until the reference issue is addressed.","headline":"A useful benchmark suite with one solid regression dataset and one over-claimed segmentation reference; deserves peer review but needs to fix the ground-truth issue.","tokens_in":19159,"tokens_out":3229,"would_cite":false,"duration_ms":33539,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces three Earth observation benchmark datasets — for regression, segmentation, and classification — that provide reference aleatoric uncertainty values, so machine-learning uncertainty estimates can be measured against a…","keywords":["uncertainty quantification","earth observation","benchmark datasets","aleatoric uncertainty","machine learning","regression","image segmentation","scene classification"],"falsifier":"Train an ensemble of, say, ten U-Nets with different initializations on the same clean segmentation renderings and feed the same 50 noisy samples through each; if the reference aleatoric entropies differ materially across ensemble members, the single-network assumption fails and the reference conflates epistemic with aleatoric uncertainty.","tokens_in":18044,"feed_emoji":"🛰️","tokens_out":12638,"duration_ms":108114,"temperature":0.7,"pith_summary":"The paper's claim is that the lack of ground-truth uncertainty labels has made it impossible to judge how reliable machine-learning uncertainty estimates are in Earth observation, and that this gap can be closed by building benchmarks where the uncertainty of the reference is known by construction. Three datasets are introduced, one per common problem type: biomass regression with controlled input noise, building segmentation rendered from 3D models with simulated image and viewpoint noise, and local climate zone classification carrying ten expert votes per patch. Each dataset comes with a reference aleatoric uncertainty — the variance of the biomass prediction, the Shannon entropy of the segmentation softmax distribution, and the distributional label — so that any UQ method can be scored against it. If the construction holds, these would be the first EO-specific datasets designed to benchmark uncertainty estimates rather than only provide labels.","feed_headline":"Three datasets give Earth-observation AI a yardstick for uncertainty","feed_subtitle":"Each set carries a reference aleatoric uncertainty, so UQ methods can be scored against a ground truth.","key_machinery":"The machinery that carries the argument is Monte Carlo propagation of controlled input noise through a trusted reference model. In the regression dataset, the reference model is the allometric equation $B = 0.0673(\\rho D^2 H)^{0.976}$ from [36], with noise-free $D$ and $H$ drawn from Gamma distributions fitted to field data; noisy input points are simulated in bulk per noise level and the variance of the output around the true biomass is smoothed into a parametric reference curve. In the segmentation dataset, the reference model is a U-Net trained on noise-free renderings whose epistemic uncertainty is assumed negligible; for each noise configuration, 50 noisy images are passed through the network, each logit is modeled as a Gaussian $z_j \\sim \\mathcal{N}(\\mu_j, \\sigma_j^2)$, Monte Carlo samples of the logits are pushed through softmax, and the Shannon entropy of the resulting probability distribution is averaged over pixels and classes to give a single reference aleatoric entropy value per patch. In the classification dataset, the reference uncertainty is the distributional label $\\mathbf{y}_{\\text{distr}} = \\mathbf{Y}/M$ formed from $M$ expert votes, and training with a KL-divergence loss against this distribution is the demonstrated use.","core_discovery":"The central claim is that reference aleatoric uncertainty can be constructed for three common Earth observation problem types by controlling the noise source. For regression, a published allometric equation for tropical-tree biomass is declared the ground-truth physical model, Gaussian noise with a known signal-to-noise ratio is added to tree height and diameter, and the variance of the equation output around the true biomass is taken as the reference uncertainty. For segmentation, noise-free baseline images are rendered from high-quality 3D building models, then Gaussian noise, Poisson noise, and camera-viewpoint variation are simulated; a baseline U-Net trained on the clean data is fed the noisy renderings, each logit is modeled as a Gaussian estimated from the 50 noisy samples, and the Shannon entropy of the resulting softmax distribution is the reference aleatoric entropy. For classification, ten remote-sensing-expert votes per image patch are turned into a distributional label that captures irreducible human label uncertainty. The authors demonstrate the benchmarks by comparing two regression UQ methods and two segmentation UQ methods against the references, and by showing that training an LCZ classifier on distributional labels roughly halves its expected calibration error.","pith_inferences":["Outside the paper's scope, the segmentation reference could be stress-tested by using an ensemble of clean-trained U-Nets; the spread of reference entropies across ensemble members would directly bound how much epistemic uncertainty leaks into the 'aleatoric' benchmark.","The same Monte-Carlo-through-a-forward-model recipe transfers to other EO tasks with a known physical model, such as radiative-transfer-based retrievals, where the hard part is identifying a trustworthy reference model rather than generating noise.","The classification dataset's multiple votes could be used to benchmark label-noise-robust training methods beyond the demonstrated KL-divergence objective, giving a common testbed for a question the paper only touches on."],"forward_implications":["Any UQ method can now be ranked by R² and RMSE against a per-patch reference uncertainty, instead of only by internal consistency or calibration.","The benchmarks show that aleatoric and epistemic uncertainties are not cleanly separable in practice: enlarging the regression training set from ×1 to ×16 improves the aleatoric-uncertainty score, implying residual epistemic contamination.","SegmentationUQ's 6 million patches enable isolating how different noise types with the same IoU affect measured uncertainty, not just prediction quality.","Using distributional labels from expert votes roughly halves expected calibration error on the LCZ classification task, indicating that human label uncertainty is a learnable signal rather than pure noise."],"supporting_citations":[{"why":"Documents that none of over 500 EO datasets is designed for UQ, establishing the gap this paper fills.","marker":"[29]"},{"why":"Supplies the allometric biomass equation used as the ground-truth physical model for RegressionUQ.","marker":"[36]"},{"why":"Source of the high-quality 3D building models used to render the SegmentationUQ imagery.","marker":"[40]"},{"why":"Provides the method of modeling each logit as a Gaussian, which the segmentation reference entropy calculation adapts.","marker":"[43]"},{"why":"One of the regression UQ methods benchmarked against the reference, and the basis of the Gaussian-logit aleatoric-uncertainty loss.","marker":"[33]"},{"why":"The second regression UQ method benchmarked, using Assumed Density Filtering layers with observation variance as input.","marker":"[37]"},{"why":"Supplies the test-time augmentation UQ method compared against BNN on the segmentation dataset.","marker":"[44]"},{"why":"Supplies the local climate zone image patches and the ten expert votes that form the ClassificationUQ dataset.","marker":"[50]"},{"why":"Provides the distributional-label training method demonstrated on the classification dataset.","marker":"[22]"}],"fun_headline_variants":["Three EO datasets give AI uncertainty a ground truth to measure against","How certain is AI's uncertainty? New EO benchmarks answer that","Ground-truth uncertainty datasets let you score EO AI's UQ methods","EO AI uncertainty gets a benchmark with known noise and human votes","Test your Earth observation AI's uncertainty with three new datasets"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the reference model is accurate enough to isolate aleatoric uncertainty: the allometric equation is treated as the true physical relationship in regression, and a single well-trained U-Net is assumed to have negligible epistemic uncertainty in segmentation.","fun_headline_variants_meta":{"raw":{"variants":["Three EO datasets give AI uncertainty a ground truth to measure against","How certain is AI's uncertainty? New EO benchmarks answer that","Ground-truth uncertainty datasets let you score EO AI's UQ methods","EO AI uncertainty gets a benchmark with known noise and human votes","Test your Earth observation AI's uncertainty with three new datasets"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000588,"raw_usage":{"total_tokens":2805,"prompt_tokens":1034,"completion_tokens":1771,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":650,"completion_tokens_details":{"reasoning_tokens":1684}},"tokens_in":650,"tokens_out":1771,"duration_ms":13410,"temperature":1.0,"reasoning_tokens":1684,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T19:37:50.056071+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Train an ensemble of, say, ten U-Nets with different initializations on the same clean segmentation renderings and feed the same 50 noisy samples through each; if the reference aleatoric entropies differ materially across ensemble members, the single-network assumption fails and the reference conflates epistemic with aleatoric uncertainty.","supporting_citations":[{"cited_title":"Improved allometric models to estimate the aboveground biomass of tropical trees,","cited_arxiv_id":null,"evidence_quote":"Supplies the allometric biomass equation used as the ground-truth physical model for RegressionUQ."},{"cited_title":"[Online]","cited_arxiv_id":null,"evidence_quote":"Source of the high-quality 3D building models used to render the SegmentationUQ imagery."},{"cited_title":"A deeper look into aleatoric and epistemic uncertainty disentanglement,","cited_arxiv_id":null,"evidence_quote":"Provides the method of modeling each logit as a Gaussian, which the segmentation reference entropy calculation adapts."},{"cited_title":"A general framework for uncertainty estimation in deep learning,","cited_arxiv_id":null,"evidence_quote":"The second regression UQ method benchmarked, using Assumed Density Filtering layers with observation variance as input."},{"cited_title":"So2sat lcz42: a benchmark data set for the classification of global local climate zones [software and data sets],","cited_arxiv_id":null,"evidence_quote":"Supplies the local climate zone image patches and the ten expert votes that form the ClassificationUQ dataset."},{"cited_title":"Going beyond one-hot encod- ing in classification: Can human uncertainty improve model performance in earth observation?","cited_arxiv_id":null,"evidence_quote":"Provides the distributional-label training method demonstrated on the classification dataset."}],"review_version":1}