{"id":"eb595302-7d3c-43bf-9ec5-4f1e0026eefd","arxiv_id":"2412.05203","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"A new benchmark dataset of 888 km2 of Cambodian LiDAR terrain with 31,141 annotated archaeological features, released with baseline semantic segmentation results.","lead":"Archaeoscape releases 888 square kilometers of laser-scanned Cambodian terrain with about 31,000 expert-labeled archaeological features, plus baseline deep learning results. It is the largest open-access dataset of its kind and gives computer vision researchers a new benchmark in landscape archaeology.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The benchmark's validity rests on the completeness of expert annotations, which the paper itself concedes is unverified; any unlabeled features in 'Background' would bias every reported IoU score.","rationale":"The reader's weakest assumption, annotation completeness, is also the most load-bearing concern I found. The central claim is that Archaeoscape provides a reusable benchmark, and every benchmark number depends on the assumption that 'Background' genuinely contains no archaeological features. The paper itself concedes this may be false, and no quantitative evidence is provided to bound the amount of missing annotation. I considered whether the gated, non-redistributable license undermines the 'open access' claim, but the paper explicitly defines open access in footnote 1 as accessible data, annotations, and splits, and the license terms are transparently disclosed; that is a definitional debate rather than a correctness failure. I also noted the inconsistent feature counts (31,141 vs 31,411 vs 31,144 in the abstract, introduction, Table 1, conclusion, and datasheet), but this does not threaten the qualitative claim of being the largest dataset. The completeness concern is concrete, acknowledged in Section 4.4, and testable, so the conditional verdict stands unchanged.","tokens_in":18451,"tokens_out":4228,"duration_ms":48635,"concrete_test":"Run an independent completeness audit on the test split: have a second team of expert archaeologists re-annotate a stratified random sample of test parcels (stratified by class and terrain type) from the same nDTM/RGB without seeing the original labels. Compute the per-class recall of the original labels relative to the union of both annotations, and the areal extent of newly found features falling in original 'Background'. If recall falls materially below 95% or the uncovered area is a non-negligible fraction of the positive classes, the reported IoU scores are biased and the benchmark should be re-evaluated with corrected labels. A faster complement is to take the top false-positive regions of the best model on the test set and ask experts whether those are real unannotated features.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The load-bearing assumption is that the expert polygons are a complete ground truth for the 888 km2 benchmark. Section 4.4 explicitly concedes 'we cannot rule out that background terrain may contain some yet uncovered features,' and the datasheet (Q7) claims only that 'the vast majority' of features have been identified, with no quantitative recall estimate. Because all metrics treat 'Background' as the negative class, any real feature that experts missed is scored as a false negative whenever a model detects it. This is not merely random noise: missed features are likely to be the subtlest ones and may be spatially correlated with terrain and class, so different architectures (e.g., U-Net vs. ViT) can be penalized unequally. The headline mIoU numbers and the model ranking in Table 2 are therefore only as trustworthy as the unmeasured label recall. The paper's own limitation statement makes this an acknowledged gap, and no inter-annotator agreement, independent validation subset, or recall estimate is reported to close it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces Archaeoscape, a LiDAR-derived archaeological dataset covering 888 km^2 in Cambodia with expert-annotated features (temples, mounds, hydrology, void/background), distributed as 0.5 m orthophotos and normalized digital terrain models. The authors benchmark CNN, ViT, and hierarchical ViT segmentation models, report mIoU values around 50, and analyze the importance of the elevation channel and input context size. They release data, annotations, and models under a credentialized open-access license and describe measures to prevent misuse. The central claims are that Archaeoscape is the largest open-access ALS archaeology dataset and the first such resource with open data, annotations, and models.","tokens_in":18621,"tokens_out":4816,"duration_ms":48281,"significance":"If the dataset and benchmark are sound, this is a substantial community resource: the area covered is several times larger than existing released ALS archaeology datasets, the annotation work is long-term and field-verified, and the split, preprocessing, and evaluation protocols are described in unusual detail. The benchmark is broad, covering modern architectures with various pretraining strategies, and the paper provides useful reproducibility information such as compute budgets, dataloader details, and hyperparameters. The main caveats are the unmeasured completeness of the expert labels, which affects all reported IoU scores, and the absence of error bars for the benchmark comparisons; both are fixable and do not negate the value of the released resource.","major_comments":[{"comment":"The annotation-completeness assumption is load-bearing. The paper states in Section 4.4 that 'we cannot rule out that background terrain may contain some yet uncovered features,' and the datasheet (Q7) claims only that the 'vast majority' of features have been identified, without quantifying recall. Since Background is the negative class in all IoU computations, any real feature missed by the annotators is scored as a false negative whenever a model detects it, and missed features are likely to be the subtlest ones, potentially correlated with terrain and class. The reported mIoU values and model rankings in Table 2 are therefore conditional on this unmeasured label recall. Please add an independent assessment of annotation completeness, for example a sample re-reviewed by independent experts or a per-class recall estimate, and discuss how the headline numbers would change under plausible miss rates.","section":"Section 4.4 / Table 2"},{"comment":"The instance count is inconsistent across the manuscript: the Abstract says 31,141, Section 1 and Table 1 say 31,411, and the Conclusion says 31,144. Because the headline claim of being the largest open-access ALS archaeology dataset depends on this number, the discrepancy must be resolved and a single canonical count used consistently throughout.","section":"Abstract, Section 1, Conclusion"},{"comment":"The paper reports no error bars, explicitly acknowledged in the checklist as being omitted due to computational constraints. Several benchmark conclusions rest on small margins, for example U-Net (50.5) versus HybViT (50.4) at 224 resolution, and PVTv2-b1 (52.1) versus SWIN-S (51.9). Without repeated-seed variance or another uncertainty measure, the relative ranking in Table 2 is not statistically supported. Please report standard deviations or confidence intervals for the main configurations, or otherwise justify that the observed differences exceed run-to-run variability.","section":"Checklist A.1(2c) / Table 2"}],"minor_comments":[{"comment":"The sentence stating that Archaeoscape 'covers a 2 × larger surface area and contains 3× more instances than the next-largest closed archaeology LiDAR dataset' does not match Table 1 for either plausible comparator: relative to AHN-2 the area is about 2.0× but instances are about 8.2×, while relative to Chactún the area is about 3.9× but instances are about 2.9×. Please specify the intended comparator and correct the multipliers.","section":"Section 2.1 / Table 1"},{"comment":"In the RGB-only PVTv2-b1 row, the OA column shows 33.9, which equals the mIoU value and appears to be a typo; the background IoU is 82.0, so the overall accuracy should be much higher.","section":"Table 4"},{"comment":"There are several typographical errors in the table and surrounding text, including 'ImagineNet1K' for ImageNet1K, 'Webold' for 'We bold', 'present report performance' for 'we present the performance', and 'a least' for 'at least' in Section 3.1.","section":"Table 2 and Section 4.2"},{"comment":"The hybrid model 'HybViT' is described but no citation or architectural reference is provided; please cite the source or specify the exact configuration used in the experiments.","section":"Section 2.2"}],"recommendation":"major_revision","confidential_remarks":"The 'open access' terminology may draw external scrutiny because the license forbids redistribution and re-georeferencing and access requires credentials; the paper's footnote definition makes the claim internally consistent, but the editors may want the distinction to be made explicit in the final version. The annotation-completeness issue is the most substantive technical concern; an independent recall estimate would substantially strengthen the benchmark. The instance-count inconsistency and missing error bars should be fixed before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nArchaeoscape is a genuine step forward for ALS archaeology. The dataset is large, well-documented, and the first open-access resource of its kind for Southeast Asia. That alone merits attention.\n\nWhat's actually new: 888 km2 of Cambodian LiDAR with 31k expert-verified polygons, plus orthophotos and normalized DTM at 0.5 m resolution. The acquisition and preprocessing are described in detail, the spatial splits with buffers are sensible, and stripping georeferencing and gating access behind a license is a defensible looting-mitigation choice. The benchmark covers a broad set of modern segmentation models, and the ablation showing elevation matters more than RGB is a useful, reproducible result. The paper is transparent about its limitations.\n\nThe soft spots, in order. (1) The completeness of expert polygons is the load-bearing assumption for any IoU number. The paper explicitly concedes that background may contain unidentified features, and there is no inter-annotator agreement or recall estimate. Unlabeled features would act as false negatives for any model that detects them, and if they are spatially correlated with subtle terrain, the model ranking in Table 2 could be biased. This is a real limitation, but the authors flag it; it undercuts the interpretation of absolute scores, not the dataset's primary value. (2) The reported feature count is inconsistent: 31,141 in the abstract, 31,411 in the intro, 31,144 in the conclusion. Sloppy and needs fixing. (3) No error bars on the benchmark. The checklist acknowledges this; it's common with expensive training, but a few multi-seed runs on a subset would help. (4) The \"open-access\" claim is partially undercut by the credentialized license: reproducible in principle, but not immediately so for anyone who hasn't signed the agreement.\n\nThe stress-test concern is valid but proportionate. The completeness issue is real, but it is an acknowledged limitation, not a hidden flaw. The central claim—largest open-access ALS archaeology dataset with a credible benchmark—holds up.\n\nThis paper deserves a serious referee. I'd send it to review, requesting that the counts be reconciled, a brief discussion of label recall be added, and ideally error bars or a timeline for them. The audience is remote sensing archaeologists and CV researchers interested in domain-specific benchmarks. I'd cite it if I worked in this niche; otherwise it's a reference worth knowing.","headline":"Large, well-documented ALS archaeology dataset with a credible benchmark; the label-completeness caveat is real but acknowledged and does not sink the main contribution.","tokens_in":19146,"tokens_out":2794,"would_cite":true,"duration_ms":30104,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper introduces Archaeoscape, a 888 km² open-access airborne laser scanning dataset of Angkorian Cambodia with 31,411 annotated archaeological features, and shows through benchmarks that deep learning models still struggle to detect…","keywords":["airborne laser scanning","LiDAR archaeology","semantic segmentation","Angkorian Cambodia","open-access dataset","benchmark","digital terrain model"],"falsifier":"A second independent team of Khmer archaeologists re-draws all polygons on a 10 km² sample of the test parcels without seeing the released labels; if the two labelings disagree on many mound and hydrology boundaries, then the 1–2 point accuracy gaps between the top models are within annotation noise and the benchmark ranking is not settled.","tokens_in":18274,"feed_emoji":"🏛️","tokens_out":10365,"duration_ms":99522,"temperature":0.7,"pith_summary":"This paper introduces Archaeoscape, an open-access airborne laser scanning (ALS) dataset covering 888 km² of Cambodia with 31,411 expert-drawn, field-verified archaeological features from the Angkorian period. The authors aim to establish that this is the largest open ALS archaeology dataset published so far and the first to release data, annotations, and models together, so that deep-learning methods can be trained and compared on a shared benchmark. Their benchmark of modern segmentation models shows the problem is not solved: the best model reaches roughly 53% mean overlap accuracy, elevation is the decisive input, and models pretrained on natural images transfer poorly. A fair reader would take the paper's contribution to be a hard, reusable benchmark that challenges computer vision to find subtle human-made structures hidden under dense jungle.","feed_headline":"Open LiDAR archaeology dataset maps 31,411 Angkor features","feed_subtitle":"Deep-learning models still miss many subtle jungle-covered structures, making this benchmark a hard test","key_machinery":"The load-bearing object is the dataset itself, assembled as 23 non-overlapping parcels of 0.5 m resolution GeoTIFFs covering 888 km², each pixel carrying RGB orthophotography, a normalized Digital Terrain Model (nDTM) elevation, and a semantic label. The annotation scheme maps 12 expert feature types into five classes (temple, mound, hydrology, background, and a void class excluded from evaluation), and the benchmark treats the problem as pixel-wise semantic segmentation with class-wise intersection-over-union as the metric. The evaluation protocol adapts RGB-pretrained networks to the extra elevation channel by randomly initializing only the new first-layer weights, and it uses spatially distinct parcels with a 100 m buffer to prevent contamination, with the test set mixing remote parcels and parcels adjacent to training areas.","core_discovery":"The paper's central claim is that Archaeoscape fills the missing resource for deep-learning archaeology: at 888 km² and 31,411 annotated instances, it is the largest open-access ALS archaeology dataset published to date and the first to distribute the imagery, the polygon annotations, and trained models openly. On its own benchmark, the authors find that modern semantic segmentation networks do not solve the problem. The best score, 52.8% mean intersection-over-union, comes from a U-Net at 512×512 input, while the best 224-level model is the hierarchical transformer PVTv2 at 52.1%; plain ViTs, CLIP, DINOv2, ScaleMAE, and DOFA all fare worse. The paper attributes the difficulty to the subtlety of earthen features, the need for kilometer-scale context, and the fact that the normalized terrain elevation, not RGB, carries most of the information. These findings lead the authors to argue that bespoke models and pretraining strategies are needed for ALS archaeology.","pith_inferences":["Beyond the paper's claims, the 'void' label concentrates ambiguous and excluded structures, so a model is never rewarded for predicting those regions; treating void as a fourth learnable class or modeling label uncertainty could alter the ranking of methods.","Because the dataset strips georeferencing and covers only Khmer sites, it should be read as a benchmark for recognizing Angkorian feature types, not as a general archaeological prospecting tool; cross-dataset tests on Maya or European ALS data would quantify how much transfers.","The test split mixes remote and adjacent parcels, which lets future work separate domain-shift failures from context-scale failures by reporting per-parcel scores; the authors describe this scenario but do not tabulate the breakdown.","Annotations accumulated over three decades, so label noise is likely structured by evolving survey methods; a bounded re-annotation study on a sample would show whether the small score differences between top models are meaningful."],"forward_implications":["Researchers can now compare semantic segmentation models on a common, open ALS archaeology benchmark with fixed train/validation/test parcels and released code, making results reproducible across groups.","RGB-only models are not viable for this task: the ablation shows they fall to about 30% mean intersection-over-union, so future systems must include elevation data.","The poor transfer of CLIP, DINOv2, ScaleMAE, and DOFA indicates that pretraining on natural or other satellite imagery does not substitute for data like Archaeoscape, and it argues for elevation-specific pretraining.","Because larger input windows improve detection of kilometer-scale hydrology features, architectures with large effective receptive fields are a promising direction, and the 512-pixel input size is a practical default.","The best reported performance, around 53% mean intersection-over-union, leaves substantial headroom, framing automatic ALS archaeology as an open problem rather than a solved application."],"supporting_citations":[{"why":"Supplies the KALC LiDAR campaign data over Angkor, the primary source for the 888 km² dataset.","marker":"[3]"},{"why":"Supplies the CALI campaign data, the second acquisition source for the dataset.","marker":"[39]"},{"why":"Defines the U-Net architecture, the standard CNN baseline that reaches the best 512×512 score.","marker":"[9]"},{"why":"Introduces PVTv2, the hierarchical vision transformer that posts the best 224×224 score and is used in the channel and initialization ablations.","marker":"[33]"},{"why":"Provides the DINOv2 pretrained features whose poor transfer supports the claim that natural-image foundation models do not suit elevation-dominated data.","marker":"[36]"},{"why":"Provides the CLIP pretraining whose low score is another data point for the same transfer-failure claim.","marker":"[34]"},{"why":"Introduces the plain ViT architecture that underperforms on this benchmark, framing the architecture comparison.","marker":"[28]"},{"why":"Provides the ScaleMAE geospatial pretraining that still underperforms, showing the difficulty is not simply RGB versus remote sensing.","marker":"[38]"},{"why":"Provides the DOFA Earth-observation foundation model whose poor performance reinforces the need for ALS-specific modeling.","marker":"[29]"},{"why":"The Arran dataset is the existing open-access ALS archaeology resource whose scale anchors the paper's 'largest open-access' comparison.","marker":"[13]"}],"fun_headline_variants":["Biggest open ALS dataset for archaeology still stumps AI","31k Angkor ruins mapped by LiDAR, but AI barely cracks 52% mIoU","New benchmark shows deep learning fails at LiDAR archaeology","AI can't see Angkor ruins hidden under jungle canopies","Open-access LiDAR dataset challenges AI to find ancient ruins"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ground truth is the set of expert-drawn polygons, and the paper concedes that some real features in the background are probably missing; if the labels are incomplete or their boundaries are systematically wrong, all reported accuracy scores are biased and the ranking of models could change.","fun_headline_variants_meta":{"raw":{"variants":["Biggest open ALS dataset for archaeology still stumps AI","31k Angkor ruins mapped by LiDAR, but AI barely cracks 52% mIoU","New benchmark shows deep learning fails at LiDAR archaeology","AI can't see Angkor ruins hidden under jungle canopies","Open-access LiDAR dataset challenges AI to find ancient ruins"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000666,"raw_usage":{"total_tokens":3023,"prompt_tokens":913,"completion_tokens":2110,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":529,"completion_tokens_details":{"reasoning_tokens":2020}},"tokens_in":529,"tokens_out":2110,"duration_ms":15715,"temperature":1.0,"reasoning_tokens":2020,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T20:49:03.455612+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A second independent team of Khmer archaeologists re-draws all polygons on a 10 km² sample of the test parcels without seeing the released labels; if the two labelings disagree on many mound and hydrology boundaries, then the 1–2 point accuracy gaps between the top models are within annotation noise and the benchmark ranking is not settled.","supporting_citations":[{"cited_title":"Uncovering archaeological landscapes at Angkor using LiDAR","cited_arxiv_id":null,"evidence_quote":"Supplies the KALC LiDAR campaign data over Angkor, the primary source for the 888 km² dataset."},{"cited_title":"Airborne laser scanning as a method for exploring long-term socio-ecological dynamics in Cambodia","cited_arxiv_id":null,"evidence_quote":"Supplies the CALI campaign data, the second acquisition source for the dataset."},{"cited_title":"U-Net: Convolutional networks for biomedical image segmentation","cited_arxiv_id":null,"evidence_quote":"Defines the U-Net architecture, the standard CNN baseline that reaches the best 512×512 score."},{"cited_title":"PVTv2: Improved baselines with pyramid vision transformer","cited_arxiv_id":null,"evidence_quote":"Introduces PVTv2, the hierarchical vision transformer that posts the best 224×224 score and is used in the channel and initialization ablations."},{"cited_title":"DINOv2: Learning robust visual features without supervision","cited_arxiv_id":null,"evidence_quote":"Provides the DINOv2 pretrained features whose poor transfer supports the claim that natural-image foundation models do not suit elevation-dominated data."},{"cited_title":"An image is worth 16x16 words: Transformers for image recognition at scale","cited_arxiv_id":null,"evidence_quote":"Introduces the plain ViT architecture that underperforms on this benchmark, framing the architecture comparison."},{"cited_title":"Scale-MAE: A scale- aware masked autoencoder for multiscale geospatial representation learning","cited_arxiv_id":null,"evidence_quote":"Provides the ScaleMAE geospatial pretraining that still underperforms, showing the difficulty is not simply RGB versus remote sensing."},{"cited_title":"https://github.com/ickramer/Arran","cited_arxiv_id":null,"evidence_quote":"The Arran dataset is the existing open-access ALS archaeology resource whose scale anchors the paper's 'largest open-access' comparison."}],"review_version":1}