{"id":"5ca09814-61e7-4546-8b93-04dc36c294a5","arxiv_id":"2507.15492","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A new benchmark dataset of 34,424 labels across 10,659 aerial forest images from a real manhunt where standard anomaly detectors perform poorly.","lead":"This paper releases a large aerial image dataset of hard-to-detect anomalies in dense forest, collected during a real manhunt, with over 34,000 labeled instances. It is intended as a benchmark to push anomaly detection methods toward context-aware search-and-rescue systems.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Multi-view labels rest on an unvalidated photogrammetric backprojection; a systematic error there would invalidate the 34,424-label claim, yet no accuracy assessment is reported.","rationale":"Reader identified the same weakest assumption; I agree. I read the paper in good faith: the data are openly downloadable, the flight and ground operation is described in unusual operational detail, and 238 of 405 findings were verified on the ground by police, which gives real independent evidence for the source labels. However, the 34,424-label count is the core scale and novelty claim. Backprojection is the only mechanism that creates those labels, and no evidence is provided that it is accurate under heavy occlusion and variable terrain. The benchmark conclusions are insulated from this problem because the paper explicitly restricts the evaluation to the images where volunteers originally placed labels, omitting backprojections. That makes the missing validation a gap in the dataset deliverable rather than in the benchmark. A systematic error could be actual (e.g., terrain-model height errors) or only a few pixels; the paper does not let a reader tell which. Because the concern is about a missing accuracy assessment and is addressable by a re-annotation check, it supports the reader's CONDITIONAL verdict rather than a REJECT: if the check passes, the dataset claim is plausible; if it fails, the propagated labels must be corrected or the claim restricted. Secondary issues, such as the apparent 'remaining 16' arithmetic inconsistency and the benchmark metric semantics, do not change this assessment.","tokens_in":15081,"tokens_out":8021,"duration_ms":92215,"concrete_test":"Select a stratified sample of ~50 of the 238 ground-verified findings (spanning all four classes and occlusion levels). For each, have two independent annotators draw bounding boxes around the visible anomaly in at least five overlapping images where a backprojected label exists (and in the original image). Compute mean IoU and median center offset between the backprojected boxes and the manual boxes, separately for ground-level and elevated objects. If mean IoU < 0.5 or the median center offset exceeds the object's minimum dimension, the propagated labels cannot support the '34,424 labeled anomalies' claim as delivered, and the dataset should be released with only the original-view labels (or with corrected projection) until revalidated.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The dataset's advertised added value is 34,424 multi-view labels across 10,659 images, generated by projecting the 405 crowd-sourced 2D findings into every overlapping view ('Data Review, Mapping, and Ground Operations': 'photogrammetry was used to project these labels onto every image depicting the same finding... bundle block adjustment, triangulation, and collinearity-based back-projection'). The manuscript gives no validation of this step: no ground-control or checkpoint RMSE, no comparison of backprojected boxes with manual annotations in other views, and no statement of whether a terrain model or multi-ray intersection was used. This matters technically: the original labels are 2D pixel boxes in a single image, so to transfer them the pipeline must recover a 3D position. For objects elevated above the terrain (people, tents, hunting stands) or under vegetation, a terrain-intersection model can introduce systematic horizontal offsets that grow with off-nadir angle and object height; at 3–5 cm GSD these offsets can be tens of pixels. Because the same 3D error is reprojected into every view, a consistent bias would corrupt the very multi-view supervision that distinguishes this dataset. The benchmark section uses only the original single-view labels, so the headline benchmark numbers are not affected; the 34,424-label dataset claim is. The Limitations section omits this validation gap.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript presents Weitefeld, a new aerial RGB image dataset for anomaly detection in forested terrain, collected during a real manhunt search. It contains 10,659 labeled images with 34,424 anomaly labels produced by photogrammetrically back-projecting 405 crowd-sourced findings across overlapping views, plus 19,795 unlabeled images from adjacent areas. The authors report benchmark results for six color anomaly detectors, showing poor performance, and provide a web interface for viewing and extending the dataset.","tokens_in":15340,"tokens_out":6361,"duration_ms":66065,"significance":"If the multi-view labels are accurate, the dataset fills a genuine gap: no existing public dataset targets dense-forest search scenarios with centimeter-resolution imagery, heavy occlusion, and multi-view labels. The involvement of police ground verification for 238 findings is a strong practical credential, and the public release of data and code is commendable. The benchmark, while intentionally simple, gives a useful baseline of current detector limitations. The main caveat is that the backprojection step is currently unvalidated, and the label-count arithmetic contains an inconsistency; these must be resolved before the dataset can be relied upon.","major_comments":[{"comment":"The statement 'Of the 405 findings, 238 were flagged as relevant, while the remaining 16 ... were deemed irrelevant' is arithmetically inconsistent: 238+16=254, leaving 151 findings unaccounted for. The later claim in 'Data Records' that 'Each of the findings was identified during ground observations by police' is also contradicted by the ground-operation description, which inspected only the 238 relevant findings. Please clarify the status (relevant, irrelevant, unverified) of all 405 findings and correct the text accordingly.","section":"Data Review, Mapping, and Ground Operations; Data Records"},{"comment":"The 34,424 multi-view labels are the dataset's main advertised contribution, but the photogrammetric backprojection has no reported accuracy assessment. No bundle-adjustment RMSE, tie-point statistics, terrain-model specification, or comparison of backprojected boxes with manual annotations in other views is provided. Because the original labels are 2D boxes in a single image, the transfer requires a 3D localization step; for elevated objects (people, tents, hunting stands) a terrain-intersection assumption can cause systematic horizontal offsets that grow with off-nadir angle and object height, at 3-5 cm GSD potentially tens of pixels. Since the same error is reprojected into every view, this could corrupt the multi-view supervision. Please add a validation section reporting the geometric accuracy of the backprojection and a sample of manual cross-view checks. The Limitations section should also mention this residual risk.","section":"Data Review, Mapping, and Ground Operations; Data Records"},{"comment":"The binary anomaly masks shown to volunteers during labeling were generated with the RX detector, and the benchmark then evaluates RX-G on labels that may be biased by exposure to those masks. While volunteers were instructed that masks are only a supplementary aid, the labeling process is not independent of the detector being benchmarked, so the reported RX-G detection rates may be partially circular. Please discuss this contamination and, if possible, quantify its effect (e.g., by comparing labels collected with and without mask viewing on a subset of images), or restrict the benchmark claim accordingly.","section":"Online Crowd Search; Technical Validation"},{"comment":"The benchmark is restricted to the original single-view labels ('omitting backprojections'), so the 34,424 multi-view labels are neither used nor validated by the reported experiments. This is a defensible choice for a first sanity check, but the paper should state clearly that the benchmark does not exercise the multi-view portion of the dataset, and it would strengthen the data descriptor to include a small evaluation (e.g., detection rate on backprojected boxes across views) to give users confidence in the transferred labels.","section":"Technical Validation"}],"minor_comments":[{"comment":"The heading 'contentious image segment' should read 'continuous image segment'.","section":"Table 2"},{"comment":"The caption lists 'FIRE' but the method is 'FRE' (Feature Reconstruction Error) in the text; please unify the notation.","section":"Figure 5"},{"comment":"The novelty claim that no existing dataset is designed for dense-forest search should be tempered in light of NOMAD [90] and WISARD [89], which cover partially occluded aerial SAR scenarios; please discuss the specific distinction.","section":"Related Works & Datasets"},{"comment":"The definition of 'average precision' as a per-image pixel-level ratio differs from the standard information-retrieval average precision; please clarify the terminology to avoid confusion with detection AP.","section":"Technical Validation"},{"comment":"The data format description is hard to parse; a small table or formal grammar for the entries in data.txt would improve usability.","section":"Downloading the Dataset"},{"comment":"The sentence 'Each of the findings was identified during ground observations by police' is an overstatement given that only 238 of 405 findings were ground-verified; see major comment 1.","section":"Data Records"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is a data descriptor rather than a methods paper. The central dataset claim (34,424 multi-view labels) currently rests on an unvalidated backprojection, and the label-count arithmetic inconsistency is a serious issue for a data paper. I would support publication after the backprojection is validated and the numbers are corrected. The partial circularity of the RX benchmark is a concern but can be addressed by a careful discussion. The journal should also consider whether the 'first dataset for forested SAR' claim is too strong given NOMAD and WISARD; the authors need to articulate the specific novelty more precisely."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe Weitefeld dataset is a genuinely new resource. No existing benchmark covers centimeter-resolution aerial color imagery of densely forested terrain from an actual manhunt, with 405 crowd-sourced findings and police ground verification for 238 of them. The paper documents the acquisition, labeling, and ground operations carefully, and the benchmarking study, while not deep, shows convincingly that current anomaly detectors struggle with heavy occlusion. That alone makes the dataset useful to the SAR and anomaly detection communities.\n\nThe soft spots are real but mostly addressable. The most serious is the unvalidated backprojection that turns 405 single-view findings into 34,424 multi-view labels. The paper states that bundle block adjustment, triangulation, and collinearity-based back-projection were used, but gives no accuracy assessment. At 3–5 cm GSD, a small error in object height or terrain model could shift labels by tens of pixels, and because the same error repeats across views, it could corrupt the multi-view supervision that is the dataset's unique selling point. The benchmark itself uses only the original single-view labels, so the headline results are not affected, but the dataset's core claim needs supporting evidence: reprojection error, manual checks, or at least an explicit limitation.\n\nThe count inconsistency is minor but jarring: 238 relevant plus 16 irrelevant does not equal 405. Presumably 167 is meant, but it needs fixing. The benchmark text also states that as the anomaly threshold increases, precision decreases and detection rate rises, which contradicts the paper's own explanation that a lower threshold flags more pixels; this looks like a typo, but as written it will confuse readers. And the labeling process was partially contaminated by showing volunteers the RX anomaly masks, so the RX-G baseline's near-perfect detection rate should be interpreted with that in mind. The Limitations section should acknowledge this.\n\nNone of these are fatal. They are the kind of issues that a careful revision can resolve. I would send this to peer review, and I would cite the dataset once the backprojection accuracy is documented.","headline":"Valuable first-of-its-kind dataset for forested SAR anomaly detection, but the multi-view label generation needs validation and the benchmark text has a few errors that should be corrected before the dataset is widely used.","tokens_in":15891,"tokens_out":5014,"would_cite":true,"duration_ms":48955,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The Weitefeld dataset, built from a real forest manhunt, supplies 34,424 anomaly labels across 10,659 aerial images and shows current detectors miss most occluded clues.","keywords":["aerial color imagery","anomaly detection benchmark","forest search and rescue","manhunt search","crowdsourced labeling","photogrammetric backprojection","occluded objects","bounding box annotations"],"falsifier":"Have independent annotators manually re-mark a random sample of 50 of the 405 findings in every image where the object is visible, then measure pixel displacement between their bounding boxes and the photogrammetrically backprojected boxes; if a substantial fraction, say more than 10 percent, are displaced by more than a few pixels or land in the wrong image, the multi-view supervision that makes the dataset unique is unreliable.","tokens_in":14912,"feed_emoji":"🔍","tokens_out":8748,"duration_ms":89140,"temperature":0.7,"pith_summary":"After a family murder in Weitefeld, Germany, a research aircraft scanned a forested search area and 160 volunteers manually combed 10,659 high-resolution aerial images for anything out of place. The paper releases the resulting dataset, Weitefeld: 34,424 bounding-box labels derived from 405 findings, each imaged many times from overlapping flight strips at roughly 4 cm per pixel, plus police ground-check protocols and 19,795 unlabeled images. The authors' central claim is that this is the first large-scale, realistic benchmark for anomaly detection in complex forested terrain for search-and-rescue and manhunt missions. Benchmarking six common color anomaly detectors on it, they find all perform poorly, especially under occlusion, and that a state-of-the-art object detector essentially fails, which motivates context-aware detection methods. A sympathetic reader should care because this turns a failed search operation into reusable evidence about what automated search tools can and cannot see in real forests.","feed_headline":"Manhunt photos expose why forest search detectors miss clues","feed_subtitle":"34,424 labeled anomalies from a real manhunt; six detectors miss most occluded forest clues","key_machinery":"The load-bearing mechanism is photogrammetric label propagation: 405 crowd findings marked on single images are projected onto every image depicting the same object using bundle block adjustment, triangulation, and collinearity-based back-projection, exploiting up to 85-fold along-strip overlap to create 34,424 multi-view labels. Around this sits the data-generation pipeline: the MACS aerial camera with 50 MP RGB and thermal sensors, radiometric calibration and DCB de-Bayering, gamma and saturation adjustment, Reed-Xiaoli (RX) binary anomaly masks to guide volunteers, and a web frontend supporting point and bounding-box labels with classification comments. The benchmark is driven by comparing detector anomaly masks against the backprojected bounding boxes using average precision and average detection rate.","core_discovery":"The central discovery is the dataset itself and what it demonstrates. Weitefeld is, according to the paper, the first anomaly-detection dataset built from an actual manhunt in densely forested terrain: 30,454 RGB images captured at 8,416 by 6,032 pixels with 3 to 5 cm ground sampling distance, of which 10,659 images in a priority zone were searched by 160 volunteers. Crowd workers reported 405 anomalies; photogrammetric bundle adjustment, triangulation, and collinearity-based back-projection expanded those findings into 34,424 labels across overlapping views, and 238 findings were checked on the ground by police. Benchmarking on this data, the paper reports that deep-learning detectors (FRE, FastFlow, EfficientAD) reach below 3.5 percent average precision, model-based detectors (RXG, RXM, PCA) reach near 100 percent detection rate but under 0.75 percent precision, and YOLOv12 object classification effectively fails with 0.016 percent average confidence. The paper's conclusion is that local, pixel-level color anomaly signals are insufficient; useful search tools must integrate broader context.","pith_inferences":["The paper argues that image context matters but never defines or measures it; a testable next step is to build detectors that suppress anomalies recurring globally, such as tree stumps, and evaluate whether that closes the precision gap.","Because the photogrammetric backprojection lacks an accuracy assessment, benchmark numbers may mix detector error with label noise; measuring reprojection error on a manually re-annotated subset would tell how much of the reported failure is genuine algorithm weakness.","The dataset's 'anomaly' labels are human judgments of oddity rather than confirmed target locations, since the suspect was never found; recall-oriented claims should be read with that caveat, though precision-oriented evaluation is well supported by ground checks.","The 19,795 unlabeled images invite a natural experiment: pretrain a forest-adapted representation on them and measure whether downstream detection on the labeled priority zone improves beyond ImageNet-initialized baselines."],"forward_implications":["None of the six tested color anomaly detectors performs well on Weitefeld: deep-learning methods stay below 3.5 percent average precision and model-based methods below 0.75 percent precision, which the paper reads as evidence that context-aware approaches are needed.","Automated object classification under dense vegetation is shown to be unrealistic even with a state-of-the-art detector (YOLOv12), meaning anomaly detection rather than classification is the viable automated route in such terrain.","Because each finding appears in multiple overlapping images, the dataset supports methods that exploit viewpoint changes and occlusion patterns, not just single-image pixel statistics.","The inclusion of police ground-check protocols for 238 findings turns the labels into more than pixel boxes: they carry operational relevance information that can be used to evaluate mission-oriented prioritization.","The unlabeled images from the two non-priority zones and the extendable web interface allow the benchmark to grow beyond the original manhunt and to support supervised fine-tuning or self-supervised pretraining."],"supporting_citations":[{"why":"Supplies the Reed-Xiaoli color anomaly detector used both to generate the volunteer guidance masks and as a benchmark model.","marker":"[1]"},{"why":"Feature Reconstruction Error is one of the deep-learning detectors benchmarked on the dataset.","marker":"[2]"},{"why":"FastFlow is one of the deep-learning detectors benchmarked on the dataset.","marker":"[3]"},{"why":"EfficientAD is one of the deep-learning detectors benchmarked on the dataset.","marker":"[4]"},{"why":"Reed-Xiaoli Modified is one of the model-based detectors benchmarked on the dataset.","marker":"[5]"},{"why":"Principal component analysis detector is one of the model-based detectors benchmarked on the dataset.","marker":"[6]"},{"why":"Representative existing anomaly-detection benchmark used in the comparison table to position Weitefeld against prior datasets.","marker":"[58]"},{"why":"Prior work cited to explain why dense vegetation occlusion makes object classification fail.","marker":"[82]"},{"why":"Describes the MACS aerial camera system used for image acquisition.","marker":"[94]"},{"why":"YOLOv12 is the object classifier trained to test classification performance on the dataset.","marker":"[96]"}],"fun_headline_variants":["Real manhunt data: 34k labeled anomalies, yet detectors fail","Aerial manhunt yields 34,424 anomalies; detectors score below 3.5% precision","Forest manhunt dataset benchmarks why existing anomaly detectors underperform","34,424 manhunt clues stump AI detectors in forest search"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The dataset's 34,424 labels are produced by photogrammetrically projecting 405 human findings onto every overlapping image that shows the same object, and the paper provides no accuracy assessment of that projection, so a systematic misalignment would repeat the same error across all views.","fun_headline_variants_meta":{"raw":{"variants":["Real manhunt data: 34k labeled anomalies, yet detectors fail","Aerial manhunt yields 34,424 anomalies; detectors score below 3.5% precision","Forest manhunt dataset benchmarks why existing anomaly detectors underperform","34,424 manhunt clues stump AI detectors in forest search"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000929,"raw_usage":{"total_tokens":3956,"prompt_tokens":902,"completion_tokens":3054,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":518,"completion_tokens_details":{"reasoning_tokens":2970}},"tokens_in":518,"tokens_out":3054,"duration_ms":26111,"temperature":1.0,"reasoning_tokens":2970,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T15:29:35.557846+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have independent annotators manually re-mark a random sample of 50 of the 405 findings in every image where the object is visible, then measure pixel displacement between their bounding boxes and the photogrammetrically backprojected boxes; if a substantial fraction, say more than 10 percent, are displaced by more than a few pixels or land in the wrong image, the multi-view supervision that makes the dataset unique is unreliable.","supporting_citations":[{"cited_title":"MVTec AD–A comprehensive real-world dataset for unsupervised anomaly detection","cited_arxiv_id":null,"evidence_quote":"Representative existing anomaly-detection benchmark used in the comparison table to position Weitefeld against prior datasets."},{"cited_title":"Search and rescue with airborne optical sectioning","cited_arxiv_id":null,"evidence_quote":"Prior work cited to explain why dense vegetation occlusion makes object classification fail."},{"cited_title":"Capabilities and Applications of MACS Aerial Camera Systems for Environmental Research","cited_arxiv_id":null,"evidence_quote":"Describes the MACS aerial camera system used for image acquisition."}],"review_version":1}