{"id":"415689c5-92f8-463a-9b52-8c2b46d488ea","arxiv_id":"2510.09458","paper_version":2,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"SilvaScenes provides an expert-annotated benchmark showing that under-canopy tree trunk segmentation is feasible (mAP 67.65%) while species-aware segmentation is still challenging (mAP 35.69%).","lead":"SilvaScenes is a new dataset of under-canopy forest images with 1,476 trees across 24 species, each labeled by forestry experts with species and trunk masks. It benchmarks modern instance segmentation models and finds that trunk detection works reasonably but species-level segmentation remains difficult.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth species labels have no inter-expert agreement or label-noise measure; annotation ambiguity could be mistaken for task difficulty and invalidate the 24-species dataset claim.","rationale":"The reader's weakest assumption identifies exactly the most load-bearing concern: the reliability of expert species labels. The paper's central contributions are (1) a dataset of under-canopy images with 24 species and (2) a benchmark showing species classification remains challenging (mAP 35.69%). Both claims depend on the ground-truth species labels being correct. If the labels contain noise or bias, the dataset's species inventory may be inaccurate, and the benchmark numbers conflate annotation ambiguity with intrinsic task difficulty. Section III-D explicitly admits that some trees cannot be reliably identified (Unknown class) and that 'most' (not all) ground truth was obtained in situ by a forestry expert, leaving the labeling process for the remainder unspecified. No inter-expert agreement or label-noise estimate is reported, which is a standard requirement for fine-grained natural-world datasets. The confusion matrix in Fig. 3 further suggests that species with similar bark are frequently confused by the model, but without label validation we cannot distinguish model error from ground-truth error. The concrete test proposed—independent re-annotation of a subset and comparison of model performance against the new labels—would directly quantify the impact of label noise. We agree with the reader that the abstract/full-text numerical discrepancies are serious reporting flaws, but they are secondary to the label-reliability issue because even after correcting the numbers, the benchmark's scientific meaning would remain uncertain without label validation. Since the paper can be revised to address this concern, the verdict stays CONDITIONAL, aligning with the reader's assessment.","tokens_in":12308,"tokens_out":4652,"duration_ms":43838,"concrete_test":"Release the dataset, then have at least two forestry experts who were not involved in the original annotation independently re-identify species from a random subset of at least 20% of the images (e.g., 35 images, ~300 trees) using the same annotation guidelines (or, if feasible, in situ at the same sites). Compute per-species Cohen's kappa and percent agreement between the original labels and each new annotation. Additionally, evaluate M2F-Large on the subset using the new labels as ground truth and compare the resulting mAP to the original. If per-species kappa is ≥0.8 and the mAP drop is within the reported fold variance, the label-reliability concern is resolved; otherwise, the reported benchmark numbers and the dataset's species inventory must be revised to account for label noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that species classification is inherently challenging (mAP 35.69%) assumes the expert-derived ground-truth labels are correct. Section III-D states that 'ground truth for most of the data was obtained in situ by a forestry expert' — 'most' leaves a subset labeled by another, unspecified process — and that trees that 'cannot be reliably identified' are placed in an Unknown class. No inter-expert agreement, no label-noise estimation, and no validation of the expert identifications are reported. If labels are noisy or biased, the reported mAP could be depressed by label noise rather than reflecting intrinsic task difficulty, and the dataset's claim of '24 species' is only as trustworthy as the expert labels. The confusion matrix (Fig. 3) shows systematic confusion between species with similar bark (e.g., BBP vs PAB), and the authors attribute misidentifications to bark detachment, which also raises the possibility that some ground-truth labels are wrong. Because the dataset and code are not yet released, these labels cannot be independently verified. This is the weakest load-bearing assumption.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The manuscript introduces SilvaScenes, an under-canopy RGB dataset of 172 images collected across five bioclimatic domains in Quebec, containing 1,476 individually annotated trees from 24 species (plus an Unknown class), with instance segmentation masks and forestry-expert species labels. The authors benchmark six model variants (Mask2Former with Swin-Small/Swin-Large, YOLOv11 Small/X-Large, YOLOv12 Small/X-Large) under a stratified five-fold cross-validation protocol using standard instance-segmentation metrics. The best species-aware mAP is 35.69% for Mask2Former with Swin-Large, while the full-text abstract reports a binary trunk-segmentation mAP of 67.65%. The authors conclude that trunk segmentation is feasible but fine-grained species classification remains a substantial challenge, and that higher image resolution yields consistent performance gains.","tokens_in":12578,"tokens_out":5305,"duration_ms":50385,"significance":"If the dataset and labels are trustworthy, SilvaScenes fills a real gap: existing ground-level datasets either lack species labels, cover only a handful of visually distinct species, or were collected in urban or snowy settings with lower clutter. The benchmark is reasonably designed — standard metrics, multi-architecture comparison, stratified cross-validation, and a useful confusion-matrix analysis. The resolution-scaling experiment is a valuable addition. However, the current manuscript contains a serious inconsistency in headline numbers between the arXiv abstract (1,421 trees, 28 species, mAP 69.9/39.2) and the full text (1,476 trees, 24 species, mAP 67.65/35.69), and the species labels — the core scientific payload — are not backed by any inter-expert agreement or label-noise analysis. These issues need to be resolved before the quantitative claims can be accepted.","major_comments":[{"comment":"The headline numbers are inconsistent across versions of the same manuscript. The arXiv abstract reports 1,421 trees, 28 species, mAP 69.9%/39.2%, whereas the full-text abstract reports 1,476 trees, 24 species, mAP 67.65%/35.69%. Table II supports 35.69% as the best species-aware mAP for M2F-Large, but no table or protocol is provided for the binary-trunk mAP of 67.65% (or 69.9%). The authors must use a single consistent set of numbers and explicitly state how the binary segmentation mAP was computed, since it is not derivable from the reported species-aware experiments.","section":"Abstract vs. full text; Table II"},{"comment":"Ground-truth species labels are load-bearing for the central claim that 'species classification remains a significant challenge.' The text states that 'ground truth for most of the data was obtained in situ by a forestry expert,' but does not specify how the remaining data were labeled, nor does it provide inter-expert agreement, label-noise estimates, or any validation of the expert identifications. Confusion patterns such as BBP/PAB in Fig. 3 could be caused by bark detachment in the field, but they could also reflect label error. The authors should report the number of trees/images labeled by the primary expert versus any other process, provide an independent expert audit on a subset, and discuss the expected impact of label noise on the 35.69% mAP figure.","section":"Section III-D, item 1; Section V"},{"comment":"The benchmark trains on 21 classes because four species with fewer than ten trees are merged with Unknown into an 'Other' class. This is methodologically defensible, but the paper presents SilvaScenes as a 24-species dataset. The result is that per-species performance for those four species cannot be measured, and 'Other' conflates rare species with damaged/unidentifiable trees, making the task harder. The authors should identify the four merged species explicitly and report per-species AP for all 24 species (even if low-confidence), or clearly state that the benchmark covers 24 species with only 21 trainable classes.","section":"Section IV-B; Table I"}],"minor_comments":[{"comment":"Typo: 'UA V' should be 'UAV' in Section I.","section":"Introduction"},{"comment":"Please specify the exact YOLO variants used (e.g., YOLOv11n/s/m/l/x and YOLOv12n/s/m/l/x) and the source versions for reproducibility.","section":"Section IV-A"},{"comment":"The 'Multi → Binary' and 'Binary → Binary' curves rely on binary labels that are not defined in detail. State whether binary masks are simply the union of all species masks and whether the same five-fold splits are used. Also define 'IQR over five folds' more precisely in the caption.","section":"Section V / Fig. 5"},{"comment":"The species code 'Pensylvanica' should be 'pensylvanica' for the epithet, and the Unknown class is missing from the taxonomic grouping. Consider adding a total column to make the per-domain sums easier to verify.","section":"Table I"},{"comment":"The manuscript says the dataset and code 'will be made available' but gives no currently accessible link, license, or data sheet. A reviewer cannot verify the central claims without access; please provide at least a clear availability statement with a persistent identifier.","section":"Dataset availability"},{"comment":"Reference [47] is cited to support the claim about typical species imbalance in natural forests, but it concerns citizen-science UAV data; a more direct forestry reference would strengthen that sentence.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"I am sympathetic to the dataset's contribution and think the benchmark itself is adequate; the main fixes are the abstract/full-text number inconsistency and the lack of label-reliability evidence. The latter is not a simple presentation issue because it directly bears on whether the reported species mAP reflects visual task difficulty or annotation noise. If the authors can provide a label audit and resolve the number discrepancy, I would be willing to reconsider. I also suggest the authors include a comparison table of existing ground-level forestry datasets to substantiate the 'first under-canopy instance segmentation dataset with 24 species' claim, since the related-work section is qualitative."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The short version: SilvaScenes is a genuine contribution — the first under-canopy benchmark for instance segmentation of tree species with 24 species and masks — and the benchmark results are plausible. But the arXiv abstract contradicts the full text on the headline numbers, and the ground-truth labels have no inter-expert agreement or noise estimate. Both are fixable, but they matter.\n\nThe dataset fills a real gap: FinnWoodlands has 3 genera, TDoUS is urban, BarkNet and CentralBark are close-up bark images without detection. SilvaScenes adds 172 images across five bioclimatic domains, 1476 trees, 24 species, with trunk masks and expert species labels. The collection methodology is careful — no tree captured twice, width threshold, off-trail, expert in situ ID. Benchmarking is standard: five-fold CV, multiple architectures, COCO metrics, confusion matrix and resolution scaling analysis. The resolution power-law finding is a nice practical insight.\n\nSoft spots. First, the abstract reports 1421 trees, 28 species, mAP 69.9/39.2 while the full text says 1476 trees, 24 species, mAP 67.65/35.69. That's not a rounding error; it's a mismatch on the two most important numbers. Must be fixed before publication. Second, the stress-test note is right: Section III-D says ground truth for 'most' of the data was obtained in situ by a forestry expert, with no inter-expert agreement or label-noise measurement, and an Unknown class for trees that can't be reliably identified. Without label-quality metrics, the species classification difficulty (mAP 35.69) could partly reflect annotation ambiguity. That's a real limitation for a dataset paper, though not fatal — expert labels are the standard, and the confusion pattern (e.g., BBP vs PAB from bark detachment) is plausibly intrinsic. Still, they should release a label audit or at least a second-expert sample. Third, the dataset and code aren't actually available yet, just a promise. That's okay for an arXiv preprint but weakens the 'benchmark' claim temporarily.\n\nWho is this for? Forestry robotics and fine-grained visual recognition researchers. It deserves a serious referee — the gap-filling is real and the experiments are competent — but it needs the abstract corrected, label-noise addressed, and artifacts released. I'd send it to peer review with major-revision expectations.","headline":"Valuable under-canopy tree species segmentation dataset with a serious abstract/full-text inconsistency and unmeasured label noise; needs revision but deserves refereeing.","tokens_in":13049,"tokens_out":1709,"would_cite":true,"duration_ms":15368,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper introduces SilvaScenes, a benchmark dataset for under-canopy tree species instance segmentation, and shows that while locating tree trunks is feasible, identifying their species remains a largely unsolved problem.","keywords":["under-canopy imagery","tree species classification","instance segmentation","forestry robotics","benchmark dataset","species imbalance","image resolution","Mask2Former"],"falsifier":"Re-annotate a random subset of SilvaScenes images with multiple independent forestry experts and measure label agreement; if agreement is low, then a large part of the species-classification gap is annotation noise rather than perceptual difficulty. Alternatively, train the same models on a carefully balanced subset and see if mAP rises sharply, which would confirm species imbalance as the dominant factor.","tokens_in":12220,"feed_emoji":"🌲","tokens_out":4306,"duration_ms":35699,"temperature":0.7,"pith_summary":"The paper introduces SilvaScenes, a dataset of 172 under-canopy images from natural forests, with 1,476 trees across 24 species annotated by forestry experts. Its central claim is that while locating and segmenting tree trunks is feasible with modern deep learning (best mAP 67.65%), identifying the species of each segmented trunk remains a hard, unresolved problem (best mAP 35.69%). The authors establish this by benchmarking several instance segmentation models under a common protocol, and they show that image resolution, species imbalance, and occlusion are key factors. A reader should care because forestry automation and inventory depend on exactly this perception capability.","feed_headline":"Trunk detection works, species ID lags in under-canopy forest benchmark","feed_subtitle":"New 24-species SilvaScenes dataset: best model hits 67.7% mAP on trunks but only 35.7% on species.","key_machinery":"The load-bearing element is the dataset itself: 172 high-resolution under-canopy images collected across five bioclimatic domains in Quebec, with trunk instance masks and fine-grained species labels assigned in situ by forestry experts. Annotation rules — trunk-only masks, a 16-pixel median-width threshold for labeling, forking trunks counted as separate trees, and grouping of rare or unidentifiable trees into an 'Other' class — define the task. The benchmark uses Mask2Former with Swin backbones and YOLO variants, trained with focal loss under five-fold stratified cross-validation, and the resolution ablation (downsampling from 1.6 MP to 0.1 MP) is the key diagnostic showing a power-law impr","core_discovery":"The central discovery is that under-canopy species-aware instance segmentation is a distinct and currently unsolved task, and SilvaScenes is offered as the benchmark to measure progress on it. The paper shows a large gap between class-agnostic trunk segmentation and species classification, with the best model reaching 67.65% mAP for the former and only 35.69% for the latter. It also finds that doubling image resolution improves mAP by about 6%, that confusion is concentrated among visually similar species such as spruces, and that the common practice of downsampling images may be holding back performance.","pith_inferences":["Because ground-truth species labels come from a single expert per site with no reported inter-expert agreement, part of the 35.69% mAP ceiling could reflect label ambiguity rather than intrinsic visual difficulty; a multi-annotator reliability study would disambiguate this.","The resolution scaling trend suggests that the current practice of downsampling to ~1 MP is a major bottleneck; if the trend holds, using full 100 MP images or learned super-resolution could close a substantial fraction of the gap.","The confusion pattern between deciduous and coniferous species (only 8% of errors) hints that a coarse 'tree type' classifier is nearly solved, and fine-grained species ID might benefit from additional cues like bark texture, context, or temporal information.","The dataset's explicit avoidance of duplicate trees across images controls data leakage, but it also means models never see the same tree from multiple viewpoints — a condition likely in real robotic deployments, so a multi-view extension would be a natural next benchmark."],"forward_implications":["Species-aware under-canopy segmentation is an open problem; current models leave large room for improvement (best mAP 35.69%).","Higher-resolution imagery is a reliable lever: mAP increases roughly 6% per doubling of resolution, suggesting the field should move beyond standard downsampled inputs.","Species imbalance and occlusion are the dominant error sources; the confusion matrix shows frequent confusion between red/sugar maples and among spruces.","The dataset provides a public benchmark for future work on forestry perception, semantic SLAM, and precision forestry.","Binary trunk segmentation results show that detection itself is close to solved in these conditions, so the bottleneck is fine-grained species discrimination."],"fun_headline_variants":["Under-canopy tree species ID: benchmark reveals tough gap","Forest tree species recognition lags trunks in new SilvaScenes dataset","New benchmark: tree trunks detectable, species ID still hard under canopy","Higher resolution improves under-canopy tree ID, benchmark shows","SilvaScenes dataset challenges AI with 28 tree species in natural forests"],"cache_read_input_tokens":2304,"weakest_assumption_plain":"The ground-truth species labels are assigned by forestry experts but with no measured inter-expert agreement, so if expert labels are noisy or biased, the benchmark's reported difficulty could be overstated.","fun_headline_variants_meta":{"raw":{"variants":["Under-canopy tree species ID: benchmark reveals tough gap","Forest tree species recognition lags trunks in new SilvaScenes dataset","New benchmark: tree trunks detectable, species ID still hard under canopy","Higher resolution improves under-canopy tree ID, benchmark shows","SilvaScenes dataset challenges AI with 28 tree species in natural forests"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000274,"raw_usage":{"total_tokens":1400,"prompt_tokens":844,"completion_tokens":556,"prompt_tokens_details":{"cached_tokens":0},"prompt_cache_hit_tokens":0,"prompt_cache_miss_tokens":844,"completion_tokens_details":{"reasoning_tokens":481}},"tokens_in":844,"tokens_out":556,"duration_ms":476853,"temperature":1.0,"reasoning_tokens":481,"cache_read_input_tokens":0,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-04T10:25:47.859740+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a random subset of SilvaScenes images with multiple independent forestry experts and measure label agreement; if agreement is low, then a large part of the species-classification gap is annotation noise rather than perceptual difficulty. Alternatively, train the same models on a carefully balanced subset and see if mAP rises sharply, which would confirm species imbalance as the dominant factor.","supporting_citations":[],"review_version":1}