{"id":"2d714fb4-aba7-4ffd-ad72-3e543e96fee8","arxiv_id":"2507.20240","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":0,"one_line_summary":"AnimalClue provides the first large-scale, multi-task dataset for recognizing animal species from indirect traces such as footprints, feces, eggs, bones, and feathers.","lead":"AnimalClue is a new image dataset with 159,605 labeled boxes of animal footprints, feces, eggs, bones, and feathers spanning 968 species. It gives computer vision researchers benchmarks for identifying species from indirect evidence, which could help automate wildlife monitoring.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Benchmark numbers inherit unverified iNaturalist labels on visually ambiguous traces; an expert label audit is needed before dataset reliability claims can be accepted.","rationale":"The reader's weakest assumption aligns with the main risk I see. I considered alternatives: (a) relation to FeathersV1—the feathers subset (555 species, 76,395 boxes) is unusually close to FeathersV1 (595 species, 28,272 images), so a hash-level overlap check would clarify novelty, but even full overlap would not invalidate the multi-trace benchmark; (b) lack of release artifacts—serious for reproducibility but not for the internal validity of the reported numbers. The label-quality issue is load-bearing because every benchmark table in Sec. 4 is a function of iNaturalist ground truth, and the paper provides no evidence that this ground truth is correct at the image level for indirect clues. The proposed audit is feasible and would settle it. The paper does have internal consistency: the published box counts sum correctly (159,605 across the five trace types; 141,314 masks after excluding footprints), and the split-by-observation protocol reduces data leakage. Since the reader already conditioned acceptance on data-quality verification, I recommend keeping the verdict unchanged; my concern strengthens rather than redirects the condition.","tokens_in":14956,"tokens_out":4735,"duration_ms":63211,"concrete_test":"Take a stratified random sample of 200 images per trace type, oversampling rare species and the visually hardest types (bones, feces), and hold it out from the released test set. Have two independent experts, blinded to iNaturalist labels, identify species from the cropped trace image alone, with no location, date, or observer context. Compute exact-match and top-3 agreement with the dataset labels; also have them flag any box or mask that contains no trace or contains multiple objects. If agreement is below, say, 95% on frequent categories and 80% on rare categories, recompute headline accuracy and mAP on the corrected subset and report label-noise bounds. If agreement is high, the label-quality concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that AnimalClue is a reliable large-scale benchmark for species identification from indirect evidence. Every reported number—classification top-1, detection/segmentation mAP, trait prediction—is scored against species labels copied from iNaturalist research-grade observations (Sec. 3.2). Research grade verifies community agreement on the observation's species, not that the specific trace image is diagnostic of that species or that the cropped region contains enough signal for identification. The authors manually removed images containing animals and overlaid text, but they report no expert re-identification of the traces themselves. For bones and feces, which the paper itself shows are hardest (species top-1 14.7–20.5% for bones, 29.6–38.6% for feces), a systematic label-error rate would distort absolute accuracies and the relative ranking of trace types; rare-category numbers near 1% could reflect label noise rather than model failure. Additionally, bounding-box and mask annotations come from the authors and an unnamed third-party service with no reported inter-annotator agreement, so detection and segmentation benchmarks carry unquantified annotation noise. Without an audit, the dataset's primary value as a benchmark is unverified.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces AnimalClue, a large-scale dataset for species identification from images of indirect animal evidence. The dataset contains 159,605 bounding boxes across five trace types (footprints, feces, eggs, bones, feathers), covers 968 species, and includes species-level labels, bounding-box/mask annotations, and 22 trait annotations. The authors establish four benchmarks—classification, detection, instance segmentation, and trait prediction—and report baseline results for representative models. The dataset is collected from iNaturalist under Creative Commons licenses, with a split that keeps images from the same iNaturalist submission in the same partition to avoid leakage. The central claims are that AnimalClue is the first large-scale multi-trace benchmark of its kind and that current models perform poorly on rare species and on visually ambiguous trace types such as bones and feces.","tokens_in":15166,"tokens_out":6457,"duration_ms":73248,"significance":"If the dataset's label and annotation quality are properly verified, AnimalClue would be a valuable contribution: it uniquely combines five trace types at a scale far beyond existing resources (e.g., OpenAnimalTracks has 18 footprint species; FeathersV1 covers only controlled-environment feather images), and it provides four tasks plus trait prediction, enabling research on an underexplored but ecologically important problem. The splitting protocol that avoids splitting the same iNaturalist submission across train and test is a sound methodological choice, and the release of data and code under permissive licenses is a strength. The paper also documents clear limitations in model performance, especially for rare categories, which can guide future work. However, the reliability of the benchmark depends on the accuracy of iNaturalist research-grade labels and on the quality of bounding-box/mask annotations, both of which are currently unquantified in the manuscript.","major_comments":[{"comment":"The species labels are inherited from iNaturalist research-grade observations without independent expert re-verification of the trace images. For ambiguous trace types such as bones and feces—where the paper's own classification results are 14.7–20.5% and 29.6–38.6% top-1 species accuracy—systematic label noise would directly distort the benchmark numbers and the relative ranking of trace types. The authors should provide an expert audit on a random sample (e.g., 1,000 images per trace type) with reported agreement rates, or otherwise demonstrate that the reported conclusions are robust to plausible label-error rates.","section":"§3.2"},{"comment":"The annotation quality for bounding boxes and masks is not quantified. Half of the footprint bounding boxes are produced by an unnamed third-party service, and egg/feather masks are initialized with SAM and then corrected; no inter-annotator agreement or quality-control metrics are reported anywhere in the paper. Since detection and instance segmentation are central benchmark contributions, the paper should report IAA on a sample or equivalent quantitative quality measures, and describe the third-party service (or at least its quality-control protocol).","section":"§3.3"},{"comment":"The source and curation protocol for the 22 trait labels (diet, activity pattern, habitat, etc.) are not described. If these traits are aggregated from external databases (e.g., EltonTraits or PanTHERIA), those sources must be cited and any mapping rules or conflict resolutions reported; if they are expert-assigned, the procedure and any validation should be described. Without this provenance, the trait-prediction results cannot be independently reproduced or interpreted.","section":"§3.3 (Trait annotations)"}],"minor_comments":[{"comment":"The phrase 'taxonomy-based split' in the Frequency categorization paragraph should presumably read 'frequency-based split' (or the intended meaning should be clarified).","section":"§3.3"},{"comment":"The Segment Anything Model is used in the annotation pipeline but is not cited; a reference should be added.","section":"§3.3"},{"comment":"The sentence 'Overall, RT-DETR achieves the best results across all categories' is not supported by Table 3: for example, on egg species RT-DETR obtains mAP 0.04 versus YOLOv8's 0.13, and on feather species 0.17 versus YOLOv8's 0.25. Please qualify the claim (e.g., by taxonomy level) or correct it.","section":"§4.3"},{"comment":"The paper reports results for 'frequent' and 'rare' categories but does not give the number of categories or the average number of training instances per category in each group; providing these statistics would help interpret the near-zero rare-category accuracies (e.g., 1.00% for ResNet-50 on rare bone species).","section":"§4.1 / Table 2"},{"comment":"The description of iNaturalist research-grade status is slightly imprecise: research-grade means community agreement on the observation, not necessarily that the specific trace image has been independently confirmed by multiple experts. Please rephrase to avoid overstating the verification.","section":"§3.2"},{"comment":"Traits classification is reported only for Swin-B; a sentence noting that this model was chosen for its best classification performance and that other models were not evaluated for traits would clarify the experimental scope.","section":"§4.5"}],"recommendation":"major_revision","confidential_remarks":"The paper's central contribution is a dataset benchmark, so the label-verification and annotation-quality concerns are the main obstacles to acceptance. If the authors can provide an expert audit of a sample and quantitative annotation quality metrics, the paper could be acceptable. I also recommend asking for trait provenance in the revision, as its absence is unusual for a dataset paper."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"AnimalClue is a solid dataset contribution, and the multi-clue scope is the real news. Existing trace datasets are single-type and mostly classification-only; this one covers footprints, feces, eggs, bones, and feathers, with 159,605 boxes, masks for four of the five trace types, and 22 species-level traits. I don't know of a competing dataset that does all of this. The split protocol is also careful: images from the same iNaturalist submission stay in the same split, which avoids the leakage that plagues many citizen-science datasets. The benchmark runs are standard, and the difficulty pattern — feathers and eggs easier, bones hard, rare species near chance — is plausible.\n\nLabels. Everything is scored against iNaturalist research-grade labels, which is community agreement, not expert verification that the specific trace image is diagnostic of the species. Best species top-1 is 20.5% on bones and 38.6% on feces; rare-category accuracy for bones is 1.0%. At those levels, label noise and model failure are hard to tell apart. The authors filter out images with animals, overlaid text, and distant subjects, which helps, but I still want a sample audit — a few hundred images per trace type checked by a wildlife expert — before treating these numbers as reference points. This is a moderate concern, not a fatal one. The trace-type ranking is probably robust, and a benchmark with some label noise still has value.\n\nBoxes and masks. Half the footprint boxes came from a third-party service, and no inter-annotator agreement is reported anywhere. Detection and segmentation mAP inherit whatever noise those annotations carry. IAA on a small sample would close this.\n\nFeathersV1. The paper distinguishes itself from FeathersV1 in one sentence, but FeathersV1 also draws on iNaturalist. The image overlap, if any, should be quantified; otherwise some of the 'new' feather data may be re-annotation of the same source images. Minor: trait labels are species-level, but the paper never says where they came from.\n\nRelease. The paper says the dataset and code will be made publicly available. For a benchmark paper, the artifact is the contribution; without a downloadable release and a stable identifier, the numbers cannot be checked. This should land before or with the camera-ready.\n\nWho is this for? People working on fine-grained recognition from environmental evidence, wildlife monitoring, and dataset reliability in citizen science. It deserves a serious referee. Send it to review, and hold it to the label audit, IAA, overlap numbers, and release. Conditional accept is the right landing zone.","headline":"A genuinely useful multi-trace dataset (five clue types, 968 species, detection/segmentation/traits) whose numbers rest on iNaturalist research-grade labels that deserve a sample audit; worth refereeing, with release and label-quality caveats.","tokens_in":15658,"tokens_out":8328,"would_cite":true,"duration_ms":95816,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"AnimalClue provides the first large-scale benchmark for recognizing animal species from footprints, feces, eggs, bones, and feathers, and its own results show current models still struggle with rare species and bones.","keywords":["animal traces","indirect evidence","species identification","wildlife monitoring","dataset benchmark","instance segmentation","trait prediction","long-tail recognition"],"falsifier":"Sample, say, 400 images per trace type from the test split and have independent wildlife experts re-identify the species without seeing the original labels; if disagreement with the dataset labels varies substantially across trace types or exceeds a few percent for bones and feces, the benchmark numbers and the paper's difficulty rankings would need revisiting.","tokens_in":14801,"feed_emoji":"🐾","tokens_out":5717,"duration_ms":66468,"temperature":0.7,"pith_summary":"AnimalClue is a first attempt to turn species identification from indirect evidence into a standard computer-vision benchmark. The paper compiles 159,605 bounding boxes from five trace types — footprints, feces, eggs, bones, and feathers — covering 968 species, adds segmentation masks for four of the five types, and attaches 22 ecological traits per species. Its own experiments show the task is learnable but far from solved: classifiers and detectors perform unevenly across trace types and collapse on rare species, with bones the hardest. The value of the dataset, if the labels are trustworthy, is that it lets researchers train and compare models that could automate non-invasive wildlife monitoring from the evidence animals leave behind.","feed_headline":"New dataset IDs 968 species from footprints, feces, feathers","feed_subtitle":"AnimalClue labels 159,605 traces with boxes and traits; rare species and bones still trip up models.","key_machinery":"The load-bearing object is the dataset itself: images collected from a citizen-science observation platform using research-grade labels, filtered to remove text overlays, visible animals, distant subjects, and faces, then annotated with bounding boxes for footprints and pixel-level masks for feces, eggs, bones, and feathers. The same citizen-science submission is kept in a single split to prevent leakage, and species are binned into frequent, intermediate, and rare bands so benchmarks expose long-tail behaviour. This pipeline, plus four benchmark protocols (classification, detection, instance segmentation, and trait prediction), is what lets the paper claim that trace-based identification is a distinct and difficult visual task.","core_discovery":"The central claim is that animal species can be recognised from indirect clues at scale, and that a public dataset with classification, detection, segmentation, and trait annotations is what makes that research possible. The paper documents that model accuracy varies strongly by trace type — feathers and eggs are comparatively easy, bones are hard — and that performance on the long tail of rare species drops far below frequent-species accuracy at every task. A feature-space analysis shows that generic and even biology-oriented vision-language models separate trace categories imperfectly, and that fine-tuning on trace images is required. The paper therefore positions AnimalClue not as a solved benchmark but as a challenging resource: the best reported order-level detection mean average precision is 0.57 and order-level instance segmentation mean average precision is 0.48.","pith_inferences":["Because the paper keeps all images from one citizen-science submission in the same split, a natural extension is to test generalization across geography or across photographers, using metadata the paper does not report on; that would probe whether models learn species-specific traces or photographer- and background-specific shortcuts.","The label-quality assumption could be stress-tested by training on a deliberately corrupted subset, flipping a fraction of species labels within a family, and measuring benchmark degradation; the paper does not include such a noise analysis.","The trait annotations open a route the paper only hints at: zero-shot or text-prompted identification, where a model is asked whether a trace matches a described diet or activity pattern rather than a species name; the fine-tuned vision-language results suggest this is worth exploring.","For conservation practice, a plausible next step is fusing evidence from multiple traces of the same species in one area, such as a footprint plus a feather, which the paper does not evaluate but its multi-trace design makes possible."],"forward_implications":["Non-invasive monitoring can be automated for evidence types field biologists already use, so surveys can scale from manual inspection to model-assisted screening.","Rare-species accuracy is the binding constraint; the benchmark's frequent/intermediate/rare breakdown implies that collecting more long-tail trace images is the clearest path to better models.","Trace type matters more than species count: feathers and eggs are easier despite having many species, while bones are hardest, guiding where annotation effort should go.","Trait prediction from traces is feasible, so models could recover ecological attributes such as diet, activity pattern, and habitat without seeing the animal, which is useful for coarse filtering in monitoring pipelines.","Releasing boxes, masks, and traits makes AnimalClue a reusable testbed for detection and segmentation methods, not only classification."],"supporting_citations":[{"why":"Citizen-science observation platform that supplies the images and the research-grade species labels the whole dataset depends on.","marker":"[17]"},{"why":"Prior footprint recognition dataset with far fewer species; establishes the comparison point AnimalClue extends for the footprint clue.","marker":"[42]"},{"why":"Prior feather classification dataset; provides the comparison for the feather clue, which AnimalClue supplements with wild-caught images and masks.","marker":"[5]"},{"why":"Large-scale species classification and detection dataset built on direct animal observations; represents the direct-observation benchmarks AnimalClue contrasts with.","marker":"[15]"},{"why":"Standard vision-language model used as a baseline in the feature-space analysis to show that trace categories are not well separated without task-specific training.","marker":"[37]"},{"why":"Biology-focused vision-language model used as a stronger baseline in the same analysis, showing that even domain-specific pretraining leaves trace identification hard.","marker":"[44]"},{"why":"Hierarchical vision transformer used as the strongest classification baseline and as the backbone for the trait classification experiments.","marker":"[32]"}],"fun_headline_variants":["AI stumbles on bones and rare species in trace-based ID","AnimalClue dataset: 968 species from indirect evidence","AI needs fine-tuning to identify species from traces","Bones and rare species stump AI on trace-based ID"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire benchmark rests on the research-grade citizen-science labels being correct for the image–species pair; if a meaningful fraction of those labels are wrong, especially for visually ambiguous traces like bones and feces, every accuracy number and trait conclusion in the paper shifts.","fun_headline_variants_meta":{"raw":{"variants":["AI stumbles on bones and rare species in trace-based ID","AnimalClue dataset: 968 species from indirect evidence","AI needs fine-tuning to identify species from traces","Bones and rare species stump AI on trace-based ID"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000871,"raw_usage":{"total_tokens":3760,"prompt_tokens":925,"completion_tokens":2835,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":541,"completion_tokens_details":{"reasoning_tokens":2769}},"tokens_in":541,"tokens_out":2835,"duration_ms":24884,"temperature":1.0,"reasoning_tokens":2769,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T13:41:31.641554+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Sample, say, 400 images per trace type from the test split and have independent wildlife experts re-identify the species without seeing the original labels; if disagreement with the dataset labels varies substantially across trace types or exceeds a few percent for bones and feces, the benchmark numbers and the paper's difficulty rankings would need revisiting.","supporting_citations":[{"cited_title":"iNaturalist: Connecting People with Nature","cited_arxiv_id":null,"evidence_quote":"Citizen-science observation platform that supplies the images and the research-grade species labels the whole dataset depends on."},{"cited_title":"OpenAnimalTracks: A Dataset for Animal Track Recognition","cited_arxiv_id":"2406.09647","evidence_quote":"Prior footprint recognition dataset with far fewer species; establishes the comparison point AnimalClue extends for the footprint clue."},{"cited_title":"Feathers dataset for fine-grained visual categorization, 2020","cited_arxiv_id":null,"evidence_quote":"Prior feather classification dataset; provides the comparison for the feather clue, which AnimalClue supplements with wild-caught images and masks."},{"cited_title":"Van Horn, O","cited_arxiv_id":null,"evidence_quote":"Large-scale species classification and detection dataset built on direct animal observations; represents the direct-observation benchmarks AnimalClue contrasts with."},{"cited_title":"Learning transferable visual models from natural language supervision","cited_arxiv_id":null,"evidence_quote":"Standard vision-language model used as a baseline in the feature-space analysis to show that trace categories are not well separated without task-specific training."},{"cited_title":"BioCLIP: A vi- sion foundation model for the tree of life","cited_arxiv_id":null,"evidence_quote":"Biology-focused vision-language model used as a stronger baseline in the same analysis, showing that even domain-specific pretraining leaves trace identification hard."},{"cited_title":"Swin transformer: Hierarchical vision transformer using shifted windows","cited_arxiv_id":null,"evidence_quote":"Hierarchical vision transformer used as the strongest classification baseline and as the backbone for the trait classification experiments."}],"review_version":1}