{"id":"30a209bf-6751-4bfc-bf82-72ae278b48c6","arxiv_id":"2505.18477","paper_version":3,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"A new benchmark dataset for detecting and classifying zooplankton in cluttered marine imagery, with depth-based evaluation tracks and experiments showing modern detectors still struggle.","lead":"This paper presents ZooplanktonBench, a dataset of 285,733 annotated zooplankton images and videos from the northern Gulf of Mexico, with depth and oceanographic metadata. The authors test current object detection and vision-language models on this data and show that fine-grained zooplankton recognition remains difficult.","discovery_kind":"extension","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The paper's own §3 states copepods and appendicularians in nearly every image were not labeled; §4.1.2 then treats all unlabeled regions as marine snow, so the reported binary detection mAP of 0.871 is not actually living-vs-non-living performance.","rationale":"The paper has real value: a large expert-annotated collection with geospatial metadata, 285,733 crops, and multiple tasks. The reader's verdict of CONDITIONAL is appropriate. The most load-bearing flaw is the mismatch between the stated binary task and the actual labels. The paper itself discloses the missing labels, so this is not a hidden error, but disclosure does not remove the invalidity of the performance metric for the advertised task. I considered dataset availability and the GPT-4V prompt issues; both are important but secondary, because they affect reproducibility and one experiment rather than the definition of the benchmark's central ground truth. The proposed test directly checks whether the reported mAP survives exhaustive labeling. If the benchmark is intended only for detecting large labeled organisms, the abstract and task definitions should say so; as written, the central claim 'living zooplankton vs. marine snow' is not supported by the data. This does not warrant rejection because the dataset and code may still be useful for a clearly re-scoped task, and the authors could add an exhaustively labeled subset.","tokens_in":12783,"tokens_out":5931,"duration_ms":47828,"concrete_test":"Select a random sample of 200 raw images from each depth (10, 25, and 35 m) from the original ISIIS collection. Have two marine biologists exhaustively label every object as either living zooplankton (including copepods and appendicularians) or non-living marine snow, using the same image-processing pipeline but without the size cutoff. Re-run the YOLOv8 living-vs-marine-snow evaluation on this exhaustively labeled subset and compare mAP50 with the paper's 0.871. If mAP50 drops by more than 0.05, or if a large fraction of the newly labeled positives are detected but counted as false positives, the published binary result is an artifact of incomplete labels. The same re-annotated subset also enables a corrected fine-grained mAP for the sampled images.","verdict_should_be":"UNCHANGED","load_bearing_attack":"In Section 3, the authors state that 'smaller and more abundant organisms, such as copepods and appendicularians, were present in almost every image but were not identified.' Section 4.1.2 then defines the living-zooplankton-vs-marine-snow task by merging the fine-grained categories into 'living zooplankton' and treating 'all other distractors in the images' as marine snow. Because unlabeled copepods and appendicularians are living zooplankton, the negative class is systematically contaminated: a detector that correctly localizes an unlabeled copepod is scored as a false positive, and the marine-snow class actually contains living animals. Thus the mAP50 of 0.871 in Table 3 is not a measure of distinguishing living from non-living objects; it measures detection of the selected large taxa against an ill-defined background. The same incompleteness also affects Table 2's fine-grained detection mAP: unlabeled instances of the 17 categories, if detected, are counted as false positives, so the reported numbers are not a fair estimate of detector capability on a completely annotated benchmark. This is not a peripheral detail; it is the label definition underlying the paper's central challenge. Before ZooplanktonBench can support the stated living-vs-marine-snow task, the ground truth must be corrected or the task must be explicitly re-scoped to 'large labeled zooplankton vs. unlabeled background.'","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces ZooplanktonBench, a benchmark dataset built from 374,378 in-situ ISIIS shadowgraph images collected in the northern Gulf of Mexico at three depths (10, 25, and 35 meters), with 285,733 cropped and human-annotated zooplankton instances assigned to 17 broad categories and accompanied by geospatial and environmental metadata. The authors define two core tasks: fine-grained detection and classification of zooplankton categories, and binary detection of living zooplankton versus marine snow. They also define tracks for cross-depth generalization and for augmenting image training with unlabeled benchtop video, and they report baselines using fine-tuned YOLOv8, zero-shot Grounding DINO, and GPT-4V with and without geo-metadata. The stated goal is to provide a challenging testbed for computer vision systems in cluttered marine environments and to support marine science applications.","tokens_in":13115,"tokens_out":4109,"duration_ms":36025,"significance":"If the labeling issues identified below are resolved, ZooplanktonBench has clear value as a large, real-world benchmark: it contains over a quarter-million expert annotations, includes rich geospatial and environmental context, and defines tasks that are well-motivated by marine science. The paper provides standard mAP evaluations, multiple baseline methods, and a public project website, which are useful starting points for the community. The central claim of a living-versus-marine-snow benchmark, however, is currently weakened by the fact that many living zooplankton in the images are unlabeled and are consequently treated as negative examples in the binary task. The fine-grained detection numbers are also affected by this label incompleteness. These issues are fixable by re-scoping the tasks or by correcting the ground truth, so the underlying dataset contribution remains promising.","major_comments":[{"comment":"The binary living-zooplankton-versus-marine-snow task is defined on contaminated labels. Section 3 states that copepods and appendicularians were present in almost every image but were not identified, while Section 4.1.2 defines the task by merging the fine-grained categories into \"living zooplankton\" and treating \"all other distractors in the images\" as marine snow. Unlabeled living copepods and appendicularians are therefore scored as marine snow (negative examples), and a detector that correctly localizes such an unlabeled animal is penalized as a false positive. Consequently, the mAP50 of 0.871 in Table 3 is not a measure of distinguishing living from non-living objects; it measures detection of the selected large taxa against an ill-defined background. The task must either be re-scoped explicitly to \"large labeled zooplankton versus unlabeled background,\" or the ground truth must be corrected to include all living organisms, or a partial-label-aware evaluation protocol must be adopted.","section":"Section 3 and Section 4.1.2"},{"comment":"The fine-grained detection metrics are also affected by incomplete labeling. Because unlabeled instances of the 17 categories, notably copepods, are present in test images, any correct detection of such an instance is counted as a false positive in the mAP computation behind Table 2. This means the reported fine-grained mAP values are not an unbiased estimate of detector performance on a fully annotated benchmark. At a minimum, the authors should quantify the fraction of unlabeled category instances in the test sets or evaluate with a protocol that ignores detections matching unlabeled instances of the same category, and they should discuss how this changes the reported numbers.","section":"Section 4.1.1 and Table 2"},{"comment":"The GPT-4V evaluation lacks the reproducibility details needed for a benchmark paper. The text says \"We selected a dataset of 2,188 representative zooplankton images\" without specifying the selection procedure, the split between training and test (if any), or whether the selection was random, stratified, or exhaustive for a subset of classes. The \"Instances\" column in Table 6 sums to 2,188, which suggests the selected set may simply be all images from the listed classes, but this should be stated explicitly. The geo-metadata prompt also appears to contain a fixed set of values (latitude 29.5063167, longitude 88.5215167, depth 34.5 meters) with a note that the underlined part will be replaced; the paper must clarify whether each image receives its own corresponding metadata or whether a single metadata string is used for all images, and it must provide the exact prompt templates and selection code as supplementary material.","section":"Section 5.3 and Table 6"},{"comment":"The claimed benefit of the image-plus-video track is not supported by the reported results. The text states that an object assigned a track \"id\" is likely living zooplankton and that \"we can enhance the detection accuracy by disregarding labeled objects that are untracked,\" but Table 3 reports Images+video mAP50 of 0.840, which is lower than the Image Only mAP50 of 0.871. The paper does not report the result of the proposed filtering step separately, nor does it give the number of tracked objects, track lengths, or the fraction of detections removed. Without an ablation, the reader cannot tell whether tracking helps, hurts, or is irrelevant; the current presentation is internally inconsistent with the stated rationale.","section":"Section 5.1 and Table 3"}],"minor_comments":[{"comment":"There is a missing space in the sentence \"Our dataset presents unique challenges and opportunities\" in the abstract and the introduction; it should read \"Our dataset presents unique challenges.\"","section":"Abstract and Section 1"},{"comment":"The phrase \"otherwise purely relaying on images\" should be \"otherwise purely relying on images.\"","section":"Section 4.2.2"},{"comment":"The class name \"Thaliacea\" in Table 1 appears as \"thaliac\" in the GPT-4V prompt; these should be normalized to avoid ambiguity in the reported results.","section":"Table 1 and GPT-4V prompt"},{"comment":"The geo-metadata prompt says the image was captured \"situated 34.5 meters beneath the seafloor,\" which is likely a typo for \"below the sea surface\" or \"at 34.5 meters depth\"; this should be corrected and the exact meaning clarified.","section":"Section 5.3"},{"comment":"The paper states that code and settings are on the project website but does not specify where the dataset itself can be downloaded, what license applies, or how to access the annotations and raw videos; this information should be included for a benchmark paper.","section":"Section 6"},{"comment":"The superscripts on \"Correct\" and \"Classification\" in Table 6 are not explained in the caption; the caption should define what Image and Image+Geo denote and how the counts were computed.","section":"Table 6"}],"recommendation":"major_revision","confidential_remarks":"The manuscript is within the scope of a KDD datasets and benchmarks track and the underlying data collection effort appears substantial. The main technical concern is the label incompleteness and its effect on the benchmark's central binary task; this is a correctness issue that should be resolved before publication. The GPT-4V evaluation also needs a clearer protocol. I do not see circularity problems or novelty concerns; the dataset is externally sourced and the evaluation metrics are standard."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nThe dataset is real and potentially useful, but the headline binary task is not what it claims. Section 3 admits that copepods and appendicularians were present in almost every image and were not labeled; Section 4.1.2 then defines 'marine snow' as everything unlabeled. So the living-vs-non-living detector is really being scored on large labeled taxa against an unlabeled background that contains living animals. The mAP50 of 0.871 in Table 3 is thus not a measure of distinguishing living from non-living objects.\n\nWhat's genuinely new: a 17-class annotated zooplankton set from ISIIS in the northern Gulf of Mexico, 285,733 crops, with depth-specific splits (10/25/35 m) and associated oceanographic metadata, plus a paired benchtop video set. That is a useful resource for marine ecologists and for CV researchers working on cluttered, low-contrast imagery. The experiments with YOLOv8, Grounding DINO, and GPT-4V provide a reasonable first baseline picture, and the authors are transparent about many limitations (e.g., no video annotations, unknown category excluded).\n\nThe soft spots beyond the label contamination: the fine-grained detection mAP numbers are also affected, since unlabeled instances of the 17 categories count as false positives. The GPT-4V subset of 2,188 images is unspecified, and the geo-prompt contains an error ('beneath the seafloor' instead of below the sea surface). No error bars or multiple runs anywhere. The dataset and code are not directly linked in the paper, only via a website.\n\nNone of this makes the dataset worthless. For training detectors on large zooplankton, the partial labels may still be serviceable if evaluation is re-scoped to 'large labeled zooplankton vs. background'. But as a benchmark with the stated living vs. marine snow task, it needs a corrected ground truth or an explicit redefinition.\n\nWho this is for: marine scientists working with in situ imaging, and CV researchers who want a hard, real-world detection task. It deserves a serious referee—the resource is real and the flaw is fixable—but the authors should be asked to either re-annotate a subset completely or rename the task.\n\nRecommendation: send it to peer review, but with a request for major revision on the label definition and evaluation protocols.","headline":"A useful dataset undermined by a contaminated negative class in its central task; the binary mAP is not living-vs-marine-snow.","tokens_in":13597,"tokens_out":2389,"would_cite":false,"duration_ms":19653,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"ZooplanktonBench offers 285,733 expert-labeled zooplankton images with depth and location metadata, and shows current detectors still stumble on separating living animals from marine snow.","keywords":["Computer vision","Object detection","Fine-grained classification","Video analysis","Marine science","zooplankton benchmark","marine snow","geospatial metadata"],"falsifier":"Have a marine scientist fully label every living organism, including copepods and appendicularians, in a random sample of the training images, then retrain and evaluate the binary living-versus-marine-snow detector on a fully labeled test subset. If mAP50 falls well below the reported 0.871, the unlabeled living organisms in the negative class were inflating the apparent detection performance; if it stays near 0.871, the contamination is not decisive. A cheaper check is to count the detector's false positives that are identifiable copepods or appendicularians.","tokens_in":12601,"feed_emoji":"🦐","tokens_out":9016,"duration_ms":61533,"temperature":0.7,"pith_summary":"ZooplanktonBench is a benchmark dataset built from 374,378 shadowgraph images taken by a towed camera in the northern Gulf of Mexico at three depths, from which experts cropped and identified 285,733 zooplankton in 17 broad categories. The paper's central claim is that this resource, together with its depth and oceanographic metadata, defines a set of computer vision tasks — fine-grained detection and classification, living-zooplankton versus marine-snow detection, and tracking — that are realistic for marine monitoring and challenging for current models. Baseline results back that claim: a fine-tuned YOLOv8 detector reaches mAP50 of 0.434 on the mixed-depth fine-grained task and 0.871 on the binary living-versus-marine-snow task, while a zero-shot open-set detector and GPT-4V perform far worse, and adding geo-metadata to GPT-4V often increases errors. If correct, the dataset gives marine scientists a common testbed for automated zooplankton monitoring and gives computer vision a benchmark where background clutter, small objects, and class imbalance are the obstacles. The paper also states that the benchtop video data are not annotated, so the image-plus-video track remains a proposed pipeline rather than a fully labeled training resource.","feed_headline":"New zooplankton benchmark exposes detector limits in cluttered seas","feed_subtitle":"285,733 expert-labeled images from three ocean depths set hard tests for detection, classification, and tracking.","key_machinery":"The load-bearing object is the dataset itself, produced by shadowgraph imaging: the ISIIS towed vehicle photographs zooplankton as dark silhouettes against a lit background, and marine snow creates visually similar clutter. The annotation pipeline (ImageJ-assisted expert cropping into 17 classes) turns raw video frames into 285,733 labeled instances; the task definitions and the depth-based and image-plus-video tracks turn those instances into measurable problems, scored with mAP50 and mAP50-95. The image-plus-video track adds ByteTrack as a mechanism for using track consistency across five frames to distinguish living organisms from drifting marine snow.","core_discovery":"The paper constructs and publicly describes ZooplanktonBench, a dataset of in situ shadowgraph imagery collected with the ISIIS towed vehicle in the northern Gulf of Mexico on 24 July 2011, at 10 m, 25 m, and 35 m depth, with geospatial and oceanographic sensor data attached (coordinates, depth, temperature, salinity, dissolved oxygen, chlorophyll fluorescence). A total of 374,378 raw images were analyzed in roughly 800 person-hours, producing 285,733 cropped organisms classified by experts into 17 categories, with fish larvae further classified to family. The proposed benchmark tasks are fine-grained detection and classification of these zooplankton and a binary living-zooplankton versus marine-snow detection task; the tracks evaluate models under cross-depth transfer and under an image-only versus image-plus-unlabeled-video training regime. Quantitative baselines on the authors' protocol show a fine-tuned YOLOv8 reaching mAP50 0.434 on the mixed-depth fine-grained task and 0.871 on the binary task, a Grounding DINO open-set detector with much lower, threshold-sensitive performance, and GPT-4V classifying only 758 of 2,188 representative crops correctly, with geo-metadata raising error messages from 84 to 323. The paper's conclusion is that the dataset captures a challenging regime—variable clutter, similar shapes, small objects, and extreme class imbalance—where general-purpose computer vision systems still fall short.","pith_inferences":["Because the authors state that copepods and appendicularians were present in almost every image but were not labeled, the negative class in the binary task is contaminated with living zooplankton; the reported 0.871 mAP50 is therefore likely an optimistic estimate of living-versus-non-living discrimination, and a fully labeled negative set could lower it.","The depth-split results imply a clean experiment the paper did not run: a model that genuinely exploits depth or location metadata should outperform a depth-agnostic baseline on held-out depths, so ZooplanktonBench's geospatial fields could anchor a benchmark for geo-aware representation learning.","The unannotated benchtop videos are a natural self-supervised pretraining source; measuring whether video pretraining improves the fine-grained image tasks would directly test the paper's stated motivation for including them.","The GPT-4V outcome suggests that appending environmental variables to a text prompt is not the right interface for geo-awareness; learned fusion of image features with sensor readings is a plausible alternative that the dataset could evaluate."],"forward_implications":["Marine scientists get a common, labeled testbed for detecting and classifying zooplankton in a productive coastal ecosystem, with depth and sensor metadata attached to every image.","The depth track makes ocean depth a concrete domain-shift axis: training on a mix of depths outperforms training on any single depth in the reported fine-grained results, so future geo-aware models can be scored by cross-depth transfer.","The image-plus-video track offers a path to use unlabeled benchtop videos, through tracking-based consistency, to reduce marine-snow false positives without expensive manual video annotation.","The baseline numbers set a public bar: binary living-versus-marine-snow detection is tractable at mAP50 0.871, while fine-grained classification at mAP50 0.434 and zero-shot open-set detection are far from solved.","The reported increase in GPT-4V errors when geo-metadata is appended means the benchmark can test whether and how environmental context should be integrated into vision-language models."],"supporting_citations":[{"why":"Supplies the ISIIS towed shadowgraph imaging system that collected the raw 374,378 images.","marker":"[5]"},{"why":"Supplies the benchtop shadowgraph imaging system whose videos form the image-plus-video track.","marker":"[8]"},{"why":"ImageJ is the software used to crop and annotate the 285,733 organisms.","marker":"[32]"},{"why":"YOLOv8 is the base detector fine-tuned for the reported fine-grained and binary baselines.","marker":"[13]"},{"why":"Grounding DINO is the open-set detector evaluated zero-shot on ZooplanktonBench.","marker":"[17]"},{"why":"GPT-4V is the large vision-language model tested for fine-grained classification with and without geo-metadata.","marker":"[26]"},{"why":"ByteTrack provides the tracking consistency mechanism used in the image-plus-video track.","marker":"[39]"},{"why":"Demonstrates that coupling plankton images with context metadata can improve classification, motivating the geo-aware design.","marker":"[6]"}],"fun_headline_variants":["ZooplanktonBench: AI flunks cluttered ocean tests","New dataset reveals AI blind spots in zooplankton ID","285k images, three depths, one hard benchmark for vision","Geo-aware zooplankton benchmark trips up GPT-4V","Marine snow fools detectors: ZooplanktonBench challenge"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that every region not labeled as a zooplankton can be treated as marine snow; the paper itself notes that small living copepods and appendicularians were present in almost every image but were deliberately left unlabeled, so the binary task's negative examples are not guaranteed to be non-living.","fun_headline_variants_meta":{"raw":{"variants":["ZooplanktonBench: AI flunks cluttered ocean tests","New dataset reveals AI blind spots in zooplankton ID","285k images, three depths, one hard benchmark for vision","Geo-aware zooplankton benchmark trips up GPT-4V","Marine snow fools detectors: ZooplanktonBench challenge"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000255,"raw_usage":{"total_tokens":1654,"prompt_tokens":1112,"completion_tokens":542,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":728,"completion_tokens_details":{"reasoning_tokens":462}},"tokens_in":728,"tokens_out":542,"duration_ms":4451,"temperature":1.0,"reasoning_tokens":462,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T14:30:13.036450+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have a marine scientist fully label every living organism, including copepods and appendicularians, in a random sample of the training images, then retrain and evaluate the binary living-versus-marine-snow detector on a fully labeled test subset. If mAP50 falls well below the reported 0.871, the unlabeled living organisms in the negative class were inflating the apparent detection performance; if it stays near 0.871, the contamination is not decisive. A cheaper check is to count the detector's false positives that are identifiable copepods or appendicularians.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ISIIS towed shadowgraph imaging system that collected the raw 374,378 images."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the benchtop shadowgraph imaging system whose videos form the image-plus-video track."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"ImageJ is the software used to crop and annotate the 285,733 organisms."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"YOLOv8 is the base detector fine-tuned for the reported fine-grained and binary baselines."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"GPT-4V is the large vision-language model tested for fine-grained classification with and without geo-metadata."},{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Demonstrates that coupling plankton images with context metadata can improve classification, motivating the geo-aware design."}],"review_version":1}