{"id":"0a27d3a5-f4c9-4276-b1bd-760a4e2131f5","arxiv_id":"2509.03499","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"DeepSea MOT, a public four-video benchmark with manually corrected ground truth, enables the first standardized HOTA comparisons of multi-object detectors and trackers on deep-sea ROV footage.","lead":"Researchers at MBARI built the first public benchmark for tracking multiple animals in deep-sea video: four short ROV sequences with human-checked labels, plus code and documents for scoring. They use it to compare three detection models and show that tracker settings, not habitat, drove most accuracy differences, a result that matters to anyone building automated video analysis for ocean exploration.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"ML-seeded ground truth may encode the tuned tracker's own IDs and the 452k model's blind spots, making the ~15% tuning gain and model rankings circular.","rationale":"The reader's weakest assumption—that ML-seeded, manually corrected annotations constitute an unbiased gold standard—is exactly the load-bearing point. I agree with the reader's CONDITIONAL verdict and do not propose moving it; the concern strengthens the conditionality rather than overturning the paper's resource contribution. The benchmark may still be valuable as a first public deep-sea MOT dataset, and the paper honestly reports a null result for further tracker tuning and unexplained BoxMOT discrepancies. But the quantitative headline (the 15% tuning gain and the comparison across detection models) is only interpretable if the GT is not systematically contaminated by the seeding model's detections or the tuned tracker's identity assignments. The proposed test is concrete and feasible because the GT, model outputs, and code are supposedly public: one can directly measure how much of the GT overlaps the seed output, and one can re-annotate at least one sequence to check whether conclusions survive an independent ground truth. This is a correctness risk, not a novelty or scope issue. No ad hominem is intended; the concern is about the annotation protocol's documented dependence on the evaluated systems.","tokens_in":6269,"tokens_out":4900,"duration_ms":57636,"concrete_test":"Download the published GT plus the raw outputs of MBARI 452k with tuned ByteTrack and with default ByteTrack for all four sequences. Compute (1) per class the fraction of GT boxes with no overlapping seed detection (boxes the human had to add) and (2) the fraction of GT boxes whose track ID follows a single seed track ID for at least 80% of the object's lifespan. If under-represented classes have very few added boxes, or if the ID-match fraction is high (e.g., >80%), the GT is not independent of the tuned seed. Then have one independent annotator, blind to ML output, re-annotate at least MWD (and ideally all sequences) from scratch, and recompute HOTA for MBARI 315k, MBARI 452k, default ByteTrack, and tuned ByteTrack. If the rank order or the ~15% tuning advantage changes by more than a few HOTA points, the central comparative claims are not robust.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The paper's central quantitative claims—the ~15% advantage of tuned over default ByteTrack (Figure 3) and the model rankings in Table II—require that the benchmark ground truth be independent of the systems being evaluated. Section II states that GT was produced by first running each video through MBARI's 452k model plus tuned ByteTrack, then manually fitting localizations or creating missing ones in RectLabel, after which the files 'were considered a gold-standard benchmark.' The paper reports no repair statistics, no inter-annotator agreement, and no audit of track-ID association. Two concrete mechanisms make the non-circularity assumption insecure. First, if human annotators edited boxes but left the ML-assigned track IDs intact, the tuned ByteTrack—whose parameters generated those IDs—is scored against a target partly produced by itself, while the default tracker must recover the same IDs without those parameters. This alone could manufacture a substantial portion of the reported 15% gap. Second, the manual pass is anchored by the seeded detections: classes the 452k model misses or localizes poorly (the paper itself mentions faint Poeobius meseres in MWD) may be under-added, inflating the 452k model's relative HOTA. The reported BD difference (70.202 vs 70.508) is smaller than expected annotation noise, yet no noise estimate is provided. This is not an accusation of fraud; it is an unresolved confound in GT construction that directly bears on the headline evaluation.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces DeepSea MOT, a publicly released benchmark for multi-object tracking in deep-sea ROV video footage. The dataset consists of four 600-frame sequences (midwater simple/difficult and benthic simple/difficult) with manually corrected ground-truth tracks. The paper evaluates three object detection models (MBARI 315k, MBARI 452k, FathomNet Megalodon) combined with ByteTrack, comparing default and tuned tracker hyperparameters, and reports HOTA scores. The central claims are that this is the first public deep-sea MOT benchmark, that model rankings and the tuned-versus-default tracker comparison are meaningful, and that the full pipeline (data, code, documentation, notebooks) is publicly available.","tokens_in":6502,"tokens_out":3777,"duration_ms":44598,"significance":"If the resource is sound, DeepSea MOT fills a genuine gap: deep-sea video has no standard MOT benchmark despite unique challenges (transparent organisms, marine snow, camera motion, dense aggregations). The paper ships the dataset on Hugging Face, code on GitHub, and detailed documentation, which makes the primary resource claim externally checkable and lowers the burden of proof for existence. The authors also provide a reproducible workflow for extending the benchmark, which is valuable. However, the quantitative evaluation—the ~15% tuned-versus-default gap and the model rankings in Table II—rests on ground-truth construction that is seeded by the very model and trackers under evaluation, with no reported inter-annotator agreement, repair statistics, or training-data separation analysis. These issues must be resolved before the numeric conclusions can be accepted.","major_comments":[{"comment":"The ground-truth construction is the load-bearing component for all quantitative claims. The paper states that each video was first run through MBARI's 452k model and tuned ByteTrack to produce preliminary ML-generated tracks, and that annotators then manually fit localizations or created missing ones. If the annotators retained ML-assigned track IDs or if the manual pass was anchored by the seeded detections, the tuned ByteTrack is partially scored against its own output, and the reported 15% improvement in Figure 3 may be inflated. The paper also mentions in Section III that faint Poeobius meseres are often missed, so the 452k model's blind spots could be encoded in the ground truth. Please provide per-video repair statistics (boxes added, deleted, moved, tracks re-associated, IDs re-assigned), an inter-annotator agreement study, and an explicit statement of how track IDs were created/","section":"Section II (Ground Truth Generation)"},{"comment":"All HOTA values are single point estimates with no variance, confidence intervals, or inter-annotator uncertainty. This matters directly for the model ranking: the BD difference between MBARI 452k (70.202) and MBARI 315k (70.508) is only 0.306 HOTA points, which is likely within annotation noise, yet it is used to state that 315k 'garnered the highest score' on BD. Similarly, the default-vs-tuned ByteTrack comparison in Figure 3 has no error bars. At minimum, report repeated inference runs with different random seeds (if any), or provide a calibration of annotation noise by having at least two annotators label a subset and computing HOTA stability.","section":"Table II / Figures 2-3"},{"comment":"The three MBARI detection models were trained on MBARI imagery from the same archive from which the benchmark videos were selected. The paper does not state whether the benchmark sequences, or frames temporally near them, were excluded from model training. If any of the four benchmark videos overlap with training data, the reported HOTA scores are optimistic in a way that varies per model, biasing the model rankings in Table II. Please disclose the temporal and dive identity of the benchmark videos and confirm they were held out from training for all evaluated models.","section":"Section II / Section III (Training Data Independence)"},{"comment":"The detection inference parameters differ across models: MBARI models use conf=0.001, while FathomNet Megalodon uses conf=0.15. This is likely intentional because the latter was trained differently, but it means the comparison in Table II is not a controlled comparison of detection architectures; it is a comparison of deployment-ready systems. The text should explicitly acknowledge that the rankings could change if a common inference threshold were used, and ideally report the sensitivity of HOTA to the confidence threshold for at least one model.","section":"Appendix A and Table II"}],"minor_comments":[{"comment":"The abstract says 'human-generated test data,' but the ground truth is ML-seeded and then manually corrected. Recommend changing to 'expert-corrected, ML-seeded ground truth' to avoid ambiguity.","section":"Abstract / Section II"},{"comment":"Several formatting issues: 'I NTRODUCTION' in Section I, 'V ARS' should be 'VARS', and 'T rackEval' should be 'TrackEval'.","section":"Throughout"},{"comment":"References [9] and [10] are the same URL and should be merged or clearly differentiated. Also, some references lack publisher information.","section":"References"},{"comment":"The appendix says 'evaluated in section II' but the experiments are in Section III. Please correct the cross-reference.","section":"Appendix A"},{"comment":"The term '452k model' is used without definition. Even if the model name is conventional internally, a one-line description (e.g., number of training images or resolution) would help external readers.","section":"Section II"},{"comment":"The bar charts show HOTA scores but do not include the submetrics (DetA, AssA, LocA) that are mentioned in the text. Including them or making the notebook visible would strengthen reproducibility.","section":"Figures 2-3"}],"recommendation":"major_revision","confidential_remarks":"This is a resource paper, and the public availability of the dataset and code is a real strength. The main risk is overclaiming the benchmark as a 'gold-standard' evaluation without documenting the seeding/repair process or demonstrating independence from training data. If the authors can provide the requested annotation statistics, uncertainty estimates, and holdout confirmation, the paper would be suitable for publication; if not, the quantitative conclusions should be substantially softened."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"You should know two things about this paper before reading it. First, the resource is real: four deep-sea ROV sequences (midwater and benthic, simple and difficult), ground-truth files, documentation, and evaluation notebooks are all public, and the claim to be the first public deep-sea MOT benchmark holds up against the cited literature. Second, the quantitative story—especially the ~15% HOTA gain from tuning ByteTrack—is not fully trustworthy because the ground truth itself was generated by running MBARI 452k plus tuned ByteTrack, then manually corrected. The paper gives no inter-annotator agreement, no repair statistics, and no audit of how track IDs were handled during the manual pass. If the annotators kept the ML-assigned IDs, the tuned tracker is being scored against a target it helped produce, and the default tracker has to recover those IDs without the tuned parameters. That alone could explain a chunk of the reported gap. The stress-test concern lands.\n\nWhat the paper does well: the benchmark fills a real gap, the data and code are openly available, and the authors are honest about limitations. They report a null result for further tracker optimization, say plainly that BoxMOT results are unexplained, and use a standard metric (HOTA via TrackEval) with the standard pipeline. The four-video design, with clear difficulty labels, is sensible for a first cut.\n\nSoft spots, in proportion: the ground-truth construction is the load-bearing issue. The paper asserts the files became a \"gold-standard benchmark\" after manual review, but that review was anchored by the seeded detections, so classes the 452k model misses (like faint Poeobius in MWD) may be underadded, inflating the 452k model's relative scores. The reported BD difference (70.202 vs 70.508) is well within annotation noise, yet no noise estimate is given. Two of three checkpoints are by-request only, and the tuning footage for ByteTrack is not disclosed, so test-set leakage is possible. These are fixable with transparency, but they cap how much weight the current numbers can bear.\n\nVerdict: this deserves a serious referee. The resource is valuable and checkable, and the paper is coherent and honest. But the referee should push for disclosure of tuning footage, annotation agreement and repair statistics, and an independent audit of the GT. With those additions, it becomes a solid benchmark paper. As is, cite it for the dataset, not for the tuning comparison.","headline":"A genuinely new and publicly checkable deep-sea MOT benchmark, but the headline tuning gains and model rankings sit on ground truth seeded by the same tuned tracker, so treat the quantitative claims as provisional.","tokens_in":7104,"tokens_out":1994,"would_cite":true,"duration_ms":23106,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper presents DeepSea MOT, the first publicly available benchmark for multi-object tracking in deep-sea video, with four annotated ROV sequences and HOTA baselines.","keywords":["multi-object tracking","deep-sea video","benchmark dataset","HOTA","ByteTrack","object detection","ROV","midwater and benthic habitats"],"falsifier":"Re-annotate a sample of frames from each of the four videos from raw footage without showing annotators the ML-generated tracks, then compare the blind annotations frame-by-frame to the released ground truth; if the blind pass adds, moves, or deletes boxes in ways that correlate with the seeding model's known weaknesses, the reported HOTA scores and the tuned-versus-default gap would change.","tokens_in":6076,"feed_emoji":"🌊","tokens_out":2413,"duration_ms":26099,"temperature":0.7,"pith_summary":"The paper introduces DeepSea MOT, a benchmark dataset of four ROV video sequences from midwater and benthic habitats, and claims it is the first publicly available resource for evaluating multi-object tracking in deep-sea footage. The authors argue that existing benchmarks focus on pedestrians, vehicles, or terrestrial animals and do not capture the deep-sea challenges of low visibility, marine snow, transparent organisms, high density, and occlusion. They evaluate three object detection models with ByteTrack and report performance with HOTA, finding that a manually tuned tracker outperforms default settings by roughly 15% on average. The contribution is a reusable evaluation resource plus a documented workflow and notebooks so other groups can build and use similar benchmarks.","feed_headline":"First deep-sea multi-object tracking benchmark goes public","feed_subtitle":"Four annotated ROV videos let teams compare detectors and trackers for midwater and benthic life.","key_machinery":"The load-bearing object is the benchmark dataset itself: four 600-frame sequences with manually corrected, ML-seeded ground-truth tracks, spanning habitats and difficulty levels. The evaluation machinery is HOTA (Higher Order Tracking Accuracy), a metric that equally weights detection, localization, and association, computed with TrackEval. ByteTrack, as implemented in Ultralytics, supplies the tracker; its default versus tuned hyperparameters are the experimental variable that produces the paper's main quantitative finding.","core_discovery":"The central claim is that DeepSea MOT fills a gap in tracking benchmarks by providing human-verified ground truth for deep-sea ROV video. The dataset contains simple and difficult sequences for both midwater and benthic environments, with 8 to 94 tracks per sequence and up to 47.8 boxes per frame. Using HOTA, the authors report that the MBARI 452k model with tuned ByteTrack reaches 76.776% HOTA on the simplest midwater video, while the hardest midwater video yields the lowest scores, down to 43.497% for the FathomNet Megalodon detector. The paper further reports that tuning ByteTrack's association thresholds and buffer settings improved HOTA by about 15% on average, and that additional track","pith_inferences":["If the ML-seeded, manually corrected annotations carry any systematic bias from the seeding model, all reported HOTA scores and the tuned-versus-default comparison could shift; an independent blind re-annotation study would test this directly.","The workflow could be adopted by other deep-sea and marine video archives, potentially creating a family of habitat-specific MOT benchmarks that together cover more taxa and camera geometries than this first set.","The deep-sea failure modes named here—transparent bodies, marine snow, camera motion, and dense aggregations—may make this benchmark useful for stress-testing trackers designed for other transparent or low-visibility domains.","Because the paper reports that no metric-driven tuning improved on the visually tuned tracker, the data could be used to study how HOTA gradients behave near an apparently well-optimized tracker configuration."],"forward_implications":["Deep-sea object detectors and trackers can now be compared on a common, public evaluation set instead of ad hoc internal tests.","Groups with their own ROV footage can follow the documented workflow to create additional benchmark sequences tailored to their habitats and taxa.","HOTA submetrics such as recall, precision, and association accuracy provide structured feedback for improving detection and tracking in deep-sea conditions.","Reported scores give a baseline for future models: improving on 76.776% HOTA for the simple midwater sequence and 70.508% for the difficult benthic sequence would demonstrate progress.","The large gap between default and tuned tracker settings suggests that domain-specific tracker calibration matters as much as model choice in this environment."],"supporting_citations":[{"why":"Establishes the prior multi-animal tracking benchmarks that DeepSea MOT extends, and supplies the comparison for why a deep-sea benchmark is missing.","marker":"[4]"},{"why":"Provides the Ultralytics implementation of YOLOv8 and ByteTrack, including the default tracker parameters used as the untuned baseline.","marker":"[10]"},{"why":"Supplies ByteTrack, the tracker whose tuned and default hyperparameters drive the paper's main quantitative comparison.","marker":"[11]"},{"why":"Defines HOTA, the metric used to balance detection, localization, and association in all reported scores.","marker":"[12]"},{"why":"Provides TrackEval, the evaluation software used to compute HOTA and submetrics on the benchmark videos.","marker":"[13]"},{"why":"Identifies the FathomNet Megalodon detector, the single-class model that serves as one of the three evaluated baselines.","marker":"[14]"},{"why":"Describes the VARS video annotation system and archive context from which the deep-sea footage and annotation expertise are drawn.","marker":"[1]"}],"fun_headline_variants":["First deep-sea multi-object tracking benchmark goes live","Deep-sea MOT dataset: first benchmark for underwater tracking","New deep-sea video dataset benchmark for tracking","Deep-sea ROV tracking benchmark now open for testing","First benchmark to track deep-sea creatures in video"],"cache_read_input_tokens":2688,"weakest_assumption_plain":"The benchmark's ground-truth annotations are treated as a gold standard even though they were seeded by the same kind of detector being evaluated and then manually corrected, with no reported check for systematic misses that the seeding model may have introduced.","fun_headline_variants_meta":{"raw":{"variants":["First deep-sea multi-object tracking benchmark goes live","Deep-sea MOT dataset: first benchmark for underwater tracking","New deep-sea video dataset benchmark for tracking","Deep-sea ROV tracking benchmark now open for testing","First benchmark to track deep-sea creatures in video"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000281,"raw_usage":{"total_tokens":1477,"prompt_tokens":697,"completion_tokens":780,"prompt_tokens_details":{"cached_tokens":256},"prompt_cache_hit_tokens":256,"prompt_cache_miss_tokens":441,"completion_tokens_details":{"reasoning_tokens":708}},"tokens_in":441,"tokens_out":780,"duration_ms":7918,"temperature":1.0,"reasoning_tokens":708,"cache_read_input_tokens":256,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-05T10:52:27.687829+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate a sample of frames from each of the four videos from raw footage without showing annotators the ML-generated tracks, then compare the blind annotations frame-by-frame to the released ground truth; if the blind pass adds, moves, or deletes boxes in ways that correlate with the seeding model's known weaknesses, the reported HOTA scores and the tuned-versus-default gap would change.","supporting_citations":[{"cited_title":"AnimalTrack: A benchmark for multi-animal tracking in the wild,","cited_arxiv_id":null,"evidence_quote":"Establishes the prior multi-animal tracking benchmarks that DeepSea MOT extends, and supplies the comparison for why a deep-sea benchmark is missing."},{"cited_title":"Ultralytics YOLOv8,","cited_arxiv_id":null,"evidence_quote":"Provides the Ultralytics implementation of YOLOv8 and ByteTrack, including the default tracker parameters used as the untuned baseline."},{"cited_title":"ByteTrack,","cited_arxiv_id":null,"evidence_quote":"Supplies ByteTrack, the tracker whose tuned and default hyperparameters drive the paper's main quantitative comparison."},{"cited_title":"HOTA: A higher order metric for evaluating MOT,","cited_arxiv_id":null,"evidence_quote":"Defines HOTA, the metric used to balance detection, localization, and association in all reported scores."},{"cited_title":"TrackEval,","cited_arxiv_id":null,"evidence_quote":"Provides TrackEval, the evaluation software used to compute HOTA and submetrics on the benchmark videos."},{"cited_title":"FathomNet Megalodon object detector,","cited_arxiv_id":null,"evidence_quote":"Identifies the FathomNet Megalodon detector, the single-class model that serves as one of the three evaluated baselines."},{"cited_title":"MBARI’s video annotation and reference system,","cited_arxiv_id":null,"evidence_quote":"Describes the VARS video annotation system and archive context from which the deep-sea footage and annotation expertise are drawn."}],"review_version":1}