{"id":"23549daa-853c-4232-8959-91c33c59d3c0","arxiv_id":"2505.12066","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"An automated pipeline that turns point labels into SAM-generated bounding boxes trains YOLOv8 for beluga whale and harp seal detection in satellite imagery, beating buffer-based boxes on most classes.","lead":"This paper tests whether boxes generated by the Segment Anything Model from simple point clicks can replace hand-drawn boxes for training a whale detector in satellite images. It reports higher F1 scores than fixed buffer boxes for belugas and harp seals, but its uncertain-whale class actually performs worse.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Ground-truth boxes are expert-refined versions of the very SAM boxes under test, so the YOLO-SAM advantage may reflect benchmark construction rather than a real labeling improvement; a feature-matched recheck is needed.","rationale":"The reader's weakest assumption is exactly the ground-truth circularity concern: the evaluation ground truth is produced by manually refining SAM boxes, the same family of boxes YOLO-SAM trains on and predicts. This is the most load-bearing issue because the paper's headline claim is comparative ('SAM-labeled boxes significantly improved detection performance over buffer-based annotations'), and the comparison is scored against a yardstick derived from the method being promoted. I read the paper in good faith: the pipeline is clearly described, the ablation of YOLO-Buffer vs YOLO-Box vs YOLO-SAM is sensible, the code is public, and the use of expert refinement as a quality check is reasonable as a practical choice to obtain a usable dataset. The correction rates (19%/21%/4%) actually give some transparency about how much expert intervention remained, and the YOLO-SAM vs YOLO-Box comparison is more robust because both use the same box family. However, those strengths do not remove the structural issue: no independent ground truth is used, and Section III.C.1 explicitly attributes YOLO-SAM's gains to tight shape-aligned boxes and reduced NMS overlap, which are exactly the properties of the refined-SAM ground truth. The paper also lacks error bars or significance tests despite five runs, so we cannot tell whether the reported gaps (e.g., 72.2 vs 63.0 whale overall F1) are within run-to-run noise. But the single most load-bearing concern remains the ground-truth ancestry, because if the benchmark favors SAM-shaped boxes then even a perfectly executed experiment would produce a misleading headline. My concrete test directly addresses this by re-scoring against independently drawn boxes and sweeping the IoU threshold; it is feasible with the public kit and existing checkpoints. I therefore keep the reader's CONDITIONAL verdict; the concern does not demand rejection because the test could reasonably pass, but the paper should not be accepted as fully established without it, and the abstract 'significantly improved' language should be softened if the recheck changes the margin.","tokens_in":7223,"tokens_out":2014,"duration_ms":17542,"concrete_test":"Have two or more experienced annotators independently draw bounding boxes directly on a held-out subset of the 538 patches, without seeing SAM outputs or the point labels, following the same 'certain/uncertain/seal' rule. Then re-run the Table II evaluation of YOLO-SAM, YOLO-Buffer, and YOLO-Box against this independent ground truth, with IoU thresholds of 0.25, 0.5, and 0.75. If YOLO-SAM still beats YOLO-Buffer by a similar margin on whale overall and harp seal F1 at IoU=0.25 and the ordering is stable across thresholds, the concern is resolved. If the F1 gap shrinks substantially or reverses, the reported advantage is an artifact of the benchmark being built from SAM-derived boxes.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The evaluation in Section II.B.2 and Table II is not an independent test of the labeling pipeline. The ground truth for all three YOLO variants is built by having experts manually correct the SAM-generated masks/boxes (approximately 19% of certain whale labels, 21% of uncertain whale labels, and 4% of harp seal labels were corrected). Consequently, the test-set boxes used to score YOLO-SAM are, by construction, the same family of tight, shape-aligned boxes that YOLO-SAM was trained on, whereas YOLO-Buffer is being scored against boxes that deliberately differ from its square, fixed-size outputs (Section III.C.1 even credits tight shape alignment and reduced overlap for the gains). This shared ancestry can inflate YOLO-SAM's IoU-based metrics and depress YOLO-Buffer's, independent of any genuine detection-quality improvement. The paper does not report whether scores were computed with any alternative ground truth, nor does it quantify sensitivity to the 0.25 IoU threshold. Because the central claim is that SAM labeling matches or exceeds manual annotation while reducing labor, the validation must show the advantage survives when the comparator is not scored against a target derived from the tested method. The missing element is an evaluation where the ground truth is either expert boxes drawn without reference to SAM outputs, or at least an IoU-sweep showing the YOLO-SAM-vs-YOLO-Buffer ordering is stable.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes an automated bounding-box labeling pipeline for beluga whale and harp seal detection in very high-resolution satellite imagery. The pipeline uses point annotations together with the Segment Anything Model (SAM) to generate masks that are converted to boxes, and an overlapping-pixel assignment step to separate individuals in dense groups. These SAM-generated boxes are used to train YOLOv8, and the resulting detector is compared against two alternatives: one trained on fixed buffer boxes and one trained on manually refined SAM boxes. On a test set of 538 object-containing patches, the authors report that YOLO-SAM achieves an overall whale F1-score of 72.2% and harp seal F1-score of 70.3%, closely approaching the manually refined YOLO-Box performance and exceeding YOLO-Buffer. The paper argues that the pipeline reduces annotation labor while improving detection, especially in dense scenes, and that the method generalizes to other species and platforms.","tokens_in":7512,"tokens_out":3535,"duration_ms":32431,"significance":"If the central comparison is valid, the paper offers a practical contribution: it shows that point-driven SAM box generation can produce training annotations competitive with (or better than) fixed-size buffers, potentially reducing the manual effort needed for marine mammal monitoring in satellite imagery. The study addresses a concrete annotation bottleneck, and the authors state that the code is publicly available, which supports reproducibility. However, the main comparison is threatened by a structural evaluation bias: the test-set ground truth is built by manually refining the very SAM outputs that the proposed method produces, and the five averaged runs are reported without variance or significance tests. These issues must be resolved before the headline F1 improvements can be accepted as evidence of a real labeling advantage.","major_comments":[{"comment":"The evaluation ground truth for all three YOLO variants is constructed by experts manually refining the SAM-generated masks/boxes, with the paper reporting correction rates of approximately 19% for certain whales, 21% for uncertain whales, and 4% for harp seals. Because YOLO-SAM is trained on the unrefined SAM boxes and evaluated against boxes produced by the same generator, while YOLO-Buffer is evaluated against boxes of a different shape family, the reported F1 ordering (e.g., whale overall 72.2 vs. 63.0) may reflect benchmark construction rather than a genuine labeling-quality improvement. Please provide an independent ground-truth set drawn without reference to SAM outputs (e.g., a fresh expert annotation pass on the test patches), or at minimum an IoU-threshold sweep to demonstrate that the YOLO-SAM versus YOLO-Buffer ordering is robust to the matching criterion.","section":"Section II.B.2 and Table II"},{"comment":"The paper states that each experiment was conducted five times and results are reported as averages, but it does not provide standard deviations, confidence intervals, or any significance test. The claim in Section III.B that YOLO-SAM \"significantly outperformed\" YOLO-Buffer is therefore unsupported. Please report the per-run results or a paired statistical test (e.g., paired bootstrap or McNemar-type test) to establish that the differences are not due to run-to-run variation.","section":"Section II.C.3 and Section III.B"},{"comment":"The confidence threshold is tuned on the validation set to maximize the F1-score for certain whales, and the same tuned threshold is then applied to the test set to compare models. This procedure can favor the model whose threshold happens to best fit the validation set, and it does not account for threshold-selection variance. Please report precision-recall curves or results across a range of confidence thresholds to show that the comparative conclusions are not an artifact of the chosen threshold.","section":"Section II.C.3"},{"comment":"The retained dataset includes only image patches containing objects of interest, and the test set is a 20% split of these object-containing patches. Evaluation on such patches inflates precision and recall relative to a realistic full-scene deployment, where most of the water surface contains no animals. Please report results on full scenes or on a data set that includes negative patches, so that false-positive rates in barren areas are captured.","section":"Section II.A.3 and Section II.C.2"}],"minor_comments":[{"comment":"The column layout of Table II is ambiguous because precision, recall, and F1 values are written as consecutive numbers without clear subheadings or spacing; please reformat the table to make each metric identifiable at a glance.","section":"Table II"},{"comment":"The phrase \"accuracy across all categories\" should be replaced with \"precision\" or \"precision and recall,\" since the reported metric is precision, not accuracy in the standard classification sense.","section":"Section III.B"},{"comment":"The choice of buffer sizes for the SAM prompts (4 meters for whales and 2 meters for harp seals) is stated without justification or sensitivity analysis; a brief rationale or a small ablation would improve the reproducibility.","section":"Section II.B.2"},{"comment":"The header \"NUMBER OF (SHORT FOR NO.)\" is awkward and nonstandard; please replace with a clear column header such as \"Number of objects\" or \"Objects per class.\"","section":"Table I"},{"comment":"Reference [22] is marked \"in press\" without a DOI or year; please update it with complete publication details if available.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The central outcome depends on resolving the shared-ancestry evaluation issue: the ground-truth boxes are expert-refined versions of the tested SAM boxes. If the authors can provide an independent ground-truth evaluation or a convincing IoU-sweep robustness analysis, the paper could become publishable; without that, the headline comparison should be reframed as an analysis of annotation bias. The lack of variance reporting is a more readily fixable issue, but it should still be addressed. I also suggest the editor ask the authors to verify that the GitHub repository provides the complete label and detection pipeline, since the URL is prominent in the abstract."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth reading if you care about cutting annotation cost in satellite marine surveys; worth reading with skepticism if you care about benchmarking. The new thing here is a concrete point-to-box pipeline: SAM-H with a point prompt and small buffer, overlapping-pixel nearest-point assignment to separate touching whales, then YOLOv8s trained on the resulting boxes. The dataset is new (538 patches, WorldView-2/3, Clearwater Fiord and Churchill River), the code is public, and the paper includes an honest account of how often expert correction was needed (19% certain whales, 21% uncertain, 4% seals). I checked the numbers in Table II; the headline F1s parse correctly.\n\nThe core comparison is YOLO-SAM vs YOLO-Buffer, and SAM boxes win. That result is real-looking, but the stress-test note is right and it matters: the ground truth used to score every model is expert-refined versions of the SAM boxes themselves. The experts corrected only a minority of boxes, so most test-set boxes are close to SAM's output, and YOLO-Buffer's fixed square boxes are scored against a target that was partly built from the method it is being compared against. That shared ancestry can inflate YOLO-SAM's IoU metrics and depress YOLO-Buffer's independent of detection quality. The paper never tests sensitivity to the 0.25 IoU threshold or reports scores against an independent set of boxes drawn without reference to SAM. This is a load-bearing flaw in the comparison, not a quibble.\n\nOther soft spots are smaller. Five runs are averaged but no standard deviations or significance tests appear anywhere. The confidence threshold is tuned on the validation set per model, which is fine but should be disclosed as part of the pipeline. Evaluation is on object-containing patches only, so false alarms over the full 146–1281 km² scenes are not measured. And the abstract's claim that the pipeline 'enhances the detection of uncertain whales' is contradicted by Table II: YOLO-SAM's uncertain-whale F1 is 46.1, lower than YOLO-Buffer's 50.2 and YOLO-Box's 51.3. The authors actually admit this in the body text; the abstract should be aligned with it.\n\nWho is this for? Ecologists and remote-sensing people who want a cheaper annotation route for whale/seal surveys, and anyone building point-to-box label pipelines. The paper deserves serious referee time, but the referee should send it back for an independent ground-truth check or at least an IoU sweep showing the ordering survives, plus error bars and a corrected abstract. As it stands, the pipeline is plausible and usable, but the headline comparison is not fully established.","headline":"Useful, well-scoped pipeline paper with a real dataset and code, but the headline comparison is less clean than it looks because the ground truth is derived from the very SAM boxes under test.","tokens_in":8013,"tokens_out":4057,"would_cite":false,"duration_ms":34836,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that an automated point-to-box labeling pipeline using the Segment Anything Model can produce training boxes for beluga whale and harp seal detection that match manually refined boxes, reaching 72.2% F1 for whales…","keywords":["beluga whale detection","satellite imagery","point annotations","Segment Anything Model","YOLOv8","harp seal detection","marine mammal monitoring","bounding box labeling"],"falsifier":"Re-annotate the test images from scratch with experts who never see SAM output, then retrain YOLO-SAM and YOLO-Buffer and compare on those boxes; if SAM-labeled training no longer beats buffer-labeled training at the reported margins, the central advantage claim would collapse.","tokens_in":7055,"feed_emoji":"🐋","tokens_out":7649,"duration_ms":70976,"temperature":0.7,"pith_summary":"The paper tries to show that a human only needs to click a point on each animal in very high-resolution satellite imagery; an automated pipeline can then generate the bounding boxes needed to train a detector. It reports that boxes produced by prompting the Segment Anything Model (SAM) with those points outperform fixed-size buffer boxes and nearly match boxes that experts refined by hand, yielding an F1-score of 72.2% for beluga whales overall and 70.3% for harp seals. The study also keeps 'uncertain whales' in the training data, individuals that earlier detection studies tended to discard. If correct, this would cut the annotation labor for marine-mammal monitoring while making counts more complete.","feed_headline":"Point clicks plus SAM beat hand-drawn boxes for beluga counts","feed_subtitle":"Automated point-to-box labeling matches manual work while cutting effort and catching uncertain whales.","key_machinery":"The load-bearing object is the automated box-labeling pipeline, not any single network. For each manual point annotation, SAM receives the point plus a small buffer box as a prompt and outputs a per-animal mask; in dense scenes, an overlapping-pixel assignment rule gives every contested pixel to the nearest annotation point, producing non-overlapping instance masks that are converted to tight bounding rectangles. These rectangles become the training boxes for YOLOv8s, a single-stage object detector. The pipeline converts cheap point clicks into shape-following boxes, avoiding the fixed-square-box problem of buffer labeling and providing labels for both certain and uncertain whales as well as harp seals.","core_discovery":"The central claim is that segmentation-driven annotation from point labels is a viable substitute for manual box drawing in this remote-sensing setting. The study compares three training-label regimes for YOLOv8: fixed-size buffers around each point, raw SAM-generated boxes, and SAM boxes refined by experts. YOLO-SAM reaches the highest whale-overall and harp-seal F1 scores, 72.2% and 70.3%, improves recall for certain whales and harp seals relative to buffer labels, and performs best in dense groups where fixed buffers merge adjacent animals. The paper interprets this as evidence that the annotation bottleneck can be moved from drawing boxes to clicking points, and that SAM-based boxes are precise enough to train an effective detector.","pith_inferences":["If the pipeline generalizes, historical point-count surveys could be retroactively converted into detection training sets, unlocking decades of archived imagery without new fieldwork.","A natural next test is to apply the pipeline to a different species or sensor without retuning the buffer sizes; success would indicate the method depends on SAM's generalization rather than on dataset-specific tuning.","Because the expert-refined ground truth was derived from SAM boxes, the absolute F1 numbers may be optimistic; the fairest cross-check would be a test set labeled independently from scratch.","The point-to-box idea could be paired with active learning, where SAM boxes flag uncertain animals for human review, to improve label quality exactly where the model struggles."],"forward_implications":["Manual box annotation can be replaced by point clicking plus an automated segmentation step for beluga and seal monitoring, reducing the human labor bottleneck.","Detection models can be trained to include 'uncertain' whales rather than excluding them, making operational abundance estimates more comprehensive.","Dense aggregations of animals, where fixed buffers merge multiple individuals into one box, become countable with instance-level boxes.","The same point-to-box pipeline should transfer to other species and other very high-resolution imagery sources, since it relies on generic segmentation rather than species-specific features.","The resulting masks can also supply biometric measurements such as body length and width, adding ecological information beyond counts."],"supporting_citations":[{"why":"Supplies the Segment Anything Model used to turn point prompts into instance masks and bounding boxes.","marker":"[18]"},{"why":"Defines YOLOv8, the single-stage detector trained on the generated boxes and evaluated in the comparisons.","marker":"[12]"},{"why":"Establishes the prior gray whale detection approach and the fixed-buffer box-labeling baseline that SAM labeling is compared against.","marker":"[13]"},{"why":"Provides the protocol for classifying whales as certain or uncertain and for placing point annotations in satellite imagery.","marker":"[16]"},{"why":"Supplies pretrained weights used to initialize YOLOv8 for transfer learning on the whale and seal detection task.","marker":"[21]"}],"fun_headline_variants":["Point clicks plus SAM outdo drawn boxes for beluga counts","SAM labels from clicks lift beluga detection F1 scores","Point-to-box with SAM beats manual labeling for whale spotting","Automated point labels sharpen beluga detection over buffers","YOLO trained on SAM boxes sees more belugas with less work"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The comparison assumes the expert-refined boxes used as evaluation ground truth are an accurate and independent standard, even though they were created by manually adjusting the same SAM-generated boxes the pipeline produces.","fun_headline_variants_meta":{"raw":{"variants":["Point clicks plus SAM outdo drawn boxes for beluga counts","SAM labels from clicks lift beluga detection F1 scores","Point-to-box with SAM beats manual labeling for whale spotting","Automated point labels sharpen beluga detection over buffers","YOLO trained on SAM boxes sees more belugas with less work"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000866,"raw_usage":{"total_tokens":3773,"prompt_tokens":981,"completion_tokens":2792,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":597,"completion_tokens_details":{"reasoning_tokens":2708}},"tokens_in":597,"tokens_out":2792,"duration_ms":17817,"temperature":1.0,"reasoning_tokens":2708,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-15T20:41:16.023927+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the test images from scratch with experts who never see SAM output, then retrain YOLO-SAM and YOLO-Buffer and compare on those boxes; if SAM-labeled training no longer beats buffer-labeled training at the reported margins, the central advantage claim would collapse.","supporting_citations":[{"cited_title":"Ultralytics YOLO,","cited_arxiv_id":null,"evidence_quote":"Defines YOLOv8, the single-stage detector trained on the generated boxes and evaluated in the comparisons."},{"cited_title":"Mapping arctic cetaceans from space: A case study for beluga and narwhal,","cited_arxiv_id":null,"evidence_quote":"Provides the protocol for classifying whales as certain or uncertain and for placing point annotations in satellite imagery."},{"cited_title":"Microsoft COCO: Com- mon objects in context,","cited_arxiv_id":null,"evidence_quote":"Supplies pretrained weights used to initialize YOLOv8 for transfer learning on the whale and seal detection task."}],"review_version":1}