{"id":"797d6657-aca7-4aef-80a6-cf64b286a41e","arxiv_id":"2501.04873","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":4,"one_line_summary":"A coast-labeled seashell dataset and a ConvNeXt-Tiny pipeline that predicts Pacific versus Caribbean provenance with 86.28% balanced accuracy.","lead":"The authors built a 19,000-image dataset of Costa Rican seashells labeled by Pacific or Caribbean coast, and trained a lightweight model that predicts a shell's coast with 86% accuracy. The system is deployed for wildlife officers, and the goal is to return confiscated shells to the correct coastline.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The 86.3% test accuracy is computed on an image-level split that almost certainly shares species between training and test; without a species-disjoint split, the claimed generalization to never-before-seen confiscated shells is not established.","rationale":"Reader's weakest assumption is the load-bearing one and I agree with it. The strongest empirical claim in the abstract is the held-out balanced accuracy, and that claim is only meaningful for deployment if the test set contains specimens the model has not seen at the species level. The paper's split description in Section 4.1 guarantees family balance, not species exclusivity, and the deployment claim is specifically about confiscated shells that are new to the system. The released dataset and deployed web service are genuine contributions, but they do not substitute for a species-disjoint evaluation. Minor inconsistencies exist (19,051 vs 19,058 images; 167 vs 168 rejected OOD images between Section 5.2 and supplementary Table 5), but they are not the primary threat. No fraud or misconduct is indicated; the issue is a protocol gap that can be closed with metadata and a re-evaluation. Verdict stays conditional.","tokens_in":16116,"tokens_out":6680,"duration_ms":68880,"concrete_test":"On the released HuggingFace dataset, inspect the per-image species metadata and verify whether any species appears in both the training and test splits. Then re-run the evaluation under a strict species-disjoint split: hold out all images of a random 15% of species for testing, train only on images of the remaining species, and report balanced accuracy with confidence intervals. If this accuracy falls materially below 86.28%, the abstract's generalization and 'safe repatriation' claims are not supported by the current evaluation.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 4.1 describes a 70/15/15 split that 'ensured a balanced representation of families across all subsets,' but it does not state that images from the same species were confined to one split. With 516 species and 19,058 images (about 35 images per species on average), a family-stratified image-level split will put multiple images of nearly every species into both training and test. The reported 86.28% balanced accuracy therefore measures recognition of previously seen species, not performance on 'completely new seashells' as the deployment scenario requires. Since the conclusion uses this number to claim safe repatriation, the central claim hinges on the species-exclusivity of the split. The supplementary species lists even show some species (e.g., Bulla punctulata, Semele purpurascens, Strigilla dichotoma) appearing on both coasts, so the check must use per-image species metadata rather than coast labels alone.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces BackHome19K, a dataset of 19,058 photographs of 516 seashell species labeled by Costa Rican coast (Pacific vs. Caribbean), and a two-stage pipeline: a SqueezeNet-based k-nearest-neighbor similarity filter (PaDiM-inspired) that rejects out-of-domain inputs, followed by a ConvNeXt-Tiny binary classifier. The authors report 86.28% balanced test accuracy (mean over ten runs), a 93% rejection rate on 180 out-of-domain images, zero false negatives on 40 seashell controls, and a deployed web application with sub-3-second end-to-end latency. Ablations examine the number of unfrozen layers and training schedules. The dataset is publicly released on HuggingFace.","tokens_in":16315,"tokens_out":7183,"duration_ms":68608,"significance":"If the generalization numbers hold, this is a useful applied contribution: it provides the first coast-level seashell image corpus for a conservation task, demonstrates a lightweight deployable architecture, and includes practical deployment measurements. The public dataset release and the reporting of mean±std over ten runs are commendable. However, the central claim of generalization to never-before-seen shells currently rests on an image-level, family-balanced split rather than a species-disjoint split, and the anomaly-filter threshold is tuned on in-domain data and then evaluated on in-domain controls. Both issues are fixable within the manuscript's scope, but they are load-bearing for the abstract's quantitative claims and for the conclusion that the system enables safe repatriation.","major_comments":[{"comment":"The split protocol is a family-balanced image-level split, not a species-disjoint split. The paper states only that the split 'ensured a balanced representation of families across all subsets', with no statement that images of the same species were confined to one subset. With 516 species and roughly 19,000 images (about 35 images per species on average), a family-stratified image-level split will almost certainly place multiple images of the same species in both training and test. The reported 86.28% balanced accuracy therefore measures recognition of previously seen species, not performance on 'completely new seashells' as claimed in §4.1 and as required by the confiscation scenario. This is the load-bearing empirical claim of the paper, so the evaluation must be repeated with a species-disjoint split (or, at minimum, accuracy reported separately for species whose images were entirely held out), with the degree of species overlap quantified. The split must use per-image species metadata rather than coast labels alone, because the supplementary species lists contain species such as Bulla punctulata, Semele purpurascens, and Strigilla dichotoma on both coasts.","section":"§4.1, §5.1, Table 3"},{"comment":"The anomaly-detection evaluation is partly circular and inconsistent. The threshold λ=0.955 and neighborhood size k=5 are determined 'empirically by analyzing the distribution of similarity scores between known seashell images' (§3.3), and the 'zero false negatives' result is then reported on 40 in-domain seashell controls drawn from the same population used to set the threshold. This makes the recall claim circular, and n=40 gives a one-sided 95% upper bound of roughly 7.5% on the false-negative rate even with zero observed failures. Please set λ and k on a held-out tuning set or via cross-validation, and report recall with an uncertainty estimate. In addition, the OOD evaluation numbers are inconsistent: §5.2 first reports '200 images across 20 object categories (10 images each, except 40 seashell images)' and '90.5% of non-seashell objects', then reports 180 images across 18 categories with 93% rejection, while Table 5 lists 19 non-shell categories whose 'below threshold' counts sum to 168. These counts must be reconciled.","section":"§3.3, §5.2, Table 5"},{"comment":"The abstract claims the system 'has already processed 70,000 shells for wildlife officers', but Section 6 reports a three-day public launch with about 200 unique users and roughly 36,000 classifications, with no described deployment involving wildlife officers. The 70,000 figure appears unsupported by the manuscript's own deployment section. Relatedly, the conclusion that the system enables 'safe repatriation' is stronger than the evidence: at 86.28% balanced accuracy, about 14% of shells receive the wrong coast label, and no analysis is provided of the ecological risk of releasing a shell on the wrong coast. Please align the abstract with the deployment data actually reported, or provide the missing operational details, and temper the safety claim to match the demonstrated error rate.","section":"Abstract, §6, §9"}],"minor_comments":[{"comment":"The dataset size is reported inconsistently: the abstract says 19,058 photographs, Section 3.1 says 19,051 images, and Table 1 sums to 19,058. Please harmonize these numbers.","section":"§3.1, Table 1"},{"comment":"The ablation table is difficult to interpret: the header 'Unfrozen Epochs Scheduling Accuracy' and the row values (e.g., '0 50 25 83.24') do not make the column semantics clear, and the first two rows (both '0 50 25') report different accuracies (83.24 and 84.11) without explanation. Please reformat the table with explicit column headings and clarify the duplicate configuration.","section":"Table 4"},{"comment":"The phrase '93 % true-positive rate' for out-of-domain rejection is confusing because the classifier's 'true positives' are defined differently in the classification task; consider using 'anomaly detection recall' or 'rejection rate' consistently throughout.","section":"§5.2"},{"comment":"The caption contains a typo ('mean feature vectors for each specie') and should read 'species'; it would also help to specify whether the t-SNE/DBSCAN visualization was computed on the training split or the full dataset.","section":"Figure 5 caption"}],"recommendation":"major_revision","confidential_remarks":"The stress-test concern about species leakage is not a hypothetical artifact: the split description in §4.1 explicitly mentions only family balance, and the supplementary species lists show species shared between coasts, so the species-exclusivity of the split is a genuine open question. The anomaly-threshold circularity is also real and affects the abstract's 'zero false negatives' claim. I do not see grounds for rejection if the authors provide a species-disjoint evaluation and a non-circular threshold-setting protocol, but the manuscript is not ready in its current form because the central quantitative claims depend on these fixes. The 70,000 vs. 36,000 discrepancy should also be resolved before publication."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Here's my take: the paper is worth a referee, but the main empirical claim needs a better test before we believe the deployment story.\n\nWhat's genuinely good: BackHome19K is a new public resource—19,058 images, 516 species, with Pacific/Caribbean coast labels. No existing shell dataset does this. The authors also shipped a working two-stage pipeline (ConvNeXt-Tiny classifier plus a PaDiM-style anomaly filter) and report concrete deployment numbers: 70,000 shells processed, sub-3-second latency. For conservation teams that need to triage confiscated shells, this is a real deliverable, not just a toy.\n\nThe soft spot is the evaluation protocol. Section 4.1 describes a family-balanced 70/15/15 split, but it never says images from the same species were confined to one split. With roughly 35 images per species on average, a random image-level split will put the same species, even the same specimen, into both training and test. The supplementary species lists confirm that species like Bulla punctulata, Semele purpurascens, and Strigilla dichotoma appear on both coasts, so the model could be remembering individuals or near-duplicate views rather than generalizing to \"completely new seashells\" (their words). The paper needs a species-disjoint, ideally specimen-disjoint, split, plus accuracy on independently collected confiscated shells. The reported 86.28% balanced accuracy is plausible, but it is an upper bound until that split is released.\n\nAnomaly detection also has small internal inconsistencies: the abstract says 93% rejection of 180 out-of-domain images, Section 5.2 says 168/180, and the supplementary table lists 19 categories whose below-threshold entries sum to 167. That's a minor issue, but it should be fixed.\n\nThe \"safe repatriation\" conclusion overreaches. 86% accuracy leaves a real error rate, and the paper shows no validation on actual confiscated specimens, only usage counts. The system likely helps a human triage process; it is not an autonomous repatriation decision.\n\nOverall: the dataset is the contribution, the engineering is real, and the evaluation gap is fixable. A serious referee should send this to major revision, not desk reject. I would bring it to a reading group as a case study in applying computer vision to conservation, and I would cite the dataset once the split metadata is public.","headline":"A genuinely useful new shell-provenance dataset and a real deployed system, but the headline 86.3% accuracy rests on a split that likely leaks species between training and test, so the generalization claim is not yet established.","tokens_in":16810,"tokens_out":4382,"would_cite":true,"duration_ms":40565,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A photograph of a confiscated seashell can identify its Pacific or Caribbean origin with 86.3% balanced accuracy, and an anomaly filter keeps the pipeline usable on real uploads.","keywords":["seashell identification","coastal provenance","fine-grained classification","anomaly detection","ecological restoration","BackHome19K dataset","ConvNeXt-Tiny","Pacific and Caribbean coasts"],"falsifier":"Run a species-disjoint split on BackHome19K: hold out every image of a random subset of species for testing and train on the rest; if accuracy on those unseen species falls well below 86.3%, the central generalization claim fails. The paper's own tables make this check possible by listing all 516 species.","tokens_in":1383,"feed_emoji":"🐚","tokens_out":3703,"duration_ms":78661,"temperature":0.7,"pith_summary":"The paper tries to solve a conservation bottleneck: confiscated seashells cannot be returned to the wild because officers cannot tell whether a shell came from Costa Rica's Pacific or Caribbean coast. It introduces a dataset of 19,058 labeled photographs spanning 516 species and a two-stage pipeline—an anomaly filter followed by a lightweight ConvNeXt-Tiny classifier—that predicts coastal provenance. On the held-out test set the classifier reaches 86.3% balanced accuracy, and the filter rejects 93% of 180 out-of-domain images with no seashells lost. If these numbers hold in practice, wildlife officers can sort confiscated shells at scale and repatriate most of them to their native ecosystems.","feed_headline":"Seashell photos reveal Pacific or Caribbean origin 86% of the time","feed_subtitle":"A lightweight two-stage pipeline helps wildlife officers return 70,000 confiscated shells to the right coast.","key_machinery":"The pipeline has two stages. First, an anomaly detector runs each image through SqueezeNet and takes the global-average-pooled activations of the final convolutional layer as a 1000-dimensional embedding; a query is accepted only if its mean cosine similarity to its $k=5$ nearest neighbors is at least $\\lambda = 0.955$. Accepted images then go to a ConvNeXt-Tiny classifier initialized with ImageNet-1K weights, with the stem and first three stages frozen and only the last block unfrozen. The embedding-similarity filter is what makes the deployed system resilient to user-generated noise; the frozen-feature re-learning scheme is what lets a 28-million-parameter network capture fine-grained shell cues without overfitting.","core_discovery":"The central claim is that a photograph of a seashell carries enough subtle morphological information—texture, hue, growth rings, micro-geometry—for a compact convolutional network to distinguish Pacific from Caribbean origin at 86.3% balanced accuracy, even when species from the two coasts look nearly identical. The paper further claims that a PaDiM-inspired embedding filter can screen out non-shell uploads with 93% rejection on 180 out-of-domain images and zero false negatives on 40 shell controls, making the system safe to deploy in a public web tool. Together these results are presented as evidence that automated provenance determination is feasible enough to support real repatriation decisions.","pith_inferences":["This inference goes beyond the paper: a species-disjoint holdout would turn the reported accuracy into a true unseen-species estimate; the released species lists make this test easy to run.","This inference goes beyond the paper: the same two-stage design could be retrained for other coast-pair classification tasks in other countries with distinct shorelines, where habitat-matching decisions face the same data bottleneck.","This inference goes beyond the paper: the reported failure mode of marine debris such as coral and rocks passing the filter suggests that a targeted negative-class dataset could improve the anomaly threshold without sacrificing recall."],"forward_implications":["Wildlife officers can process a confiscated shell in under three seconds per image, turning a storage backlog into a fast triage queue.","Most confiscated shells can be returned to their native coast, reducing the risk of introducing parasites or disrupting local gene pools.","The public release of BackHome19K gives other researchers a coast-labeled benchmark for ecosystem-level, not just species-level, shell classification.","The filter-plus-classifier design shows that anomaly detection can protect a fine-grained classifier from noisy user uploads in a deployed conservation tool.","An 86.3% accuracy rate still leaves roughly one in seven shells misrouted, so the pipeline's output needs human review before repatriation."],"supporting_citations":[{"why":"Supplies the ConvNeXt-Tiny backbone and the pretrained weights the classifier is built on.","marker":"[28]"},{"why":"Provides the patch-distribution embedding logic that the anomaly filter adapts.","marker":"[9]"},{"why":"Supplies the lightweight SqueezeNet feature extractor used to build the image embeddings for the filter.","marker":"[18]"},{"why":"One of the comparison baselines the paper's classifier must beat on the same test set.","marker":"[15]"},{"why":"Another baseline architecture compared on the same test set.","marker":"[38]"},{"why":"The earlier large shell recognition benchmark that BackHome19K extends and contrasts with.","marker":"[49]"},{"why":"Contributes out-of-domain images used to test the anomaly filter's rejection rate.","marker":"[26]"},{"why":"Contributes out-of-domain scene images to the anomaly filter's negative test corpus.","marker":"[50]"},{"why":"Used as pretraining data for the backbone and as a source of out-of-domain test images.","marker":"[10]"}],"fun_headline_variants":["AI tells Pacific from Caribbean seashells with 86% accuracy","Seashell selfies reveal which coast they came from","Photo-based AI reunites confiscated seashells with their home coast","86% accurate: AI pinpoints seashell origin from a single photo","Mobile AI identifies seashell coast of origin in under 3 seconds"],"cache_read_input_tokens":19072,"weakest_assumption_plain":"The evaluation assumes that a photo-level 70/15/15 split, balanced by family, measures performance on confiscated shells the model has never seen; if images of the same species occur in both training and test, the reported 86.3% balanced accuracy can overstate real-world results.","fun_headline_variants_meta":{"raw":{"variants":["AI tells Pacific from Caribbean seashells with 86% accuracy","Seashell selfies reveal which coast they came from","Photo-based AI reunites confiscated seashells with their home coast","86% accurate: AI pinpoints seashell origin from a single photo","Mobile AI identifies seashell coast of origin in under 3 seconds"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000801,"raw_usage":{"total_tokens":3487,"prompt_tokens":878,"completion_tokens":2609,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":494,"completion_tokens_details":{"reasoning_tokens":2518}},"tokens_in":494,"tokens_out":2609,"duration_ms":17577,"temperature":1.0,"reasoning_tokens":2518,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T21:23:02.516495+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a species-disjoint split on BackHome19K: hold out every image of a random subset of species for testing and train on the rest; if accuracy on those unseen species falls well below 86.3%, the central generalization claim fails. The paper's own tables make this check possible by listing all 516 species.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"Supplies the ConvNeXt-Tiny backbone and the pretrained weights the classifier is built on."},{"cited_title":"PaDiM: a patch distribution modeling framework for anomaly detection and localization","cited_arxiv_id":null,"evidence_quote":"Provides the patch-distribution embedding logic that the anomaly filter adapts."},{"cited_title":"Iandola, Song Han, Matthew W","cited_arxiv_id":null,"evidence_quote":"Supplies the lightweight SqueezeNet feature extractor used to build the image embeddings for the filter."},{"cited_title":"Deep residual learning for image recognition","cited_arxiv_id":null,"evidence_quote":"One of the comparison baselines the paper's classifier must beat on the same test set."},{"cited_title":"Mobilenetv2: Inverted residuals and linear bottlenecks","cited_arxiv_id":null,"evidence_quote":"Another baseline architecture compared on the same test set."},{"cited_title":"A shell dataset for shell features extraction and recognition","cited_arxiv_id":null,"evidence_quote":"The earlier large shell recognition benchmark that BackHome19K extends and contrasts with."},{"cited_title":"Lawrence Zitnick, and Piotr Doll´ar","cited_arxiv_id":null,"evidence_quote":"Contributes out-of-domain images used to test the anomaly filter's rejection rate."},{"cited_title":"Places: A 10 million image database for scene recognition","cited_arxiv_id":null,"evidence_quote":"Contributes out-of-domain scene images to the anomaly filter's negative test corpus."}],"review_version":1}