{"id":"f8c8dd9d-e124-4c57-9545-db1bd3f59db1","arxiv_id":"2412.03633","paper_version":4,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":1,"one_line_summary":"The NBM dataset provides precise time and frequency annotations for nocturnal bird calls and a Faster R-CNN-style detector that reaches mAP 0.67 and near-BirdNet classification accuracy.","lead":"This paper presents a new open dataset of 13,359 manually annotated nighttime bird calls from 117 European species, plus a deep learning detector that locates calls in spectrograms. It matters because it gives researchers a fine-grained resource for monitoring nocturnal migration and counting birds by call.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The mAP and BirdNet comparison are only meaningful if NBM-XC training files and the XC test files are disjoint; the paper reports no deduplication check, so test-set overlap could inflate both headline numbers.","rationale":"I read the paper as making two central claims: the NBM dataset is a first-of-its-kind fine-grained resource, and the NBM detector is competitive with BirdNet despite much less training data. The first claim is supported by the dataset release and annotation counts; the second depends on the evaluation. The evaluation setup has two XC-derived components: NBM-XC training files and a test set drawn from XC. Without deduplication, the mAP numbers cannot be interpreted as generalization performance. This is not a disagreement about the value of the method; it is a missing control in the experiment. The reader identified exactly this point, and I agree. The 270/271 file-count discrepancy reinforces the need for a precise manifest. My recommended verdict remains CONDITIONAL, unchanged from the reader's, since the concern is concrete but testable and the dataset contribution is independent of the model's exact score.","tokens_in":11419,"tokens_out":2864,"duration_ms":27482,"concrete_test":"Check the Zenodo metadata and the model's training manifest for Xeno-Canto recording IDs; intersect the IDs of the NBM-XC training files against the 270/271 test files. If IDs do not overlap, additionally compute an audio fingerprint (e.g., spectrogram hashing or cross-correlation after resampling) to catch near-duplicate segments, which can occur when multiple XC recordings capture the same individual or event. Retrain and re-evaluate on a strictly disjoint subset and compare mAP@IoU0.5 and multi-label mAP; if either drops materially (e.g., >0.05), the headline competitive results are inflated by overlap.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.3 states the test set is composed of 'Six audio files... extracted for each species from XC and manually annotated, for a total of 270 test files,' while Section 4.2.1 refers to '271 hand-annotated XC recordings.' Section 3.1 describes supplementing the crowd-sourced base with manually annotated samples 'from the Xeno-Canto database' (the NBM-XC component). The paper never reports a provenance or deduplication check between the NBM-XC training files and the test files, even though both are drawn from the same source. If the same XC recording, or a segment of it, appears in both sets, the object-detection mAP@IoU0.5 of 0.67 and the aggregated multi-label mAP of 0.71 would be inflated by instance-level memorization rather than species-level generalization. The comparison with BirdNet in Section 4.2.2 is separately acknowledged to be potentially affected by test set overlap with BirdNet's own XC training base. The load-bearing assumption is therefore that the two XC-derived splits are effectively disjoint; the text provides no evidence for it.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper introduces the NBM (Nocturnal Bird Migration) dataset, a crowd-sourced collection of 13,359 time-frequency annotated vocalizations from 117 Western Palearctic bird species, supplemented by manually annotated Xeno-Canto recordings. It also presents a two-stage Faster R-CNN object detection model with self-attention and positional encodings for localizing bird calls on spectrograms. The model achieves an mAP@IoU0.5 of 0.67 on the NBM test set, and an aggregated multi-label mAP of 0.71 when compared with BirdNet's 0.78 on the same 45-species scope. All data and code are released openly.","tokens_in":11629,"tokens_out":5572,"duration_ms":51908,"significance":"If the overlap concern is resolved, the NBM dataset is a valuable, first-of-its-kind fine-grained benchmark for nocturnal bird call detection in the Western Palearctic. The authors' decision to release audio, annotations, and code under open licenses is a concrete strength, as is the use of a held-out, manually annotated test set as a baseline for future work. The model result is a useful proof of concept that object detection on spectrograms can localize short flight calls; the BirdNet comparison, while qualified by the authors, is meaningful enough to motivate further study. The main uncertainties concern the provenance of the test set and the statistical robustness of the reported metrics, not the internal logic of the dataset construction.","major_comments":[{"comment":"The paper never specifies whether the NBM-XC training files and the XC test files are disjoint, and the test-set count is inconsistent: Section 3.3 states six files per species for a total of 270 test files, while Section 4.2.1 refers to 271 hand-annotated XC recordings. Since both sets are drawn from Xeno-Canto, identical or near-identical recordings in both splits could inflate both the object-detection mAP@IoU0.5 (0.67) and the aggregated multi-label mAP (0.71) through instance-level memorization rather than species-level generalization. Please provide the full list of XC recording IDs used for training and testing, report any overlap, and either exclude overlapping recordings or quantify the effect of doing so. This is load-bearing for the headline numbers.","section":"Section 3.3 and 4.2.1"},{"comment":"All reported metrics are single-run point estimates without confidence intervals or variance across training runs. With only six test files per species, the per-species AP values in Table 1 are highly variable (e.g., 0.26 for Pluvialis apricaria and 0.31 for Emberiza citrinella), and the 0.07 mAP gap to BirdNet is within the range one would expect from random seed variation. Reporting the mean and standard deviation over several training runs, or bootstrap confidence intervals over test files, is necessary to support the claim of competitive accuracy.","section":"Section 4.2.1 and Table 1"}],"minor_comments":[{"comment":"The concluding sentence of Section 4.2.2 calls the results 'highly competitive' even though the preceding paragraph acknowledges a possible overlap with BirdNet's XC training base; I suggest softening this to 'indicative' in the abstract and conclusion as well.","section":"Section 4.2.2"},{"comment":"The statement that 'all p(f) taking the same value' is ambiguous; please state explicitly that a uniform prior over frequency is assumed in the posterior calculation shown in Figure 4.","section":"Section 4.1, Eq. (1)"},{"comment":"The description of the manual annotation review would benefit from a quantitative reliability measure, such as the number of annotators per file or an inter-annotator agreement estimate.","section":"Section 3.1"},{"comment":"Reference [39] is cited for the launch of the NBM project, but the cited article is a general study of participatory science data and does not describe the NBM project; please replace it with the project website or a publication that actually documents the initiative.","section":"References"}],"recommendation":"major_revision","confidential_remarks":"The dataset contribution is genuine and the open release is a strength. The main barrier is the unresolved provenance between the NBM-XC training component and the XC test set; this is fixable with a recording-ID manifest and a deduplication check. Please also ensure the authors provide variance estimates or bootstrapped intervals for the headline metrics."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The NBM dataset is a real contribution: 13,359 time- and frequency-localized annotations across 117 Western Palearctic species, with data and code openly released. That alone justifies a look. The model work is a solid application rather than a new paradigm, and the comparison with BirdNet is informative even if not apples-to-apples. The paper is honest about BirdNet's possible XC training overlap, but it misses the same problem on its own side: the NBM-XC training files and the test files are both drawn from Xeno-Canto, and no deduplication or provenance check is reported. If the same recordings appear in both splits, the mAP of 0.67 and the multi-label mAP of 0.71 could be inflated by instance-level memorization. The 270/271 test-file discrepancy reinforces the need for a careful cleanup. Single-run metrics without confidence intervals are a lesser issue for a baseline dataset paper.\n\nWhat is actually new: the first time-frequency annotated nocturnal bird call collection for the Western Palearctic at this scale. The positional encoding analysis in Figure 4 is a nice sanity check, not a circular prediction; it shows the model uses frequency location as intended. The authors also deserve credit for discussing species-level variance and covariate shift.\n\nThe soft spots are real but fixable. The load-bearing one is the train/test overlap. A reviewer can test this directly by comparing recording IDs or fingerprints between the NBM-XC training files and the test files. If they are disjoint, the headline numbers stand. If not, the authors need to re-split or re-report. The test-set count should be reconciled.\n\nWho is this for? Bioacousticians and conservation researchers who need a fine-grained benchmark for nocturnal flight calls, and ML researchers working on sound event detection. It deserves a serious referee. The dataset is open, the code is available, and the overlap concern is empirically checkable. My recommendation: send it to peer review, with the proviso that the authors add a provenance/deduplication analysis and fix the test-set inconsistency before the performance claims are taken at face value.","headline":"A genuinely useful open dataset for nocturnal bird acoustic monitoring; the performance claims need a train/test provenance check before they can be taken at face value.","tokens_in":12127,"tokens_out":2805,"would_cite":false,"duration_ms":26386,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A fine-grained dataset of 13,359 nocturnal bird calls lets a small object-detection model nearly match BirdNet's accuracy on 45 migratory species.","keywords":["nocturnal bird migration","passive acoustic monitoring","object detection","spectrogram analysis","bioacoustics dataset","Faster R-CNN","Western Palearctic","BirdNet comparison"],"falsifier":"A concrete check would be to compute audio fingerprints (e.g., spectral hashes) for all NBM-XC training files and all 270 test files and look for matches above a high similarity threshold; if any test file shares content with a training file, the reported mAP of 0.67 and the BirdNet comparison would need to be recomputed on a deduplicated split.","tokens_in":11232,"feed_emoji":"🐦","tokens_out":2869,"duration_ms":28241,"temperature":0.7,"pith_summary":"This paper introduces the Nocturnal Bird Migration (NBM) dataset, the first collection of its kind for Western Palearctic birds, containing 13,359 precise time-and-frequency annotations of vocalizations from 117 species. The authors argue that such fine-grained annotation enables a fundamentally different approach to bird sound recognition: treating calls as objects to be localized on a spectrogram, rather than classifying whole audio windows. To prove the dataset's value, they train a two-stage object-detection model that localizes calls in time and frequency. On a multi-label classification task over 45 species, this model reaches a mean average precision of 0.71, close to the 0.78 of BirdNet, a system trained on a vastly larger weakly-annotated corpus. If this holds, it suggests that careful, medium-scale annotation efforts can rival massive weakly-labeled datasets, and that object detection on spectrograms is a viable path for monitoring nocturnal migration.","feed_headline":"13,359 night bird calls become a detection benchmark","feed_subtitle":"Precise time-frequency labels let a small model rival BirdNet's 0.78 mAP with 0.71, without millions of recordings.","key_machinery":"The load-bearing machinery is the reformulation of bird call recognition as object detection on spectrograms, carried by a two-stage Faster R-CNN architecture with an EfficientNetV2 backbone and a feature pyramid network. Two adaptations are specific to sound: a self-attention module that lets local predictions draw on the broader acoustic context, and sinusoidal positional encodings added to region-of-interest features, with an absolute encoding along the frequency axis and a relative encoding along the time axis. These encodings break the spatial invariance typical of image detectors, letting the model associate fundamental frequency with species identity. On the data side, the annotation protocol—marking the smallest identifiable call or song phrase in both time and frequency using Audacity—is what makes fine-grained detection possible.","core_discovery":"The central claim is that a crowd-sourced, precisely annotated dataset of nocturnal migratory bird calls, combined with an object-detection architecture adapted for sound, can localize individual calls in both time and frequency and compete with state-of-the-art multi-label classifiers trained on orders of magnitude more data. The NBM dataset contains 13,359 annotations across 2,077 files totalling almost 38 hours, assembled by volunteers across France and supplemented with manually annotated Xeno-Canto recordings to balance species coverage. The detection model, built on a Faster R-CNN backbone with a feature pyramid network, self-attention, and frequency-aware positional encodings, achieves a detection mAP@IoU0.5 of 0.67 on a 271-file test set and a multi-label mAP of 0.71 versus BirdNet's 0.78. The authors further show that the positional encoding causes the model to learn sensible frequency priors for each species, with predicted posterior frequency distributions aligning with training data, confirming that frequency location is informative for identification.","pith_inferences":["If the approach generalizes beyond the 45 well-represented species, passive acoustic monitoring could yield species-level counts of nocturnal migration at continental scale, complementing radar-based passage estimates.","The alignment between learned frequency posteriors and training-set frequency distributions offers a simple diagnostic: a model whose posterior drifts from expected band could flag out-of-distribution recordings or recording-device differences.","The paper does not report any deduplication or provenance check between the Xeno-Canto-derived training files and the 270 test files, both sourced from Xeno-Canto; if near-identical recordings appear in both sets, the reported mAP and the BirdNet comparison could be inflated.","A natural ablation study would test whether the self-attention and positional-encoding modules each contribute independently; such an ablation could also clarify when object detection outperforms sliding-window classification for other bioacoustic datasets."],"forward_implications":["Object detection on spectrograms can enumerate individual calls, opening the door to counting birds in a migrating flock rather than only detecting species presence.","A medium-scale, precisely annotated dataset of about 306 events per species can achieve multi-label accuracy close to BirdNet's, supporting the value of targeted annotation over sheer data volume.","The NBM test set provides a reusable, versioned benchmark for nocturnal bird call detection in the Western Palearctic, allowing future models to be compared on the same 45 species.","The demonstrated benefit of frequency-aware positional encodings suggests that other sound-event detection tasks where pitch or frequency position is discriminative could adopt similar architectural choices."],"supporting_citations":[{"why":"Xeno-Canto is the source of the supplementary NBM-XC training files and the 270 test recordings, making it the dataset's main external audio supplier.","marker":"[28]"},{"why":"BirdNet is the state-of-the-art multi-label classifier that the NBM model is compared against on the shared 45-species scope.","marker":"[24]"},{"why":"Faster R-CNN provides the two-stage region-proposal detection framework that the NBM model is built upon.","marker":"[40]"},{"why":"Feature Pyramid Network supplies the multi-scale feature hierarchy used to detect calls of varying duration and bandwidth.","marker":"[41]"},{"why":"EfficientNetV2 is the backbone network that extracts spectrogram features for the detector.","marker":"[59]"},{"why":"The self-attention mechanism is borrowed from transformers to let local predictions use broader acoustic context.","marker":"[62]"},{"why":"Shrestha et al. is cited as the rare prior work that formulates bird sound recognition as object detection with Faster R-CNN on spectrograms.","marker":"[38]"},{"why":"BirdDB introduces the 'phrase' concept of minimal identifiable vocalization segments, which the NBM annotation protocol explicitly resembles.","marker":"[45]"}],"fun_headline_variants":["13k precise night calls open a new acoustic benchmark","Precise audio labels let small model beat giants","Open dataset of night calls rivals BirdNet with fewer samples","Crowd-sourced bird audio with pin-point labels rivals big datasets","Open benchmark: 13,359 labeled night calls"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the Xeno-Canto-derived training recordings and the Xeno-Canto-derived test recordings are effectively disjoint, since the paper reports no check for duplicated or near-duplicated audio between the two sets.","fun_headline_variants_meta":{"raw":{"variants":["13k precise night calls open a new acoustic benchmark","Precise audio labels let small model beat giants","Open dataset of night calls rivals BirdNet with fewer samples","Crowd-sourced bird audio with pin-point labels rivals big datasets","Open benchmark: 13,359 labeled night calls"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000766,"raw_usage":{"total_tokens":3397,"prompt_tokens":946,"completion_tokens":2451,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":562,"completion_tokens_details":{"reasoning_tokens":2370}},"tokens_in":562,"tokens_out":2451,"duration_ms":17762,"temperature":1.0,"reasoning_tokens":2370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-11T22:14:53.483977+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"A concrete check would be to compute audio fingerprints (e.g., spectral hashes) for all NBM-XC training files and all 270 test files and look for matches above a high similarity threshold; if any test file shares content with a training file, the reported mAP of 0.67 and the BirdNet comparison would need to be recomputed on a deduplicated split.","supporting_citations":[{"cited_title":"https://xeno-canto.org","cited_arxiv_id":null,"evidence_quote":"Xeno-Canto is the source of the supplementary NBM-XC training files and the 270 test recordings, making it the dataset's main external audio supplier."},{"cited_title":"Birdnet: A deep learning so- lution for avian diversity monitoring.Ecological Informatics, 61:101236, 2021","cited_arxiv_id":null,"evidence_quote":"BirdNet is the state-of-the-art multi-label classifier that the NBM model is compared against on the shared 45-species scope."},{"cited_title":"Faster r-cnn: Towards real-time object detection with region proposal networks","cited_arxiv_id":null,"evidence_quote":"Faster R-CNN provides the two-stage region-proposal detection framework that the NBM model is built upon."},{"cited_title":"Feature pyramid networks for object detection","cited_arxiv_id":null,"evidence_quote":"Feature Pyramid Network supplies the multi-scale feature hierarchy used to detect calls of varying duration and bandwidth."},{"cited_title":"Efficientnetv2: Smaller models and faster training","cited_arxiv_id":null,"evidence_quote":"EfficientNetV2 is the backbone network that extracts spectrogram features for the detector."},{"cited_title":"Bird audio diarization with faster r-cnn","cited_arxiv_id":null,"evidence_quote":"Shrestha et al. is cited as the rare prior work that formulates bird sound recognition as object detection with Faster R-CNN on spectrograms."},{"cited_title":"Bird-db: A database for annotated bird song sequences.Ecological Infor- matics, 27:21–25, 2015","cited_arxiv_id":null,"evidence_quote":"BirdDB introduces the 'phrase' concept of minimal identifiable vocalization segments, which the NBM annotation protocol explicitly resembles."}],"review_version":1}