{"id":"975cefc0-2096-4998-92b2-23a8c2a399e2","arxiv_id":"2504.18650","paper_version":1,"verdict":"CONDITIONAL","confidence":"HIGH","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":6,"one_line_summary":"An unsupervised pipeline using autoencoders plus hierarchical clustering flags out-of-species sounds in Xeno-Canto bird recordings, but accuracy varies strongly across species.","lead":"This paper tests whether unsupervised machine learning can spot mislabeled bird recordings in the crowd-sourced Xeno-Canto database, so researchers can clean their training data. It compares three neural-net compression methods on 50 bird species and finds performance varies widely by species.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"All reported outlier-precision numbers rest on one non-expert human's subjective labels, and no null baseline or base noise rate is measured, so the label-noise-reduction claim is not yet established.","rationale":"The reader's weakest assumption is exactly the single-human subjective ground truth in Section 10, and my reading of the paper agrees that this is the load-bearing point. All performance numbers in Table 4 are filtered through the author's auditory and spectrographic judgment, with acknowledged room for discretion. Moreover, the metric labeled TPR is precision, and no base noise rate or random-discard baseline is provided, so even a reliable precision value would not by itself quantify 'meaningful reduction' in label noise. The paper does have real strengths: it uses actual Xeno-Canto data, provides code, reports honest caveats, and identifies plausible factors (diversity, dataset size, entropy, outlier density) that correlate with performance. But those strengths do not compensate for the absence of independent label validation. My proposed test would settle the concern: expert agreement on a stratified sample would either confirm or refute the reported numbers. Since the reader already judged the paper CONDITIONAL with high confidence on this basis, my stress-test does not move the verdict; it sharpens the condition that must be met before the central claim can be accepted.","tokens_in":16674,"tokens_out":2763,"duration_ms":29101,"concrete_test":"Select a random sample of roughly 200 flagged and 200 unflagged clips from each of five species spanning the precision range (e.g., OCWA, BEWR, CAKI, SOSP, RBSA). Have two independent expert bird audio labelers (not the author) classify each clip as target species sound, non-target sound, or indeterminate, using a written protocol. Compute Cohen's kappa between the experts and between each expert and the author, then recompute precision and the estimated noise reduction using only clips where the experts agree. If kappa is below 0.6 or the recomputed precision differs from Table 4 by more than 0.1, the reported estimates are not reliable and the central claim is not supported.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim that the pipeline 'can meaningfully reduce label noise' depends entirely on the evaluation in Sections 10 and 11. What is called 'TPR' in Table 4 is actually precision: the fraction of clips flagged as outliers that the author, a self-described birding enthusiast rather than an expert, judged to be true non-target sounds. Section 10 explicitly notes 'ample opportunity for judgement in practice' and gives only soft rules. If these labels are unreliable or systematically biased (e.g., toward confirming the method), every reported number changes and the claim has no quantitative support. Even if the labels were accepted, the paper does not measure the base outlier rate in the original datasets or compare against a null baseline such as randomly discarding the same fraction of clips. Without either, a precision of 0.6 among flagged clips does not demonstrate a 'meaningful' reduction in label noise, especially for the many species with precision below 0.4. The few inlier-class samples (Sections 11.2, 11.4) are limited to five species, so false negatives and overall noise reduction remain unquantified for most of the dataset. Thus the weakest link is not the machinery but the evidence connecting flagged-clip precision to the claimed outcome.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents an unsupervised pipeline for reducing label noise in bird audio datasets derived from Xeno-Canto. Audio recordings are segmented, converted to 32×40 mel-spectrograms, screened by SINR, and then embedded with one of three neural dimensionality-reduction methods: convolutional autoencoder (CAE), convolutional variational autoencoder (CVAE), or variational deep embedding (VaDE). Outlier candidates are identified per model by hierarchical agglomerative clustering (for CAE/CVAE) or GMM membership probability (for VaDE), and an ensemble of models votes by majority to flag clips. The method is applied separately to 50 Southern California bird species. Performance is assessed by manually inspecting sampled flagged clips and the fraction judged to be true non-target sounds is reported as 'TPR' (Table 4). The authors find large variation across species and attribute this to sound diversity, dataset size, high-entropy sound types, and outlier density. They conclude that the pipeline can meaningfully reduce label noise, with CVAE/CAE ensembles outperforming VaDE.","tokens_in":16921,"tokens_out":2754,"duration_ms":28946,"significance":"If the central claim survives scrutiny, this is a useful practical contribution to an applied ML problem: cleaning crowdsourced audio datasets for bird classification. The study is unusually broad (50 species, three model families), reports confidence margins, and provides open-source code on GitHub. The strengths are the reproducible pipeline, the explicit treatment of preprocessing choices, and the honest reporting of performance variability across species. The weakness is that the quantitative evidence for label-noise reduction is presently incomplete: the reported metric is not a true positive rate, no null baseline or base noise rate is measured, and the ground-truth labels come from a single non-expert observer. These issues are fixable but require additional analysis, so the significance is conditional.","major_comments":[{"comment":"The quantity reported as 'TPR' in Table 4 is not a true positive rate; it is the fraction of sampled flagged clips that the author judged to be true outliers, i.e., precision at the fixed discard level D. Section 9 says 'TPR provides an appropriate performance metric for our application because the goal is to detect as many of the non-target bird species sounds as possible,' but the reported numbers do not measure detection rate at all. Without a false-negative estimate, the numbers cannot support the claim that the method 'detects' outliers; they only characterize the purity of the flagged set. The authors do sample the inlier class for a few species (Sections 11.2 and 11.4), but these are explicitly exceptional. The metric should be renamed (e.g., 'outlier precision') throughout, and the interpretation of Table 4 and the abstract's 'meaningfully reduce' claim should be revised accordingly.","section":"Section 9 and Table 4"},{"comment":"The central claim that the pipeline 'can meaningfully reduce the label noise' is not supported without a comparison baseline. The paper never measures the base outlier rate in the post-preprocessed datasets, nor does it compare against a null baseline such as randomly discarding the same number of clips. A precision of 0.6 among flagged clips is evidence of noise reduction only if the base outlier rate is below 0.6; for species where the best precision is 0.2–0.4, the flagged set could be less clean than the original dataset. The authors should either estimate the base outlier rate from a random sample of the original (unflagged) data or simulate random discards and show that the ensemble-flagged set has significantly higher outlier concentration. Without such a comparison, the abstract's quantitative claim is ungrounded.","section":"Section 11 and Conclusions"},{"comment":"The entire performance evaluation rests on the author's manual labeling of a sample of flagged clips, and the paper explicitly acknowledges 'ample opportunity for judgement in practice' with only two soft decision rules. The author is described as 'a birding enthusiast but not an expert in identifying birds by ear.' No second observer, no inter-annotator agreement measure, and no comparison against expert-verified labels is provided. Since every reported precision value in Table 4 is conditional on this labeling, the reliability of the ground truth is load-bearing. At minimum, the authors should report a random subset of clips independently labeled by a second observer (or by the author at a different time) and compute agreement; if that is not feasible, the conclusion should be explicitly downgraded to a proof-of-concept with anecdotal validation.","section":"Section 10"},{"comment":"The four 'cause factors' are assigned post hoc and the paper itself admits 'This study did not yield a quantitative assessment of the extent to which each of these factors influenced the outlier detection performance' (Section 11). As presented, the diversity categories are defined by the author's subjective reading of Birds of the World, and the cause codes in Table 4 are selected by looking at the results. This is acceptable as hypothesis generation but should not be framed as findings. Please clearly label this subsection as qualitative, exploratory analysis, and consider adding a simple quantitative check (e.g., correlation between entropy and best precision) if the claims are to be retained.","section":"Section 11 (factors 1–4)"}],"minor_comments":[{"comment":"The 'signal-to-interference and noise ratio (SINR)' is never formally defined. Please provide the exact formula used for segment SINR and for the 75th-percentile screening.","section":"Section 5"},{"comment":"The entry for HOWR CVAE MoE shows '####' instead of a numeric value. This appears to be a missing or corrupted data point and should be corrected or annotated.","section":"Table 4"},{"comment":"The text says 'Figure 5 shows the best TPR performance...' but the reference should be to Figure 6. Check the figure numbering throughout the manuscript.","section":"Section 11.4 and Figure 6"},{"comment":"The text states RBSA has 'only 227 clips' while Table 4 lists the number of clips as 226. These should be reconciled.","section":"Section 11.2"},{"comment":"Several typographical issues: Section 6 'clustering effectiveness to be synonyms with' should be 'synonymous with'; Section 7 'the concatenated HAC' is likely 'the HAC combined with the learned representation'. The reference list also has inconsistent formatting (e.g., 'S. M. Billerman' entry lacks proper title formatting).","section":"Various"}],"recommendation":"major_revision","confidential_remarks":null},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Quick take: this is a genuinely useful applied study—someone actually ran CAE, CVAE, and VaDE with HAC-based outlier voting across 50 Xeno-Canto species and posted code—but the evidence section doesn't yet support the abstract's claim of “meaningfully reduce label noise.” The reported TPR is precision, the ground truth is one birder's subjective judgment with “ample opportunity for judgment in practice,” and there is no null baseline or direct measurement of noise reduction. The machinery is standard; the contribution is the large-scale comparison and the honest reporting of species variability.\n\nWhat's new: the 50-species comparison of three unsupervised dimensionality-reduction approaches for flagging mislabeled clips is not in the cited literature. The paper also documents failure modes—high-entropy call types, small data size, outlier density—with concrete examples, and it clearly labels its diversity categories as subjective. Code is on GitHub. That's real value for practitioners.\n\nSoft spots, in order of severity. First, the metric problem is not cosmetic. Table 4 calls the number TPR but it is precision: the fraction of flagged clips the author judged to be true non-target sounds. TPR requires knowing the actual outlier count, which they do not. Section 10's rules are soft, and the author is a self-described non-expert; a systematic bias toward confirming flags would inflate every number. Second, no baseline. Randomly discarding the same fraction of clips would likely also remove some outliers, and without that comparison “meaningful reduction” is unsupported. Third, inlier-class sampling is limited to five species, so false negatives and net noise reduction remain unquantified for most species. The cause-factor analysis is post hoc and largely qualitative—fine as hypothesis generation, not conclusion.\n\nOn the positive side, margins of error are reported, the limitations of the sampling are openly discussed, and the conclusion recommends CVAE+HAC with appropriate caveats. The central claim may survive better evaluation, but the current paper does not establish it.\n\nWho it is for: researchers building bird-audio training sets from Xeno-Canto, and anyone comparing UOD methods on real-world noisy audio. A revised version with corrected metrics, a random-discard baseline, and ideally a downstream classifier comparison would be worth publishing.\n\nVerdict: worth engaging. Not a desk reject—the empirical scope is substantial and the author is honest about limits. But a serious referee would require major revision before acceptance.","headline":"Useful applied comparison of UOD methods for cleaning bird audio labels, but the evaluation rests on one non-expert human's labels and a mislabeled metric, so the central claim is not yet established.","tokens_in":17425,"tokens_out":1558,"would_cite":false,"duration_ms":16185,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Unsupervised autoencoder ensembles can flag a large share of mislabeled bird-audio clips, though success varies by species.","keywords":["unsupervised outlier detection","label noise","bird audio","spectrogram","convolutional autoencoder","variational deep embedding","hierarchical clustering","ensemble learning"],"falsifier":"Have several expert birders independently label the same flagged clips and recompute the true-positive rates; if the experts' labels disagree substantially with the author's decisions, or if the estimated true-positive rate changes by more than the reported margins, the claim that the pipeline meaningfully reduces label noise would need revision. A second check would be to train identical classifiers on cleaned and uncleaned versions of the same species and compare held-out accuracy on independently verified recordings.","tokens_in":16453,"feed_emoji":"🐦","tokens_out":6975,"duration_ms":66233,"temperature":0.7,"pith_summary":"The paper argues that label noise in bird-audio training sets can be reduced without supervision by a cleaning pipeline: split recordings into spectrogram clips, screen them by signal-to-interference ratio, embed the clips in a low-dimensional space with a convolutional autoencoder or a variational-deep-embedding model, then flag clusters far from the main species sounds as outliers. Using 50 species drawn from a crowd-sourced audio repository, the author shows that flagged clips are often genuine non-target sounds, so discarding them removes some mislabeled training examples. The effect is real but uneven: estimated true-positive rates for flagged outliers range from near zero to nearly one, with simpler convolutional autoencoder ensembles doing as well as or better than the more complex variational deep embedding on average. A careful reader should care because this offers an automated way to clean large bioacoustics datasets where manual screening is impractical.","feed_headline":"Bird-audio label noise drops under unsupervised outlier voting","feed_subtitle":"Embedding clips into a latent space and voting across models flags recordings that do not match their species label.","key_machinery":"The load-bearing object is the low-dimensional latent space that each single-species model learns from spectrogram clips, together with the cluster geometry in that space. After preprocessing clips into 32-by-40 mel-spectrograms and screening by signal-to-interference ratio, each model compresses a species' clips into a latent embedding; hierarchical agglomerative clustering with average linkage then partitions the latent points into flat clusters, and clusters that sit far from the large 'big clusters'—ordered by distance and size up to a discard budget—are marked as candidate outliers. Because individual models learn unstable representations, the pipeline repeats this on several models and keeps only clips flagged by the majority. This machinery carries the argument because it converts 'is this clip from the labeled species?' into a geometric question about where a point lies relative to the dense parts of the species' own sound manifold.","core_discovery":"The paper's central claim is that a per-species unsupervised cleaning process can flag a meaningful share of the clips in a bird-audio dataset that do not come from the labeled species. For each of 50 species, the author trains multiple instances of three unsupervised dimensionality-reduction models—convolutional autoencoder, convolutional variational autoencoder, and variational deep embedding—on spectrogram clips; flags candidate outliers by hierarchical clustering in the latent space, or by low Gaussian-mixture membership probability for the variational deep embedding; and keeps clips that a majority of models flag. Sampling the flagged clips shows estimated true-positive rates that are often high, sometimes above 0.9, but that vary widely across species and models. The author attributes this variation to four factors: the diversity of the target species' sounds, low training-data quantity, high-entropy vocalizations that resist compact representation, and the underlying density of true outliers. A further finding is that the simpler convolutional autoencoder and variational autoencoder ensembles outperform the more complex variational deep embedding ensembles on average while being cheaper and more stable to train.","pith_inferences":["The reported true-positive rates are precision on flagged clips, not recall; a practical workflow would still need to decide, per species, how many clips to discard and would need downstream validation that discarded clips actually improve classifier accuracy.","The four explanatory factors could be turned into a screening rule: before running the pipeline, estimate a species' sound diversity and the prevalence of high-entropy vocalizations from metadata or spectrogram statistics, and skip or tune species where cleaning is unlikely to help.","The same one-label-per-recording structure appears throughout bioacoustics, so the pipeline should transfer to other crowd-sourced animal-sound collections; a direct test would be to run it on species not used in this study and compare cleaned versus uncleaned classifier performance."],"forward_implications":["A practitioner can apply the same preprocessing-plus-ensemble recipe to other bird species and obtain flagged clips that are substantially enriched in true non-target sounds, so dropping them should reduce label noise in downstream classifiers.","The simpler convolutional autoencoder and variational autoencoder ensembles are the safer default: they matched or beat the variational deep embedding on average and avoided its numerical instability below roughly 1,000 clips.","Species with very diverse vocal repertoires, especially call types with little tonal structure, will see weaker cleaning; the paper recommends treating each species as its own outlier-detection problem rather than expecting one global result.","Because the discard budget caps how many clips can be removed, the true-positive rate is measured on the outlier class only, which makes comparisons across ensembles fair only when their flagged sets are similar in size."],"supporting_citations":[{"why":"Supplies the variational deep embedding method whose Gaussian-mixture cluster-membership probabilities score how anomalous each clip is.","marker":"(Jiang, 2017)"},{"why":"Provides the hierarchical agglomerative clustering algorithm and linkage definitions used to cut the latent space into flat clusters.","marker":"(Nielsen, 2016)"},{"why":"Motivates why autoencoder reconstruction error alone is unreliable when outliers are present in the training data, supporting the choice to cluster the latent space instead.","marker":"(Abhaya, 2022)"},{"why":"Stands behind the claim that unsupervised model selection is unsolved, which is why the paper resorts to human sampling to estimate true-positive rates.","marker":"(Ma, 2023)"},{"why":"Provides the sound reference used by the human evaluator to decide whether a flagged clip is a legitimate species sound or a true outlier.","marker":"(Pieplow, 2019)"},{"why":"Supplies the majority-voting ensemble combination used to stabilize outlier flags across multiple trained models.","marker":"(Sun, 2022)"}],"fun_headline_variants":["Unsupervised voting flags mislabeled bird calls","Autoencoder outlier scoring cuts bird audio label noise","Cleaning bird audio labels with unsupervised outlier detection","Outlier voting reduces label noise in bird audio datasets","Unsupervised models spot noisy labels in bird recordings"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The entire accuracy assessment rests on one person's auditory and spectrographic judgments about whether a flagged clip is truly a non-target sound, and that person is a birding enthusiast rather than an expert, leaving ample room for inconsistent labels.","fun_headline_variants_meta":{"raw":{"variants":["Unsupervised voting flags mislabeled bird calls","Autoencoder outlier scoring cuts bird audio label noise","Cleaning bird audio labels with unsupervised outlier detection","Outlier voting reduces label noise in bird audio datasets","Unsupervised models spot noisy labels in bird recordings"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000854,"raw_usage":{"total_tokens":3740,"prompt_tokens":1004,"completion_tokens":2736,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":620,"completion_tokens_details":{"reasoning_tokens":2663}},"tokens_in":620,"tokens_out":2736,"duration_ms":21129,"temperature":1.0,"reasoning_tokens":2663,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-16T10:12:29.447526+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Have several expert birders independently label the same flagged clips and recompute the true-positive rates; if the experts' labels disagree substantially with the author's decisions, or if the estimated true-positive rate changes by more than the reported margins, the claim that the pipeline meaningfully reduces label noise would need revision. A second check would be to train identical classifiers on cleaned and uncleaned versions of the same species and compare held-out accuracy on independently verified recordings.","supporting_citations":[],"review_version":1}