REVIEW 3 major objections 4 minor 39 references
On a low-label soundscape task, frozen bioacoustic embeddings outperform both discrete codec tokens and general-purpose transformer embeddings under the same evaluation protocol.
Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →
T0 review · deepseek-v4-flash
2026-08-02 01:59 UTC pith:UVU4WLOI
load-bearing objection A solid, reproducible BirdCLEF baseline and a useful non-bird prototype trick; the headline claim about token representations is undercut by an asymmetric probe comparison. the 3 major comments →
Can Tokens Compete? Token Representations against Supervised CNN Backbones for BirdCLEF+ 2026
The pith
A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.
Core claim
The paper's central empirical claim is that, on a soundscape classification task with roughly one hour of labeled data, the strongest practical system is an ensemble built on a frozen bioacoustic backbone (Perch v2) probed with MLP heads and extended by a prototypical head that scores non-bird species by cosine difference to averaged positive/negative prototypes. This ensemble reaches a private leaderboard score of 0.936. The paper's comparative claim is that token representations cannot compete under the same protocol: a speech-trained neural codec (WavTokenizer) collapses Perch-embedding cosine similarity to 0.313 against a 0.90 gate, and general-purpose self-supervised transformer encoder
What carries the argument
The prototypical head: for each non-bird species, it averages the frozen Perch embeddings of labeled windows where that species is present (μ+) and absent (μ−), then scores each window by clip(cos(e_t, μ+) − cos(e_t, μ−), 0, 1), merged with the SED output via element-wise maximum. This extends a bird-only backbone to non-bird taxa and adds almost no inference cost. The comparison protocol is equally load-bearing: every encoder is evaluated by fitting a ridge head on frozen embeddings using the focal recordings, which is what determines the token-versus-specialist ranking.
Load-bearing premise
The comparison assumes that evaluating each encoder with a frozen ridge head on the focal recordings—rather than adapting token encoders to the labeled hour—captures what it means for token representations to 'compete'; the codec verdict also rests on a single speech-trained codec.
What would settle it
Re-run the comparison with the labeled soundscape hour used to adapt the token encoders (e.g., LoRA on the 66 labeled files) while keeping the frozen-probe protocol for the specialists; if any token encoder then reaches or exceeds the 0.936 private score, the conclusion that tokens cannot compete collapses.
If this is right
- A frozen Perch v2 backbone probed with MLP heads scores 0.920 on the public leaderboard; adding the trained HGNetV2-B0 SED member raises the blend to 0.929, and the non-bird prototype head lifts the private score to 0.936, all within the 90-minute CPU limit.
- The codec representation fails every predefined gate: round-trip Perch cosine is 0.313 against a 0.90 target, round-trip Precision@1 is 0.09 against 0.80, and focal retrieval Precision@10 is 0.195 against 0.80, so reconstruction-oriented tokens are not competitive for bioacoustic retrieval.
- The offline LOSO ranking of encoders does not predict leaderboard standing—BirdNET overtakes BEATs despite lower offline AUROC—so evaluation protocol matters for transfer conclusions.
- Most SED gains come from frontend and architecture choices (dB-scale mel normalization, higher mel resolution, backbone swap) rather than training tricks; 20+ single-variable submissions show offline AUROC anti-correlated with the public leaderboard.
- If the paper is right, the practical route to improved soundscape models is not discrete tokenization but sequence modeling over full-band 1-D token streams, a direction the authors flag as the natural next step.
Where Pith is reading between the lines
- If the pattern holds, the 'token' research direction for soundscapes should target sequence models over full-band 1-D token streams (128×1 columns) rather than codec reconstruction; the paper's Figure 4 points that way but its own SSAMBA run is inconclusive at 55k clips.
- The one-hour labeled set shifts the winning recipe from self-supervision on focal audio to sample-efficient probing of a specialist backbone; a testable extension is whether the gap persists when the labeled set grows, where adaptive finetuning of transformer encoders may catch up.
- The prototype head's success on non-bird taxa suggests a general recipe: extend any frozen specialist embedding to out-of-vocabulary classes by positive/negative prototype differences, but the paper only demonstrates this for Perch on the 47 non-bird species present in the labels.
- The codec comparison used a single speech-trained codec; a bioacoustic-trained codec (or one with more tokens per second) could plausibly change the outcome, and the paper's per-species table already shows low-frequency vocalizations survive round-trip better.
Editorial analysis
A structured set of objections, weighed in public.
Referee Report
Summary. The paper reports the DS@GT ARC team's entry to BirdCLEF+ 2026. The first contribution is a supervised baseline that ensembles a frozen Perch v2 backbone read out by MLP probes, a trained HGNetV2-B0 SED network, and a non-bird prototypical head, reaching a private leaderboard score of 0.936 at rank 1894 within a 90-minute CPU budget. The second and title contribution is a comparison of token-based representations against this CNN baseline: codec tokens from WavTokenizer-large are evaluated via Perch round-trip and retrieval tasks, and semantic encoders (BEATs, AST, EAT, SSAMBA, plus Perch and BirdNET as anchors) are scored with a ridge probe fit on frozen focal-recording embeddings. The paper concludes that the codec route fails on bioacoustic retrieval and that semantic embeddings transfer only modestly due to domain gap.
Significance. If the comparative claims are established, this is a useful negative result for the BirdCLEF-style low-label soundscape setting: it would show that frozen bioacoustic specialist features plus supervised probes outperform general-purpose codec and transformer token representations without task-specific fine-tuning. The baseline itself is competently engineered, with careful ablations, leaderboard verification, and a public repository; the non-bird prototype head is a particularly interesting sample-efficient idea supported by a large offline gain (macro AP 0.895 vs 0.074). The paper also ships a reproducible experimental record (Appendix A, D, E), which is a genuine strength.
major comments (3)
- [Table 6] The central negative claim about semantic embeddings rests on an asymmetric evaluation protocol. In Table 6, every encoder—including Perch—is scored with a ridge head fit on focal training recordings, whereas the Perch member of the baseline (§4.1) uses MLP probes fit on the 59 fully labeled soundscapes via 5-fold OOF. Appendix C shows that simply changing the probe training set lifts Perch from 0.620 (focal-trained ridge, LOSO) to 0.749 (labeled-soundscape ridge with empirical-Bayes prior), a +0.129 shift that exceeds most of the inter-encoder gaps in Table 6. Thus the conclusion 'semantic embeddings transferred only modestly due to domain gap' is not established: the comparison confounds encoder representation quality with the supervision used for the probe. Please either evaluate all encoders under the labeled-soundscape supervised protocol (e.g., MLP probes with GroupKFold OOF) or re
- [Table 4] The codec conclusion is based on a single neural audio codec, WavTokenizer-large, which the authors note was trained on human speech. The 'predefined success gates' in Table 4 appear to be defined for this experiment rather than drawn from an external benchmark, and no second codec (e.g., EnCodec or SoundStream) or domain-adapted codec is tested. As a result, the claim that 'the codec route failed on bioacoustic retrieval' is too broad. A single speech-trained codec failing on high-frequency bird calls is interesting but does not support a generalization about codec representations as a family. Please add at least one additional codec or substantially weaken the conclusion.
- [§6.1] The geometric analysis supporting the leaderboard ordering is shallow. Figure 3 uses eight hand-selected species, and the kNN accuracy is computed over recording centroids used in the same embedding plot. The text claims 'the leaderboard ordering is partly recoverable' but no confidence intervals, species sampling rationale, or comparison to random centroids is given. This does not block the baseline contribution, but it should be either removed as a supporting claim or replaced with a systematic evaluation over the 206 focal species with statistical error bars.
minor comments (4)
- [Conclusion] Typo: 'brough' should be 'brought'.
- [Table 6] The caption says 'AUROC is the leaderboard-aligned score from the 9-fold LOSO ridge probe' but Table 2 reports 'macro AUROC' for the baseline. It would help to explicitly state that the Table 6 Perch row is the ridge-probe configuration (0.866 public), not the MLP-probe baseline (0.920), to avoid confusion with §4.
- [Figure 2] The caption states 'Public LB scores are shown in green,' but the figure is grayscale in the PDF; if color is used, ensure it is distinguishable in print or add numerical labels.
- [Appendix D, Table 14] The reported 'Full BEATs (teacher)' accuracy of 84.0% seems inconsistent with Table 13, where BEATs linear probe obtains 95.8% on the same ESC-50 task; clarify which setup produces the teacher number and why it is lower.
Circularity Check
Peripheral in-sample prototype evaluation; central benchmark claims are externally grounded
specific steps
-
fitted input called prediction
[Section 4.1 (Non-Bird Prototypical Head) and Section 4.2 (Leaderboard Results)]
"Of the 72 non-bird taxa, 47 appear in the labeled soundscapes, allowing us to construct positive (μ+s) and negative (μ−s) prototypes by averaging the Perch embeddings of all windows where the species is present, and all windows where it is absent, respectively. ... Offline evaluation on the 66 labeled soundscapes shows the prototype substantially outperforms the SED model alone for non-bird taxa (macro AP 0.895 vs. 0.074)."
The prototypes are computed as the mean Perch embedding over every labeled soundscape window for each species, and the reported 'offline evaluation' is on the same 66 labeled soundscapes. The proto score is thus a nearest-centroid rule whose centroids include the very windows being scored, so the 0.895 macro AP is a training-set fit rather than a held-out prediction. It is not independent evidence that the Perch embedding space 'carries useful information for non-bird species.' However, the final leaderboard gain from this head (+0.002 public, +0.011 private) is external, so this circularity is peripheral rather than load-bearing.
full rationale
The paper's main empirical claims are anchored to external benchmarks: the private LB score of 0.936 (Table 2) and ESC-50 screening (Table 13). The token-vs-CNN comparison in Table 6 uses one protocol for all six encoders, including the Perch and BirdNET anchors: 'we fit a ridge head on each encoder's frozen embeddings using the focal training recordings.' Table 2 confirms that the Perch ridge line (0.866) is the same head replicated in Table 6, so the general-encoder deficit is not an artifact of giving the baseline labeled-soundscape MLP probes while denying them to token encoders. The codec verdict rests on explicit external thresholds and Perch as an independent round-trip evaluator, not on a fitted input. The one genuinely circular passage is the offline validation of the non-bird prototypical head, as detailed in the step above; it is a training-set score and is not load-bearing because the head is also validated externally on the leaderboard. Self-citations to the team's prior BirdCLEF notes are narrative, and no uniqueness theorem or ansatz is imported from them. The paper itself notes that offline validation AUROC anti-correlated with the public leaderboard, a calibration limitation, but that is not a circularity. Overall: no significant central circularity; one peripheral in-sample evaluation.
Axiom & Free-Parameter Ledger
free parameters (8)
- Ensemble weight w_Perch =
0.5
- Ridge regularization strength λ =
not reported
- MLP probe hyperparameters =
not reported
- Site/hour prior parameters =
not reported
- Prototype positive/negative embeddings =
derived from labeled windows
- WavTokenizer success gates =
cos>0.90; P@1>0.80; P@10>0.80
- k in kNN class-separability =
10
- LoRA rank =
8
axioms (6)
- domain assumption The Kaggle leaderboard macro-AUROC is a valid and reliable measure of target-task performance.
- domain assumption Frozen Perch v2 embeddings carry transferable structure for non-bird taxa.
- ad hoc to paper WavTokenizer-large is representative of neural audio codecs for this comparison.
- domain assumption A frozen ridge probe is a fair way to rank encoders on the target task.
- ad hoc to paper Success gates in Table 4 are pre-specified.
- standard math GroupKFold/LOSO splits and PaCMAP provide unbiased validation and visualization.
read the original abstract
This paper details the DS@GT ARC team's approach to BirdCLEF+ 2026, multi-label detection of animal vocalizations in soundscapes from the Pantanal wetlands. The 2026 edition adds about an hour of labeled soundscapes, shifting the task toward supervised pipelines fit to the labeled set. First, we build a competitive supervised baseline that ensembles a frozen Perch v2 backbone, a trained HGNetV2-B0 sound-event-detection network, and a non-bird prototypical head, reaching a private leaderboard score of 0.936 at rank 1894 within a 90-minute CPU budget. Second, we ask whether token-based representations can compete, contrasting codec representations from neural audio codecs against semantic representations from foundational embeddings. We compare two bioacoustic specialist models against four token-based encoders trained on AudioSet. The repository for this work can be found at https://github.com/dsgt-arc/birdclef-2026.
Figures
Reference graph
Works this paper leans on
-
[1]
Overview of
Picek, Lukas and Adam, Luk. Overview of. International Conference of the Cross-Language Evaluation Forum for European Languages (CLEF) , year=
-
[2]
Overview of
Kahl, Stefan and Denton, Tom and Sugai, Larissa and Piatti, Litiana and Arruda Moreno, Wener Hugo and Barbosa, Mariana Motti and Eibl, Maximilian and Fieker, Carolline Zatta and Garcia, Carolina Martins and Hokama Sousa, Daiene Louveira and J. Overview of. Working Notes of CLEF 2026 -- Conference and Labs of the Evaluation Forum , year=
2026
-
[3]
Advances in neural information processing systems , volume=
wav2vec 2.0: A framework for self-supervised learning of speech representations , author=. Advances in neural information processing systems , volume=
-
[4]
Overview of
Ca. Overview of. Working
-
[5]
Chen, Sanyuan and Wu, Yu and Wang, Chengyi and Liu, Shujie and Tompkins, Daniel and Chen, Zhuo and Che, Wanxiang and Yu, Xiangzhan and Wei, Furu , year = 2023, series =. Proceedings of the 40th. 2212.09058 , pages =
Pith/arXiv arXiv 2023
-
[6]
Proceedings of the
Chen, Wenxi and Liang, Yuzhe and Ma, Ziyang and Zheng, Zhisheng and Chen, Xie , year = 2024, eprint =. Proceedings of the
2024
-
[7]
D. High. 2210.13438 , archiveprefix =
-
[8]
2021 , eprint=
Improving Bird Classification with Unsupervised Sound Separation , author=. 2021 , eprint=
2021
-
[9]
Dosovitskiy, Alexey and Beyer, Lucas and Kolesnikov, Alexander and Weissenborn, Dirk and Zhai, Xiaohua and Unterthiner, Thomas and Dehghani, Mostafa and Minderer, Matthias and Heigold, Georg and Gelly, Sylvain and Uszkoreit, Jakob and Houlsby, Neil , year = 2021, eprint =. An. International
2021
-
[10]
and Ellis, Daniel P
Gemmeke, Jort F. and Ellis, Daniel P. W. and Freedman, Dylan and Jansen, Aren and Lawrence, Wade and Moore, R. Channing and Plakal, Manoj and Ritter, Marvin , year = 2017, month = mar, pages =. Audio. 2017
2017
-
[11]
Scientific Reports , volume=
Global birdsong embeddings enable superior transfer learning for bioacoustic classification , author=. Scientific Reports , volume=. 2023 , publisher=
2023
-
[12]
Mind the
Liang, Jinhua and Nolasco, Ines and Ghani, Burooj and Phan, Huy and Benetos, Emmanouil and Stowell, Dan , year = 2024, month = mar, eprint =. Mind the
2024
-
[13]
doi:10.21437/Interspeech.2021-698 , archiveprefix =
Interspeech 2021 , author =. doi:10.21437/Interspeech.2021-698 , archiveprefix =. 2104.01778 , pages =
Pith/arXiv arXiv 2021
-
[14]
Gu, Albert and Dao, Tri , year = 2023, month = dec, eprint =. Mamba:
2023
-
[15]
Hagiwara, Masato , year = 2022, month = oct, eprint =
2022
- [16]
-
[17]
and Kennard, Robert W
Hoerl, Arthur E. and Kennard, Robert W. , year = 1970, month = feb, journal =. Ridge
1970
-
[18]
and Shen, Yelong and Wallis, Phillip and
Hu, Edward J. and Shen, Yelong and Wallis, Phillip and. International. 2106.09685 , archiveprefix =
-
[19]
Ji, Shengpeng and Jiang, Ziyue and Wang, Wen and Chen, Yifu and Fang, Minghui and Zuo, Jialong and Yang, Qian and Cheng, Xize and Wang, Zehan and Li, Ruiqi and Zhang, Ziang and Yang, Xiaoda and Huang, Rongjie and Jiang, Yidi and Chen, Qian and Zheng, Siqi and Zhao, Zhou , year = 2024, month = aug, eprint =
2024
-
[20]
and Eibl, Maximilian and Klinck, Holger , year = 2021, month = mar, journal =
Kahl, Stefan and Wood, Connor M. and Eibl, Maximilian and Klinck, Holger , year = 2021, month = mar, journal =
2021
-
[21]
Overview of
Kahl, Stefan and Navine, Amanda and Denton, Tom and Klinck, Holger and Hart, Patrick and Glotin, Herv. Overview of. Proceedings of the
-
[22]
Overview of
Kahl, Stefan and Denton, Tom and Klinck, Holger and Reers, Hendrik and Cherutich, Francis and Glotin, Herv. Overview of. Working
-
[23]
Overview of
Kahl, Stefan and Denton, Tom and Klinck, Holger and Ramesh, Vijay and Joshi, Viral and Srivathsa, Meghana and Anand, Akshay and Arvind, Chiti and Cp, Harikrishnan and Sawant, Suyash and Vv, Robin and Glotin, Herv. Overview of. Working
-
[24]
Liu, Zhuang and Mao, Hanzi and Wu, Chao-Yuan and Feichtenhofer, Christoph and Darrell, Trevor and Xie, Saining , year = 2022, eprint =. A. Proceedings of the. doi:10.1109/CVPR52688.2022.01167 , archiveprefix =
arXiv 2022
-
[25]
Efficient
Mikolov, Tomas and Chen, Kai and Corrado, Greg and Dean, Jeffrey , year = 2013, eprint =. Efficient. International
2013
-
[26]
Miyaguchi, Anthony and Yu, Jiangyue and Cheungvivatpant, Bryan and Dudley, Dakota and Swain, Aniketh , editor =. Motif. Proceedings of the. 2206.04805 , pages =
-
[27]
Miyaguchi, Anthony and Zhong, Nathan and Gustineli, Murilo and Hayduk, Chris , editor =. Transfer. Working. 2306.16760 , pages =
-
[28]
Miyaguchi, Anthony and Cheung, Adrian and Gustineli, Murilo and Kim, Ashley , editor =. Transfer. Working. 2407.06291 , pages =
-
[29]
Miyaguchi, Anthony and Gustineli, Murilo and Cheung, Adrian , editor =. Distilling. Working. 2507.08236 , pages =
-
[30]
, year = 2015, month = oct, pages =
Piczak, Karol J. , year = 2015, month = oct, pages =. Proceedings of the 23rd
2015
-
[31]
Ramsauer, Hubert and Sch. Hopfield. International. 2008.02217 , archiveprefix =
Pith/arXiv arXiv 2008
-
[32]
Rauch, Lukas and Heinrich, Ren. Can. 2504.12880 , archiveprefix =
-
[33]
Rauch, Lukas and Schwinger, Raphael and Wirth, Moritz and Heinrich, Ren. The. 2403.10380 , publisher =
-
[34]
Foundation
Schwinger, Raphael and Zadeh, Paria Vali and Rauch, Lukas and Kurz, Mats and Hauschild, Tom and Lapp, Sam and Tomforde, Sven , year = 2025, month = aug, eprint =. Foundation
2025
-
[35]
Shams, Siavash and Dindar, Sukru Samet and Jiang, Xilin and Mesgarani, Nima , year = 2024, month = dec, pages =
2024
-
[36]
Advances in neural information processing systems , volume=
Prototypical networks for few-shot learning , author=. Advances in neural information processing systems , volume=
-
[37]
Tan, Mingxing and Le, Quoc V. , year = 2019, series =. Proceedings of the 36th. 1905.11946 , pages =
Pith/arXiv arXiv 2019
-
[38]
Tang, Chengyu and Baskiyar, Sanjeev , year = 2025, month = dec, eprint =. State
2025
-
[39]
Zeghidour, Neil and Luebs, Alejandro and Omran, Ahmed and Skoglund, Jan and Tagliasacchi, Marco , year = 2021, month = jul, eprint =
2021
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.