{"id":"94c72ee7-d4a6-40a2-9c92-0c297ebdd56d","arxiv_id":"2507.08236","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":7,"one_line_summary":"The authors report that a Word2Vec-style model on K-means spectrogram tokens classifies BirdCLEF+ 2025 soundscapes in about 6 minutes, reaching a public ROC-AUC of 0.559, far below transfer-learning baselines.","lead":"This competition paper tests fast ways to identify birds and other animals from audio on a strict CPU budget. It shows that a simple sound-token method runs in minutes, but its accuracy trails standard deep learning models.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"STSG's 'viability' claim lacks a token-free ablation on the competition metric: 0.520 private ROC-AUC may be achievable by averaging PCA-reduced spectrogram features without tokenization.","rationale":"I read the paper as an honest competition system description: it releases code, reports CPU timings, and does not hide the low absolute scores. The reader's weakest-assumption finding about the surrogate task is real, but the central claim is anchored in the Kaggle leaderboard results, so the more direct weak point is the absence of a token-free control evaluated on that same metric. The STSG pipeline is essentially PCA-feature averaging plus a learned discrete-code embedding; without removing the discrete stage, the contribution of tokenization is not identifiable. I would keep the reader's CONDITIONAL verdict rather than reject: the paper has value as a baseline and as a speed benchmark, but the 'viability' sentence should either be qualified or supported by the ablation above. The internal v1 F1 discrepancy between Table 9 and Table 10 is a secondary reproducibility issue that should be corrected but is not the main burden on the central claim.","tokens_in":13531,"tokens_out":6650,"duration_ms":76444,"concrete_test":"Construct a no-token control using the same features as STSG v2.1: compute 768-band Mel-spectrograms, apply the same normalization and 128-dim PCA, average the PCA vectors over each 5-second window, and train the same linear classification head from Table 4 under the same protocol used for Table 11. Submit or evaluate this control on the same held-out split or Kaggle leaderboard as STSG v2.1. If the control's macro ROC-AUC is within about 0.02 of 0.559/0.520, the tokenization and skip-gram embedding stage is not load-bearing for the central claim; if STSG is clearly better, the concern is resolved.","verdict_should_be":"UNCHANGED","load_bearing_attack":"The central claim is that STSG demonstrates the viability of fast tokenization approaches with static embeddings, citing 0.559 public / 0.520 private macro ROC-AUC (Table 11). This conclusion is confounded with the non-token feature representation. The STSG classifier is a linear head over averaged per-window embeddings, and the input to that representation is a 768-band Mel-spectrogram reduced by PCA (Sections 3.2.1 and 3.2.2). The only non-token comparison reported, the 'MelSpec Baseline' in Table 9, is scored on the surrogate single-label task (macro F1 0.12 vs. STSG 0.56), not on the competition's macro ROC-AUC. No ablation removes the tokenizer and Word2Vec stage while keeping the same PCA features and linear head on the actual metric. Since a 0.520 private AUC is only modestly above chance, the observed score could be carried by the PCA-spectrogram features plus the linear head, in which case the paper would demonstrate a fast spectrogram-averaging baseline rather than a viable tokenization approach. The timing and code-release evidence are useful, but they do not settle which component is responsible for the result.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"This working note describes the DS@GT BirdCLEF+ 2025 submission, which combines transfer learning from pretrained bioacoustic backbones with a newly proposed lightweight pipeline called Spectrogram Token Skip-Gram (STSG). STSG converts audio into Mel-spectrograms, reduces frames with PCA, quantizes them via Faiss K-means, learns static token embeddings with Word2Vec skip-gram, averages the embeddings over 5-second windows, and trains a linear classifier. The paper reports strong transfer-learning baselines (BirdSetEfficientNetB1: 0.810 public / 0.778 private ROC-AUC; Perch with TFLite: 0.729 / 0.711), and a fast STSG variant with a projected six-minute CPU inference time that reaches 0.559 public / 0.520 private ROC-AUC. The central claim is that this result demonstrates the viability of fast tokenization approaches with static embeddings for bioacoustic classification. Validation is performed on a surrogate single-label whole-recording task over 20 species, and the final scores are from the Kaggle leaderboard.","tokens_in":13771,"tokens_out":4341,"duration_ms":47928,"significance":"If the central claim is supported, the paper would provide a useful data point for CPU-constrained bioacoustic classification: a clustering-plus-static-embedding pipeline that runs in minutes and exceeds chance on a difficult multi-taxon task. The paper has clear strengths: the transfer-learning results are credible and useful baselines; the TFLite speedup for Perch is quantified; the STSG pipeline is described in enough detail to reproduce; hyperparameter sweeps and timing measurements are reported; and the authors release code. I also see no circularity in the evaluation: the STSG embeddings are learned unsupervised, the classification head is trained on surrogate labels, and the leaderboard scores are external. However, the central viability claim is currently confounded with a non-token feature representation (PCA-reduced Mel-spectrogram averaging), and the surrogate validation task is a weak proxy for the competition metric. These issues are load-bearing and require revision.","major_comments":[{"comment":"The central claim that STSG demonstrates the viability of tokenization is confounded by the absence of a token-free ablation on the competition metric. The STSG classifier is a linear head over per-window averaged embeddings, where the input features are PCA-reduced Mel-spectrogram frames (Sections 3.2.1 and 3.2.2). The only non-token comparison, the MelSpec Baseline in Table 9, is evaluated on the surrogate single-label F1 task (0.12 vs. 0.56), not on the competition's macro-averaged ROC-AUC. Because the STSG private score is 0.520, only modestly above chance, the observed result could be carried by the averaged PCA features plus the linear head even if the tokenizer and Word2Vec stage contributed nothing. The authors should add a token-free ablation that keeps the same PCA features, the same averaging, the same linear head, and the same test protocol, and reports the competition metric. Without such an ablation, the paper can claim a fast spectrogram-averaging baseline but not a viable tokenization approach.","section":"§3.2, §4.3, Tables 9-11"},{"comment":"The surrogate validation task is a weak proxy for the actual competition, and all STSG design choices are made on it. The surrogate is a single-label, whole-recording classification task over 20 species, while the competition is multi-label classification of 5-second intervals with macro-averaged ROC-AUC. The paper itself notes that surrogate ROC-AUC saturates near 1.0 for transfer-learning models and therefore uses F1, which is not the competition metric. Hyperparameters such as vocabulary size, PCA dimensions, and Word2Vec settings are tuned on this proxy, so the justification of these choices depends on the proxy's validity. The authors should either validate the proxy on the competition metric (e.g., by showing that surrogate rankings predict leaderboard rankings for several configurations) or explicitly reframe the paper's contribution as a fast CPU baseline rather than a fully validated bioacoustic tokenizer.","section":"§3.3 and Tables 9-10"},{"comment":"There is an unexplained internal inconsistency in the surrogate results: Table 9 reports STSG (v1) macro F1 as 0.45, while Table 10 reports 0.381 for the same model. The micro F1 also differs (0.53 vs. 0.394). If these numbers come from different validation splits or training runs, this must be stated; otherwise the surrogate evaluation is not auditable. This matters because the surrogate is used to justify hyperparameter decisions and to compare STSG variants, so a consistent reporting protocol is needed.","section":"Table 9 vs. Table 10"}],"minor_comments":[{"comment":"The vector_size sweep lists a value of 1028, which appears to be a typo for 1024; the same typo appears in the appendix.","section":"Table 8 and Appendix Table 12"},{"comment":"The caption describes the runtimes as projections for a hypothetical 700-soundscape test set, while the text presents the public and private scores as final leaderboard results. Please clarify which numbers are measured and which are extrapolated, and how the projection was computed.","section":"Table 11 caption"},{"comment":"The temperature scaling description says the teacher probabilities are softened by a squared factor of temperature T=3; the exact formula (e.g., logits divided by T^2 versus T) should be stated explicitly.","section":"§3.2.3"},{"comment":"The discussion of overfitting in the Word2Vec validation curves is qualitative; reporting a single early-stopping criterion or a selected epoch number would make the procedure reproducible.","section":"§4.1, Figure 5b"},{"comment":"The name 'RanaSierraCNN' in Table 1 differs from 'RanaSierraeCNN' in later tables; please standardize.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"This is a LifeCLEF working note, so the bar for novelty is modest, and the transfer-learning baseline results are a useful contribution in themselves. The missing token-free ablation on the competition metric is the main technical weakness; it is fixable within the scope of the paper by adding an experiment or by softening the central claim. The surrogate inconsistency should also be resolved before acceptance."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Colleague,\n\nQuick take: this is a solid, honest system description from the BirdCLEF+ 2025 competition. The transfer-learning part is genuinely useful: they benchmark seven zoo models on CPU, show a ~10x TFLite speedup for Perch, and report public/private ROC-AUC scores. The STSG pipeline — k-means tokenization of mel spectrograms plus Word2Vec skip-gram embeddings, averaged over 5-second windows and fed to a linear head — is new in this combination and is described with enough detail that you could reproduce it. Code is on GitHub. Good.\n\nWhere it gets soft: the abstract calls a 0.559 public / 0.520 private macro ROC-AUC “viable.” That is an overstatement. It beats chance, but it is far below the best transfer baseline (0.81 public), and the paper itself later admits the overall performance is low. Second, there is an internal inconsistency: STSG v1 has macro F1 0.45 in Table 9 but 0.381 in Table 10. Probably a different training run, but they don’t explain it, and that kind of thing undermines trust. Third, and more substantively, the validation narrative rests on a surrogate task that does not match the competition metric. They use a single-label, whole-recording classification over 20 species, and switch to F1 because macro ROC-AUC saturates near 1.0 for the transfer models. Design choices for vocabulary size, PCA dims, and Word2Vec hyperparameters are all made on that proxy.\n\nThe stress-test concern is real but not fatal: there is no token-free ablation on the actual 5-second multi-label metric. The one non-token baseline, MelSpec, gets 0.12 macro F1 on the surrogate versus 0.56 for STSG, which at least suggests the tokenizer helps on that proxy. But it is not the same as showing the 0.52 private AUC depends on the tokenization stage; the PCA-reduced spectrogram features plus the linear head could be carrying the signal on the real task. So the central mechanistic claim — that the tokens, not just the averaged spectrogram features, are what make this work — is not fully established. A simple ablation (average PCA-mel features, skip k-means and word2vec, score on the competition metric) would settle it.\n\nWho is this for? People working on BirdCLEF, or anyone needing a fast, lightweight baseline for constrained bioacoustic classification. It’s a useful reference point, not a breakthrough. I’d send it to a serious referee: it has external leaderboard scores, code, and an honest failure case in the student-teacher experiment. The referee should ask for the internal consistency fix, the ablation or a softened claim, and a clearer statement of the surrogate’s limits.\n\nBest.","headline":"Honest BirdCLEF working note: useful CPU benchmarks and a clean tokenization baseline, but the 'viability' claim overstates a 0.52 private AUC without a token-free ablation on the competition metric.","tokens_in":14327,"tokens_out":3137,"would_cite":false,"duration_ms":33068,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A lightweight pipeline that turns Mel-spectrograms into discrete tokens and learns static skip-gram embeddings from unlabeled soundscapes can classify 206 species in roughly 6 minutes of CPU time, beating chance but remaining far below…","keywords":["bioacoustics","spectrogram tokenization","self-supervised learning","efficient inference","acoustic monitoring","BirdCLEF","skip-gram embeddings","static embeddings"],"falsifier":"Run the STSG v2.1 configuration with 768 Mel bands, 128 PCA components, 16,384 K-means clusters, embedding dimension 1024, context window 80, and negative-sampling exponent 0.0 on the official test set, measuring macro ROC-AUC on 5-second intervals and total CPU time: if the score is not meaningfully above 0.500 or the runtime exceeds the 90-minute deadline on the specified CPU, the paper's viability claim fails.","tokens_in":13305,"feed_emoji":"🐦","tokens_out":15230,"duration_ms":125948,"temperature":0.7,"pith_summary":"This paper claims that bioacoustic soundscapes can be classified under a strict 90-minute CPU-only deadline in two complementary ways: by compiling pre-trained models to run on CPUs, and by a novel pipeline called the Spectrogram Token Skip-Gram (STSG) that turns Mel-spectrograms into discrete tokens and learns static embeddings from unlabeled audio using a skip-gram objective. The authors' key result for STSG is that it achieves a public ROC-AUC of 0.559 and a private 0.520 on the competition's 206-species soundscape classification task while running in a projected 6 minutes, about four times faster than their best transfer-learning baseline, which reached 0.810 public but took about 26 minutes. If this is right, it demonstrates that fast tokenization with static embeddings is a viable route for resource-constrained acoustic monitoring, even though the accuracy gap to pre-trained models remains large.","feed_headline":"Spectrogram tokens classify 206 species in 6 minutes on CPU","feed_subtitle":"K-means tokens plus skip-gram embeddings run in minutes, beating chance but trailing the best pretrained models.","key_machinery":"The central object is the STSG pipeline. Spectrograms are computed with 768 Mel bands at 8 frames per second, normalized, projected with PCA (128 components retain 87% of variance), and quantized by K-means into a 16,384-token codebook. A skip-gram negative-sampling model (embedding dimension 1024, context window 80, uniform negative sampling) is trained on the token sequences and produces a static lookup table; classification averages token embeddings per 5-second window and passes them through a linear layer. The skip-gram objective is what carries the argument: tokens that co-occur near each other in time receive nearby vectors, so the averaged embeddings encode the acoustic content of a window without any labeled data. The paper also relies on model compilation to make pre-trained backbones CPU-viable, which sets the baseline that STSG is compared against.","core_discovery":"The paper's central claim is that treating bioacoustic audio as a sequence of discrete spectrogram tokens, rather than as a spectrogram image, yields a classifier accurate enough to beat chance and fast enough for extreme edge constraints. The STSG pipeline clusters PCA-reduced Mel-spectrogram frames into a 16,384-token vocabulary, learns static skip-gram embeddings from token co-occurrence in unlabeled soundscapes, averages the embeddings over 5-second windows, and runs a linear classification head. On the competition's leaderboard it reports ROC-AUC 0.559 public / 0.520 private, with a projected ~6 minutes of CPU time for 700 minutes of soundscapes, about four times faster than the best-scoring pre-trained baseline (0.810 public) and about three times faster than the compiled pre-trained bird-song model (0.729 public). The authors take this as evidence that fast tokenization with static embeddings is a viable alternative in resource-constrained settings, while acknowledging that STSG trails transfer learning by a wide accuracy margin and that a student-teacher distillation from the stronger model actually hurt performance.","pith_inferences":["The 6-minute CPU runtime is projected rather than measured on the competition instance; an editor's inference is that the 90-minute deadline headroom would allow a more powerful tokenizer or a small transformer to replace the K-means tokens without leaving the intended use case.","The gap between surrogate F1 (around 0.56) and leaderboard ROC-AUC (0.52-0.56) suggests the surrogate task is much easier; a natural extension is to evaluate STSG directly on multi-label 5-second frames, using class-balanced sampling or per-species token adjustments to lift ROC-AUC.","The authors' finding that static embeddings from temporal co-occurrence help classification suggests similar token-plus-embedding pipelines could apply to other under-resourced audio monitoring domains, such as insect or amphibian surveys, where labeled data are scarce.","Since the student-teacher distillation failed, the more promising direction implied but not tested is to use the token vocabulary as a discrete input to a lightweight transformer, or to derive static embeddings from a large pretrained model via the compression approach the authors cite for future work."],"forward_implications":["Pre-trained bioacoustic backbones can be compiled for CPU with roughly a 10x speedup, making transfer learning practical under a 90-minute CPU-only deadline.","The best compiled backbone scores ROC-AUC 0.810 public / 0.778 private, while the compiled bird-song model scores 0.729 / 0.711.","STSG classifies the same test set in a projected 6 minutes at ROC-AUC 0.559 public / 0.520 private, outperforming a plain Mel-spectrogram average (surrogate F1 0.12 vs 0.56) and beating chance.","Hyperparameter choices matter: a 16,384-token vocabulary and PCA-reduced (128-dim) Mel-spectrogram input improved surrogate F1, and lowering the negative-sampling exponent helped when subsampling was less aggressive.","Distilling teacher logits into the STSG student model reduced surrogate F1 from 0.56 to 0.47, so the current embedding space does not yet absorb the teacher's knowledge."],"supporting_citations":[{"why":"defines the 206-species task, the 5-second intervals, and the macro ROC-AUC metric the paper optimizes for.","marker":"[1]"},{"why":"supports the hypothesis that pre-trained bird-song embeddings transfer across domains.","marker":"[3]"},{"why":"extends the transfer claim to novel acoustic environments, motivating the reuse of pre-trained backbones.","marker":"[4]"},{"why":"introduces the skip-gram objective that STSG adapts from word embeddings to spectrogram tokens.","marker":"[10]"},{"why":"provides the Mel-spectrogram representation that the tokenizer operates on.","marker":"[13]"},{"why":"supplies the K-means clustering and nearest-neighbor search used to build the token vocabulary and query it at inference.","marker":"[14]"},{"why":"gives the specific word2vec implementation used to train the skip-gram embeddings.","marker":"[15]"},{"why":"motivates the negative-sampling exponent and the hyperparameter sensitivity the tuning sweeps explore.","marker":"[16]"},{"why":"provides the pretrained BirdSet backbones that yield the best-scoring baseline and the comparison point for STSG.","marker":"[17]"}],"fun_headline_variants":["Spectrogram tokens: fast CPU bioacoustic classification in minutes","Tokenized spectrograms enable 6-minute CPU species classification","STSG: unsupervised token embeddings for rapid bioacoustic ID","K-means tokens and skip-grams for lightweight bird classification"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The surrogate validation task—a single-label, whole-recording classification across 20 hand-picked species—faithfully represents the real multi-label 5-second soundscape task across 206 species, even though scores on the surrogate saturate and the real leaderboard scores are far lower.","fun_headline_variants_meta":{"raw":{"variants":["Spectrogram tokens: fast CPU bioacoustic classification in minutes","Tokenized spectrograms enable 6-minute CPU species classification","STSG: unsupervised token embeddings for rapid bioacoustic ID","K-means tokens and skip-grams for lightweight bird classification"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000229,"raw_usage":{"total_tokens":1571,"prompt_tokens":1130,"completion_tokens":441,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":746,"completion_tokens_details":{"reasoning_tokens":370}},"tokens_in":746,"tokens_out":441,"duration_ms":5449,"temperature":1.0,"reasoning_tokens":370,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T18:22:47.574764+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run the STSG v2.1 configuration with 768 Mel bands, 128 PCA components, 16,384 K-means clusters, embedding dimension 1024, context window 80, and negative-sampling exponent 0.0 on the official test set, measuring macro ROC-AUC on 5-second intervals and total CPU time: if the score is not meaningfully above 0.500 or the runtime exceeds the 90-minute deadline on the specified CPU, the paper's viability claim fails.","supporting_citations":[{"cited_title":null,"cited_arxiv_id":null,"evidence_quote":"defines the 206-species task, the 5-second intervals, and the macro ROC-AUC metric the paper optimizes for."},{"cited_title":"Ghani, T","cited_arxiv_id":null,"evidence_quote":"supports the hypothesis that pre-trained bird-song embeddings transfer across domains."},{"cited_title":"Williams, B","cited_arxiv_id":null,"evidence_quote":"extends the transfer claim to novel acoustic environments, motivating the reuse of pre-trained backbones."},{"cited_title":"Řehůřek, P","cited_arxiv_id":null,"evidence_quote":"gives the specific word2vec implementation used to train the skip-gram embeddings."},{"cited_title":"Caselles-Dupré, F","cited_arxiv_id":null,"evidence_quote":"motivates the negative-sampling exponent and the hyperparameter sensitivity the tuning sweeps explore."}],"review_version":1}