Pith. sign in

REVIEW 5 major objections 5 minor 1 cited by

The ICME 2025 Audio Encoder Capability Challenge

T0 review · 5 major / 5 minor · reviewed 2026-08-10 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗ Review JSON

Pith's one-line read The paper announces a two-track competition that ranks continuous audio encoders on speech, environmental, and music tasks, including private real-world datasets, using a test-set-size-weighted scoring rule.

desk verdict Useful challenge design undercut by a scoring formula that rewards worse performance on lower-is-better metrics like EER and MSE. read the letter →

arxiv 2501.15302 v1 pith:GXYI32H7 submitted 2025-01-25 cs.SD eess.AS

Junbo Zhang , Heinrich Dinkel , Qiong Song , Helen Wang , Yadong Niu , Si Cheng , Xiaofeng Xin , Ke Li
show 3 more authors
Wenwu Wang Yujun Wang Jian Luan
This is my paper · ORCID
classification cs.SDeess.AS
keywords audioencoderchallengecontinuousembeddingsrepresentationlearningbenchmarklinearprobingk-nearest-neighborevaluationreal-worldtaskstest-set-size-weightedscoringmulti-domain
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

This is a competition-design paper rather than a technical result. The authors propose a benchmark challenge that evaluates pre-trained continuous audio encoders on a diverse suite of public tasks spanning speech, environmental sound, and music, supplemented by six private datasets from real-world industrial scenarios. Its central claim is that this design, with its two evaluation tracks and its scoring rule, can rank encoders on both adaptability and intrinsic representational quality, thereby advancing audio encoder design. A sympathetic reader would care because such a standardized comparison is currently missing: existing benchmarks cover only parts of the domain or focus on discrete tokens, and continuous embeddings are what multimodal systems actually consume.

What carries the argument

The scoring identity is the weighted normalized total: for each task $i$, the raw metric $M_i$ is mapped to $(M_i - M^{\min}_i) / (M^{\max}_i - M^{\min}_i)$ using theoretical worst and best values, and the track score is the sum of test-set size $n_i$ times the normalized metric divided by the total test-set size, so larger test sets count more. This weighted normalization, together with the two-track structure and the hidden real-world datasets, is what the argument hangs on.

What would settle it

Compute the proposed weighted score for two published encoders (e.g., wav2vec2 and Dasheng) using their known accuracies on these datasets: if the ranking reverses when the normalization bounds are changed from theoretical min/max to observed min/max of a baseline, or if a random-feature encoder lands within the noise band of the top models, the benchmark is not discriminative.

Watch

Extended reading notes

Core claim

The paper's proposal is that a single audio encoder should be judged by two complementary operations on its fixed embeddings: a linear fine-tuning probe (Track A) that measures how quickly task-specific classifiers can adapt the representation, and a K-nearest-neighbour evaluation (Track B) that measures raw representational quality without any training. Scores on each track are computed by normalizing each task metric to a 0–1 scale, then taking a test-set-size-weighted average across tasks; participants are ranked on separate leaderboards for the two tracks. The protocol deliberately spans three domains (speech, environmental sound, and music) and includes six withheld datasets that mirror industrial conditions such as car interior sounds, finger-snap wake-words, and subway broadcasts, so that the ranking reflects real-world usability, not just academic dataset performance.

Load-bearing premise

The ranking's usefulness rests on the unstated premise that the selected public tasks plus six hidden datasets are representative of real-world audio processing, and that the test-set-size-weighted average of normalized scores yields a meaningful, stable ordering of encoders; the paper offers no baseline results showing this is so.

Editorial extensions

If this is right

  • If the challenge runs as specified, the open-source evaluation system will give participants a shared API and a reusable pipeline for comparing continuous audio encoders across all three domains.
  • The two-track design will let the community observe whether linear-probe performance and KNN performance track each other, or whether some encoders excel at one but not the other.
  • The six hidden datasets will provide a transfer test that public-task fine-tuning cannot capture, since the public Table 1 datasets are excluded from training.
  • The final leaderboards will be the first benchmark to rank continuous encoders on a common weighted scale covering speech, sound, and music simultaneously.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • Because weights are test-set sizes, the final score will be dominated by a handful of large test sets; tasks with few test samples may contribute little even if they test a distinct capability, so the ranking may not reflect breadth of skill.
  • The theoretical min–max normalization assumes each metric has meaningful worst and best bounds; if one task is much noisier than others, its normalized scores will compress differences and dilute that task's effect despite its test-set weight.
  • A natural extension would apply the same protocol to discrete-token encoders and compare the two families on the same ranking; the paper hints at this by citing the discrete-audio benchmark but does not run that comparison.
  • If organizers released per-task scores, later analysis could determine whether the challenge's ranking is stable under different normalization choices (e.g., using a baseline model's observed min/max instead of theoretical bounds), a test the paper does not report.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

5 major / 5 minor

Summary. Zhang et al. describe the ICME 2025 Audio Encoder Capability Challenge, in which participants submit pre-trained continuous audio encoders through a fixed API. The encoders are evaluated on a broad set of public datasets spanning speech, environmental sound, and music, plus six hidden industrial datasets. Track A trains a linear probe on frozen embeddings using predefined hyperparameters; Track B applies KNN classification directly to the embeddings. Rankings are computed as a test-set-size-weighted average of per-task metrics normalized to [0,1] via Eq. (1). The paper is a challenge/protocol announcement: it contains no experimental evaluations, baselines, or variance analyses.

Significance. If the protocol is executed as intended, the challenge would fill a useful gap by providing a comparative benchmark for continuous audio embeddings across speech, environmental audio, and music, complementing HEAR, SUPERB, and DASB. The open-source evaluation system, the fixed model API, the two-track design, and the inclusion of real-world hidden datasets are genuine strengths. However, the stated scoring formula is incorrect for lower-is-better metrics, which directly affects the central ranking mechanism, and the paper gives no baseline evidence that the proposed weighted average is stable or discriminative. The challenge idea is valuable but the protocol as written needs correction and validation.

major comments (5)
  1. [§4.1, Eq. (1)] The normalization in Eq. (1) is directionally wrong for lower-is-better metrics. For EER, lower values are better, yet §4.1 sets M_min=0 and M_max=1, so Eq. (1) maps a perfect EER of 0 to 0 and a useless EER of 1 to 1. Under the final score in Eq. (2), a system with worse spoofing detection receives a higher contribution for the ASV2015 task, in both Track A and Track B. The same inversion applies to MSE on speechocean762. The formula must be corrected, for example by assigning M_min as the worst possible value and M_max as the best possible value so that lower-is-better metrics are transformed appropriately, or by explicitly inverting those metrics before applying Eq. (1).
  2. [§4.1, Table 1] The normalization for the speechocean762 MSE task is under-specified even apart from direction: MSE has no canonical [0,1] range, and no M_min/M_max values are given for it. Without those values, Eq. (2) cannot be computed for that task, and if M_max is intended to be 1, the inversion problem above applies. Please state the assumed range or a standardizing transformation for MSE.
  3. [§1.1 and Table 1] Track B is described as evaluating 'a subset of Track A's utterance-level classification tasks', but Table 1 marks ASV2015 (spoofing detection, EER) and DESED (sound event detection, segment-F1) as included in Track B. These are not utterance-level classification tasks, and the paper does not explain how KNN classification would be applied to them or how EER and segment-F1 would be derived from KNN outputs. The Track B protocol also omits the K value, the distance metric, and the feature pooling strategy for variable-length inputs; these details are necessary for a reproducible ranking.
  4. [§4.2, Tables 1–2] The weights n_i in Eq. (2) are the test-set sizes, but the paper never lists test-set sizes for the public datasets, and for hidden datasets Table 2 reports only total sample sizes or durations (e.g., 125 hours for subway broadcast), not the number of test examples used in scoring. Without a precise definition of n_i for every task, the final score is not reproducible and the relative weighting between tasks remains unspecified.
  5. [§1 and §4.2] The manuscript provides no baseline evaluations, so the central assumption that the test-set-size-weighted average in Eq. (2) yields a stable and discriminative ranking is unvalidated. I recommend adding results for at least two reference encoders (e.g., wav2vec2 and Dasheng) on all public tasks, including per-task normalized metrics, final scores, and a sensitivity analysis such as leaving out one dataset at a time to check whether any single large test set dominates the ranking.
minor comments (5)
  1. [Abstract and §5] There are several typos and formatting artifacts, including 'en coders', 'machie', 'us large audio', 'incorpeating', and 'pipline'; the organizer biography section also contains promotional language that should be trimmed.
  2. [Table 1] The table would be clearer if each task indicated whether its metric is higher-is-better or lower-is-better; this would have made the EER and MSE normalization issue apparent.
  3. [References [25]] The table uses the name 'ASV2015' but the reference is the ASVspoof 2015 database; align the naming for consistency.
  4. [§3] The computational constraints are described as 'Google Colab T4 environment, where the runtime is equipped with a 16 GB NVIDIA Tesla T4 GPU, 12GB RAM'; please clarify whether the 12GB RAM refers to system RAM and whether the 16GB is GPU memory, since these limits directly affect what models can be submitted.
  5. [§4.1] The definitions of M_min and M_max are stated as 'worst and best possible values', which is only consistent with higher-is-better metrics; please rephrase to cover both metric directions explicitly.

Circularity Check

0 steps flagged · score 0.0 of 10

No circularity: the paper describes an evaluation protocol with no derivation or fitted parameter whose output is equivalent to its inputs.

full rationale

This paper is a challenge description rather than a derivation or empirical study. It introduces no fitted parameters and makes no predictions from data. The only equations, Eq. (1) for metric normalization and Eq. (2) for the final weighted score, are explicitly stated definitions of the evaluation protocol; they are not claimed to be derived from anything else, so there is no self-definitional or fitted-input-called-prediction pattern. Citations to prior work are contextual rather than load-bearing: [12] is cited as one example of a strong continuous audio encoder, and [24] is cited as one of the public datasets used in the challenge; both are external resources and are not invoked to justify the challenge's ranking scheme. No uniqueness theorem or ansatz is imported from the authors' prior work, and no known result is renamed as a new contribution. The claim that the challenge 'provides a platform for evaluating and advancing the state-of-the-art in audio encoder design' is a statement of intended purpose, not a result that could reduce to its own inputs. Any concern about whether Eq. (1) handles lower-is-better metrics such as EER or MSE in the intended direction would be an internal consistency or correctness issue, not circular reasoning.

Assumptions & free parameters 0 free parameters · 1 assumptions · 0 invented entities

The paper does not introduce any fitted parameters or new entities. The only significant prior assumption is that the task and dataset selection is representative and that hidden datasets are contamination-free.

assumptions (1)
  • domain assumption The chosen public and hidden datasets are representative of real-world audio processing tasks.
    The benchmark's usefulness depends on this assumption, but the paper provides no evidence to support it.

how reviews work

0 comments
Cite this review

Pith. "Pith review of The ICME 2025 Audio Encoder Capability Challenge." pith.science (2026). https://pith.science/paper/GXYI32H7

@misc{pith2026250115302,
  author       = {Pith},
  title        = {Pith review of: The ICME 2025 Audio Encoder Capability Challenge},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/GXYI32H7}},
  note         = {Machine review of arXiv:2501.15302}
}
read the original abstract

This challenge aims to evaluate the capabilities of audio encoders, especially in the context of multi-task learning and real-world applications. Participants are invited to submit pre-trained audio encoders that map raw waveforms to continuous embeddings. These encoders will be tested across diverse tasks including speech, environmental sounds, and music, with a focus on real-world usability. The challenge features two tracks: Track A for parameterized evaluation, and Track B for parameter-free evaluation. This challenge provides a platform for evaluating and advancing the state-of-the-art in audio encoder design.

Discussion (0). Continue with ORCID to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. OpenBEATs: A Fully Open-Source General-Purpose Audio Encoder

    cs.SD 2025-07 conditional novelty 6.0 of 10

    OpenBEATs releases the BEATs audio pretraining pipeline, trains 300M-parameter models on 20k hours of multi-domain audio, and reports strong results across 25 datasets, including bioacoustics and reasoning tasks.

Reference graph

Works this paper leans on

35 extracted references · 22 canonical work pages · cited by 1 Pith paper

  1. [1]

    Neural discrete representation learning,

    A. Van Den Oord, O. Vinyals et al. , “Neural discrete representation learning,” Advances in neural information processing systems , vol. 30, 2017. 5

    work page 2017

  2. [2]

    Finit e scalar quantization: Vq-vae made simple,

    F. Mentzer, D. Minnen, E. Agustsson, and M. Tschannen, “Finit e scalar quantization: Vq-vae made simple,” arXiv preprint arXiv:2309.15505 , 2023

    arXiv 2023

  3. [3]

    High-fidelity audio compression with improved rvqgan,

    R. Kumar, P. Seetharaman, A. Luebs, I. Kumar, and K. Kumar, “High-fidelity audio compression with improved rvqgan,” Advances in Neural Information Processing Systems , vol. 36, 2024

    work page 2024

  4. [4]

    Snac: M ulti-scale neural audio codec,

    H. Siuzdak, F. Gr¨ otschla, and L. A. Lanzend¨ orfer, “Snac: M ulti-scale neural audio codec,” arXiv preprint arXiv:2410.14411, 2024

    arXiv 2024

  5. [5]

    Moshi: a speech-text foundation model for real-time dialogue,

    A. D´ efossez, L. Mazar´ e, M. Orsini, A. Royer, P. P´ erez, H. J ´ egou, E. Grave, and N. Zeghidour, “Moshi: a speech-text foundation model for real-time dialogue,” Te ch. Rep., 2024. [Online]. Available: https://arxiv.org/abs/2410.00037

    arXiv 2024

  6. [6]

    A comparative s tudy of discrete speech tokens for semantic-related tasks with large language models,

    D. Wang, M. Cui, D. Yang, X. Chen, and H. Meng, “A comparative s tudy of discrete speech tokens for semantic-related tasks with large language models,” arXiv preprint arXiv:2411.08742 , 2024

    arXiv 2024

  7. [7]

    Qwen-audio: Ad- vancing universal audio understanding via unified large-scale audio- language models,

    Y. Chu, J. Xu, X. Zhou, Q. Yang, S. Zhang, Z. Yan, C. Zhou, and J. Zhou, “Qwen-audio: Ad- vancing universal audio understanding via unified large-scale audio- language models,” arXiv preprint arXiv:2311.07919, 2023

    arXiv 2023

  8. [8]

    Mini-omni: Language models can hear, talk while th inking in streaming,

    Z. Xie and C. Wu, “Mini-omni: Language models can hear, talk while th inking in streaming,” arXiv preprint arXiv:2408.16725, 2024

    arXiv 2024

Show all 35 references
  1. [9]

    Salmonn-omni: A codec-free llm for full-duplex speech understand ing and generation,

    W. Yu, S. Wang, X. Yang, X. Chen, X. Tian, J. Zhang, G. Sun, L. L u, Y. Wang, and C. Zhang, “Salmonn-omni: A codec-free llm for full-duplex speech understand ing and generation,” arXiv preprint arXiv:2411.18138, 2024

    2024 arXiv

  2. [10]

    wav2vec 2.0: A fr amework for self-supervised learning of speech representations,

    A. Baevski, Y. Zhou, A. Mohamed, and M. Auli, “wav2vec 2.0: A fr amework for self-supervised learning of speech representations,” Advances in neural information processing systems , vol. 33, pp. 12 449– 12 460, 2020

    2020

  3. [11]

    Data2 vec: A general framework for self- supervised learning in speech, vision and language,

    A. Baevski, W.-N. Hsu, Q. Xu, A. Babu, J. Gu, and M. Auli, “Data2 vec: A general framework for self- supervised learning in speech, vision and language,” in International Conference on Machine Learning . PMLR, 2022, pp. 1298–1312

    2022

  4. [12]

    Sc aling up masked audio encoder learning for general audio classification,

    H. Dinkel, Z. Yan, Y. Wang, J. Zhang, Y. Wang, and B. Wang, “Sc aling up masked audio encoder learning for general audio classification,” in Interspeech 2024, 2024

    2024

  5. [13]

    HEAR: Holistic evaluation of audio representations,

    J. Turian, J. Shier, H. R. Khan, B. Raj, B. W. Schuller, C. J. Ste inmetz, C. Malloy, G. Tzanetakis, G. Velarde, K. McNally et al. , “HEAR: Holistic evaluation of audio representations,” in NeurIPS 2021 Competitions and Demonstrations Track . PMLR, 2022, pp. 125–145

    2021

  6. [14]

    SUPERB: Speech processing universal performance benchmar k,

    S.-w. Yang, P.-H. Chi, Y.-S. Chuang, C.-I. J. Lai, K. Lakhotia, Y. Y. Lin, A. T. Liu, J. Shi, X. Chang, G.-T. Lin et al. , “SUPERB: Speech processing universal performance benchmar k,” Interspeech 2021, 2021

    2021

  7. [15]

    DASB–discrete audio and speech benchmark,

    P. Mousavi, L. Della Libera, J. Duret, A. Ploujnikov, C. Subakan , and M. Ravanelli, “DASB–discrete audio and speech benchmark,” arXiv preprint arXiv:2406.14294 , 2024

    2024 arXiv

  8. [16]

    Speech commands: A dataset for limited-vocabula ry speech recognition,

    P. Warden, “Speech commands: A dataset for limited-vocabula ry speech recognition,” arXiv preprint arXiv:1804.03209, 2018

    2018 arXiv

  9. [17]

    Lib ricount, a dataset for speaker count estima- tion,

    F.-R. St¨ oter, S. Chakrabarty, E. Habets, and B. Edler, “Lib ricount, a dataset for speaker count estima- tion,” 2018

    2018

  10. [18]

    Voxlingua107: a dataset for spoken lan guage recognition,

    J. Valk and T. Alum¨ ae, “Voxlingua107: a dataset for spoken lan guage recognition,” in 2021 IEEE Spoken Language Technology Workshop (SLT) . IEEE, 2021, pp. 652–658

    2021

  11. [19]

    Voxceleb: L arge-scale speaker verification in the wild,

    A. Nagrani, J. S. Chung, W. Xie, and A. Zisserman, “Voxceleb: L arge-scale speaker verification in the wild,” Computer Speech & Language , vol. 60, p. 101027, 2020. 6

    2020

  12. [20]

    Librisp eech: an asr corpus based on public domain audio books,

    V. Panayotov, G. Chen, D. Povey, and S. Khudanpur, “Librisp eech: an asr corpus based on public domain audio books,” in 2015 IEEE international conference on acoustics, speech an d signal processing (ICASSP). IEEE, 2015, pp. 5206–5210

    2015

  13. [21]

    Speech model pre-training for end-to-end spoken language understanding,

    L. Lugosch, M. Ravanelli, P. Ignoto, V. S. Tomar, and Y. Bengio , “Speech model pre-training for end-to-end spoken language understanding,” arXiv preprint arXiv:1904.03670 , 2019

    1904 arXiv

  14. [22]

    Vocalsound: A dataset for impro ving human vocal sounds recognition,

    Y. Gong, J. Yu, and J. Glass, “Vocalsound: A dataset for impro ving human vocal sounds recognition,” in ICASSP 2022 - 2022 IEEE International Conference on Acousti cs, Speech and Signal Processing (ICASSP), 2022, pp. 151–155

    2022

  15. [23]

    Crema-d: Crowd- sourced emotional multimodal actors dataset,

    H. Cao, D. G. Cooper, M. K. Keutmann, R. C. Gur, A. Nenkova, and R. Verma, “Crema-d: Crowd- sourced emotional multimodal actors dataset,” IEEE transactions on affective computing , vol. 5, no. 4, pp. 377–390, 2014

    2014

  16. [24]

    spee- chocean762: An open-source non-native english speech corpus f or pronunciation assessment,

    J. Zhang, Z. Zhang, Y. Wang, Z. Yan, Q. Song, Y. Huang, K. Li, D. Povey, and Y. Wang, “spee- chocean762: An open-source non-native english speech corpus f or pronunciation assessment,” in Proc. Interspeech 2021, 2021

    2021

  17. [25]

    Autom atic speaker verification spoofing and countermeasures challenge (asvspoof 2015) database,

    T. Kinnunen, Z. Wu, E. Nicholas Evans, and J. Yamagishi, “Autom atic speaker verification spoofing and countermeasures challenge (asvspoof 2015) database,” 201 8

    2015

  18. [26]

    Esc: Dataset for environmental sound classific ation,

    K. J. Piczak, “Esc: Dataset for environmental sound classific ation,” in Proceedings of the 23rd ACM international conference on Multimedia , 2015, pp. 1015–1018

    2015

  19. [27]

    Fsd 50k: an open dataset of human-labeled sound events,

    E. Fonseca, X. Favory, J. Pons, F. Font, and X. Serra, “Fsd 50k: an open dataset of human-labeled sound events,” IEEE/ACM Transactions on Audio, Speech, and Language Proce ssing, vol. 30, pp. 829–852, 2021

    2021

  20. [28]

    A dataset and taxonom y for urban sound research,

    J. Salamon, C. Jacoby, and J. P. Bello, “A dataset and taxonom y for urban sound research,” in Pro- ceedings of the 22nd ACM international conference on Multim edia, 2014, pp. 1041–1044

    2014

  21. [29]

    Sound eve nt detection in domestic environments with weakly labeled data and soundscape synthesis,

    N. Turpault, R. Serizel, A. P. Shah, and J. Salamon, “Sound eve nt detection in domestic environments with weakly labeled data and soundscape synthesis,” in Workshop on Detection and Classification of Acoustic Scenes and Events , 2019

    2019

  22. [30]

    General-purpose tag- ging of freesound audio with audioset labels: Task description, data set, and baseline,

    E. Fonseca, M. Plakal, F. Font, D. P. Ellis, X. Favory, J. Pons, a nd X. Serra, “General-purpose tag- ging of freesound audio with audioset labels: Task description, data set, and baseline,” arXiv preprint arXiv:1807.09902, 2018

    2018 arXiv

  23. [31]

    Clotho: An audio capt ioning dataset,

    K. Drossos, S. Lipping, and T. Virtanen, “Clotho: An audio capt ioning dataset,” in ICASSP 2020-2020 IEEE International Conference on Acoustics, Speech and Sig nal Processing (ICASSP) . IEEE, 2020, pp. 736–740

    2020

  24. [32]

    Enabling factorized piano music modeling and generation with the MAESTRO dataset,

    C. Hawthorne, A. Stasyuk, A. Roberts, I. Simon, C.-Z. A. Hua ng, S. Dieleman, E. Elsen, J. Engel, and D. Eck, “Enabling factorized piano music modeling and generation with the MAESTRO dataset,” in International Conference on Learning Representations , 2019

    2019

  25. [33]

    The gtzan dataset: Its contents, its faults, t heir effects on evaluation, and its future use,

    B. L. Sturm, “The gtzan dataset: Its contents, its faults, t heir effects on evaluation, and its future use,” arXiv preprint arXiv:1306.1461 , 2013

    2013 arXiv

  26. [34]

    Neural audio synthesis of musical notes with wavenet autoencoders,

    J. Engel, C. Resnick, A. Roberts, S. Dieleman, D. Eck, K. Simony an, and M. Norouzi, “Neural audio synthesis of musical notes with wavenet autoencoders,” 2017

    2017

  27. [35]

    Fma: A dataset for music analysis,

    M. Defferrard, K. Benzi, P. Vandergheynst, and X. Bresson, “Fma: A dataset for music analysis,” arXiv preprint arXiv:1612.01840, 2016. 7 Appendices A Related Work Our challenge broadens the scope by including non-speech related t asks, enabling a more comprehensive evaluation o...

    2016 arXiv

Pith tools

Reviewed August 10, 2026 · model on record in the stance chip above.