{"id":"0d385168-62ad-4cdd-9545-3da594184afb","arxiv_id":"2501.15302","paper_version":1,"verdict":"UNVERDICTED","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"unknown","formal_verification":"none","parameter_count":0,"one_line_summary":"The paper is a call for participation in the ICME 2025 Audio Encoder Capability Challenge, describing the evaluation protocol, datasets, and scoring rules.","lead":"This paper announces a competition that tests audio encoder models on speech, environmental sound, and music tasks. It describes two tracks, one that fine-tunes a linear layer and one that uses nearest-neighbour classification, and ranks the submitted models.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Eq. (1) normalizes lower-is-better metrics as if higher were better: EER and MSE are transformed so that worse systems receive larger normalized scores, corrupting the Eq. (2) ranking.","rationale":"The reader's UNVERDICTED verdict is reasonable because the paper contains no experimental results and the benchmark's representativeness is unvalidated. My stress-test found a more specific, internal flaw: the normalization in §4.1 does not invert lower-is-better metrics. ASV2015 EER and speechocean762 MSE are explicitly listed in Table 1, and Eq. (1) maps a better (lower) value to a lower normalized score, so the weighted final score in Eq. (2) can rank a worse system above a better one. This is independent of any empirical validation and can be demonstrated from the text itself. I therefore recommend CONDITIONAL rather than UNCHANGED: the challenge description's scoring rule should be corrected (or explicitly state an inversion for lower-is-better tasks) before the benchmark can support the claimed 'evaluation of capabilities.' If the organizers' public evaluation code already inverts these metrics, then the paper should say so; the test above would then show no discrepancy and the concern would not land. This does not call into question the organizers' expertise or the challenge's general value; it targets one formula in the proposed ranking.","tokens_in":7753,"tokens_out":6058,"duration_ms":54180,"concrete_test":"Take two synthetic Track A submissions identical on every task except ASV2015 and speechocean762. Submission G has EER=0.01 and MSE=0.01; submission B has EER=0.30 and MSE=0.30. Compute Eq. (2) literally as written, then recompute with direction-corrected normalization for lower-is-better metrics (\\hat M_i=(M_max_i − M_i)/(M_max_i − M_min_i) for ASV2015 and speechocean762, keeping all other tasks unchanged). If the relative order of G and B changes between the two computations, the formula is not ranking by capability. This can be checked from Tables 1–2 alone; no model training is required.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central mechanism of the paper is the final score in Eq. (2), and it depends on the normalized metric in Eq. (1). The paper states in §4.1 that EER ranges 0–1, so M_min=0 and M_max=1. But for EER lower is better. Under Eq. (1), a perfect EER of 0 maps to \\hat M=0 and a useless EER of 1 maps to \\hat M=1, so the equation rewards worse spoofing detection. The same issue applies to the MSE task (speechocean762), which also is lower-is-better and is in Track A. No direction correction (e.g., \\hat M = (M_max − M)/(M_max − M_min)) is given, and applying the literal formula to a lower-is-better metric makes the final weighted average non-monotonic: a system that improves on ASV2015 or speechocean762 receives a lower score. Since the central claim is that the challenge ranks encoders by capability, the scoring rule as written fails for at least two of the listed tasks. This is an internal consistency problem, not a disagreement with community benchmarks.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"Zhang et al. describe the ICME 2025 Audio Encoder Capability Challenge, in which participants submit pre-trained continuous audio encoders through a fixed API. The encoders are evaluated on a broad set of public datasets spanning speech, environmental sound, and music, plus six hidden industrial datasets. Track A trains a linear probe on frozen embeddings using predefined hyperparameters; Track B applies KNN classification directly to the embeddings. Rankings are computed as a test-set-size-weighted average of per-task metrics normalized to [0,1] via Eq. (1). The paper is a challenge/protocol announcement: it contains no experimental evaluations, baselines, or variance analyses.","tokens_in":7916,"tokens_out":7155,"duration_ms":63353,"significance":"If the protocol is executed as intended, the challenge would fill a useful gap by providing a comparative benchmark for continuous audio embeddings across speech, environmental audio, and music, complementing HEAR, SUPERB, and DASB. The open-source evaluation system, the fixed model API, the two-track design, and the inclusion of real-world hidden datasets are genuine strengths. However, the stated scoring formula is incorrect for lower-is-better metrics, which directly affects the central ranking mechanism, and the paper gives no baseline evidence that the proposed weighted average is stable or discriminative. The challenge idea is valuable but the protocol as written needs correction and validation.","major_comments":[{"comment":"The normalization in Eq. (1) is directionally wrong for lower-is-better metrics. For EER, lower values are better, yet §4.1 sets M_min=0 and M_max=1, so Eq. (1) maps a perfect EER of 0 to 0 and a useless EER of 1 to 1. Under the final score in Eq. (2), a system with worse spoofing detection receives a higher contribution for the ASV2015 task, in both Track A and Track B. The same inversion applies to MSE on speechocean762. The formula must be corrected, for example by assigning M_min as the worst possible value and M_max as the best possible value so that lower-is-better metrics are transformed appropriately, or by explicitly inverting those metrics before applying Eq. (1).","section":"§4.1, Eq. (1)"},{"comment":"The normalization for the speechocean762 MSE task is under-specified even apart from direction: MSE has no canonical [0,1] range, and no M_min/M_max values are given for it. Without those values, Eq. (2) cannot be computed for that task, and if M_max is intended to be 1, the inversion problem above applies. Please state the assumed range or a standardizing transformation for MSE.","section":"§4.1, Table 1"},{"comment":"Track B is described as evaluating 'a subset of Track A's utterance-level classification tasks', but Table 1 marks ASV2015 (spoofing detection, EER) and DESED (sound event detection, segment-F1) as included in Track B. These are not utterance-level classification tasks, and the paper does not explain how KNN classification would be applied to them or how EER and segment-F1 would be derived from KNN outputs. The Track B protocol also omits the K value, the distance metric, and the feature pooling strategy for variable-length inputs; these details are necessary for a reproducible ranking.","section":"§1.1 and Table 1"},{"comment":"The weights n_i in Eq. (2) are the test-set sizes, but the paper never lists test-set sizes for the public datasets, and for hidden datasets Table 2 reports only total sample sizes or durations (e.g., 125 hours for subway broadcast), not the number of test examples used in scoring. Without a precise definition of n_i for every task, the final score is not reproducible and the relative weighting between tasks remains unspecified.","section":"§4.2, Tables 1–2"},{"comment":"The manuscript provides no baseline evaluations, so the central assumption that the test-set-size-weighted average in Eq. (2) yields a stable and discriminative ranking is unvalidated. I recommend adding results for at least two reference encoders (e.g., wav2vec2 and Dasheng) on all public tasks, including per-task normalized metrics, final scores, and a sensitivity analysis such as leaving out one dataset at a time to check whether any single large test set dominates the ranking.","section":"§1 and §4.2"}],"minor_comments":[{"comment":"There are several typos and formatting artifacts, including 'en coders', 'machie', 'us large audio', 'incorpeating', and 'pipline'; the organizer biography section also contains promotional language that should be trimmed.","section":"Abstract and §5"},{"comment":"The table would be clearer if each task indicated whether its metric is higher-is-better or lower-is-better; this would have made the EER and MSE normalization issue apparent.","section":"Table 1"},{"comment":"The table uses the name 'ASV2015' but the reference is the ASVspoof 2015 database; align the naming for consistency.","section":"References [25]"},{"comment":"The computational constraints are described as 'Google Colab T4 environment, where the runtime is equipped with a 16 GB NVIDIA Tesla T4 GPU, 12GB RAM'; please clarify whether the 12GB RAM refers to system RAM and whether the 16GB is GPU memory, since these limits directly affect what models can be submitted.","section":"§3"},{"comment":"The definitions of M_min and M_max are stated as 'worst and best possible values', which is only consistent with higher-is-better metrics; please rephrase to cover both metric directions explicitly.","section":"§4.1"}],"recommendation":"major_revision","confidential_remarks":"For the editor: this is best viewed as a challenge announcement rather than a full research paper, and it is thin on empirical validation. The open-source system and broad task coverage are useful, and the scoring-direction error is exactly the kind of issue that should be caught before public release. If the authors correct Eq. (1), specify the test-set weights, clarify the Track B protocol, and provide baseline evidence in the final challenge report, the work would be a reasonable fit for a benchmark/challenge venue. I found no evidence of circular reasoning or undisclosed reuse of prior work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This is a call for participation in the ICME 2025 audio encoder challenge, not a research paper with empirical results. What's new: it combines a broader task set than HEAR or SUPERB (speech, environmental sound, music, plus six hidden industrial datasets), offers two evaluation tracks (linear probing and KNN), and uses a test-set-size-weighted average of normalized metrics. The appendix gives a fair comparison with HEAR, SUPERB, and DASB. That is a reasonable and potentially useful benchmark proposal.\n\nThe soft spot is real and load-bearing. Equation (1) normalizes every metric as if higher were better, but EER (ASV2015) and MSE (speechocean762) are lower-is-better. With M_min=0 and M_max=1, a perfect EER of 0 maps to 0 and a useless EER of 1 maps to 1, so the final weighted score in Eq. (2) rewards worse spoofing detection and worse pronunciation. This is not a minor nitpick; it directly corrupts the rankings for at least two of the listed tasks. The fix is trivial (use M_max - M in the numerator for lower-is-better metrics), but the paper as written is internally inconsistent with its own stated goal of ranking encoders by capability.\n\nThe second issue is that there are no baseline evaluations. The paper gives no evidence that the benchmark is discriminative, stable, or even runnable as described. That is typical for a challenge proposal, but it means the design's soundness cannot be assessed. Also, Section 5 is mostly promotional bios and company descriptions; that belongs on a challenge website, not in an archival paper, and it pads the manuscript.\n\nOn balance: the challenge fills a plausible niche and the authors know the prior benchmarks. The scoring bug should be caught by any careful reader, and it is easily fixable. But until it is fixed, the central mechanism of the paper is broken. I would still send this to peer review rather than desk-reject, because a serious referee can flag the error quickly and the benchmark may be worth running. I would not cite it in its current form, though I might cite the eventual challenge results if the scoring is corrected.\n\nFor a reading group, it is a useful example of how a small formula error can undermine a benchmark design. But it is not a paper to admire; it is a paper to fix.","headline":"Useful challenge design undercut by a scoring formula that rewards worse performance on lower-is-better metrics like EER and MSE.","tokens_in":8432,"tokens_out":2285,"would_cite":false,"duration_ms":21011,"reading_group":"maybe","serious_thinker":"no","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"The paper announces a two-track competition that ranks continuous audio encoders on speech, environmental, and music tasks, including private real-world datasets, using a test-set-size-weighted scoring rule.","keywords":["audio encoder challenge","continuous embeddings","representation learning benchmark","linear probing","k-nearest-neighbor evaluation","real-world audio tasks","test-set-size-weighted scoring","multi-domain audio evaluation"],"falsifier":"Compute the proposed weighted score for two published encoders (e.g., wav2vec2 and Dasheng) using their known accuracies on these datasets: if the ranking reverses when the normalization bounds are changed from theoretical min/max to observed min/max of a baseline, or if a random-feature encoder lands within the noise band of the top models, the benchmark is not discriminative.","tokens_in":7561,"feed_emoji":"🎧","tokens_out":6063,"duration_ms":52008,"temperature":0.7,"pith_summary":"This is a competition-design paper rather than a technical result. The authors propose a benchmark challenge that evaluates pre-trained continuous audio encoders on a diverse suite of public tasks spanning speech, environmental sound, and music, supplemented by six private datasets from real-world industrial scenarios. Its central claim is that this design, with its two evaluation tracks and its scoring rule, can rank encoders on both adaptability and intrinsic representational quality, thereby advancing audio encoder design. A sympathetic reader would care because such a standardized comparison is currently missing: existing benchmarks cover only parts of the domain or focus on discrete tokens, and continuous embeddings are what multimodal systems actually consume.","feed_headline":"Two-track audio benchmark ranks encoders on speech, sound, music","feed_subtitle":"Probes continuous embeddings with linear probes and KNN, weighing test-set size and adding hidden real-world tasks.","key_machinery":"The scoring identity is the weighted normalized total: for each task $i$, the raw metric $M_i$ is mapped to $(M_i - M^{\\min}_i) / (M^{\\max}_i - M^{\\min}_i)$ using theoretical worst and best values, and the track score is the sum of test-set size $n_i$ times the normalized metric divided by the total test-set size, so larger test sets count more. This weighted normalization, together with the two-track structure and the hidden real-world datasets, is what the argument hangs on.","core_discovery":"The paper's proposal is that a single audio encoder should be judged by two complementary operations on its fixed embeddings: a linear fine-tuning probe (Track A) that measures how quickly task-specific classifiers can adapt the representation, and a K-nearest-neighbour evaluation (Track B) that measures raw representational quality without any training. Scores on each track are computed by normalizing each task metric to a 0–1 scale, then taking a test-set-size-weighted average across tasks; participants are ranked on separate leaderboards for the two tracks. The protocol deliberately spans three domains (speech, environmental sound, and music) and includes six withheld datasets that mirror industrial conditions such as car interior sounds, finger-snap wake-words, and subway broadcasts, so that the ranking reflects real-world usability, not just academic dataset performance.","pith_inferences":["Because weights are test-set sizes, the final score will be dominated by a handful of large test sets; tasks with few test samples may contribute little even if they test a distinct capability, so the ranking may not reflect breadth of skill.","The theoretical min–max normalization assumes each metric has meaningful worst and best bounds; if one task is much noisier than others, its normalized scores will compress differences and dilute that task's effect despite its test-set weight.","A natural extension would apply the same protocol to discrete-token encoders and compare the two families on the same ranking; the paper hints at this by citing the discrete-audio benchmark but does not run that comparison.","If organizers released per-task scores, later analysis could determine whether the challenge's ranking is stable under different normalization choices (e.g., using a baseline model's observed min/max instead of theoretical bounds), a test the paper does not report."],"forward_implications":["If the challenge runs as specified, the open-source evaluation system will give participants a shared API and a reusable pipeline for comparing continuous audio encoders across all three domains.","The two-track design will let the community observe whether linear-probe performance and KNN performance track each other, or whether some encoders excel at one but not the other.","The six hidden datasets will provide a transfer test that public-task fine-tuning cannot capture, since the public Table 1 datasets are excluded from training.","The final leaderboards will be the first benchmark to rank continuous encoders on a common weighted scale covering speech, sound, and music simultaneously."],"supporting_citations":[{"why":"The HEAR benchmark is the direct template the authors say they are strongly inspired by, and its task set is what they expand toward balanced speech, sound, and music coverage.","marker":"[13]"},{"why":"SUPERB defines the standard speech self-supervised-learning evaluation that the authors claim to broaden beyond speech-only tasks.","marker":"[14]"},{"why":"DASB is the discrete-token benchmark they position against, establishing the continuous-versus-discrete contrast that motivates the challenge.","marker":"[15]"},{"why":"wav2vec 2.0 is a reference speech encoder whose API shapes the required submission format, with a provided example implementation.","marker":"[10]"},{"why":"Dasheng is a recent masked audio encoder from the organizers that stands as a likely strong baseline the submitted models should match or beat.","marker":"[12]"}],"fun_headline_variants":["Dual-track audio test: linear probes vs KNN on real-world tasks","Audio encoders face hidden real-world tasks in two-track benchmark","Two-track challenge scores audio embeddings with probes and KNN","ICME 2025: Audio encoder capability showdown with six hidden tasks","Audio encoder benchmark: two tracks, six hidden real-world tasks"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The ranking's usefulness rests on the unstated premise that the selected public tasks plus six hidden datasets are representative of real-world audio processing, and that the test-set-size-weighted average of normalized scores yields a meaningful, stable ordering of encoders; the paper offers no baseline results showing this is so.","fun_headline_variants_meta":{"raw":{"variants":["Dual-track audio test: linear probes vs KNN on real-world tasks","Audio encoders face hidden real-world tasks in two-track benchmark","Two-track challenge scores audio embeddings with probes and KNN","ICME 2025: Audio encoder capability showdown with six hidden tasks","Audio encoder benchmark: two tracks, six hidden real-world tasks"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000672,"raw_usage":{"total_tokens":2981,"prompt_tokens":785,"completion_tokens":2196,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":401,"completion_tokens_details":{"reasoning_tokens":2106}},"tokens_in":401,"tokens_out":2196,"duration_ms":13026,"temperature":1.0,"reasoning_tokens":2106,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T14:23:35.732902+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Compute the proposed weighted score for two published encoders (e.g., wav2vec2 and Dasheng) using their known accuracies on these datasets: if the ranking reverses when the normalization bounds are changed from theoretical min/max to observed min/max of a baseline, or if a random-feature encoder lands within the noise band of the top models, the benchmark is not discriminative.","supporting_citations":[{"cited_title":"HEAR: Holistic evaluation of audio representations,","cited_arxiv_id":null,"evidence_quote":"The HEAR benchmark is the direct template the authors say they are strongly inspired by, and its task set is what they expand toward balanced speech, sound, and music coverage."},{"cited_title":"SUPERB: Speech processing universal performance benchmar k,","cited_arxiv_id":null,"evidence_quote":"SUPERB defines the standard speech self-supervised-learning evaluation that the authors claim to broaden beyond speech-only tasks."},{"cited_title":"wav2vec 2.0: A fr amework for self-supervised learning of speech representations,","cited_arxiv_id":null,"evidence_quote":"wav2vec 2.0 is a reference speech encoder whose API shapes the required submission format, with a provided example implementation."},{"cited_title":"Sc aling up masked audio encoder learning for general audio classification,","cited_arxiv_id":null,"evidence_quote":"Dasheng is a recent masked audio encoder from the organizers that stands as a likely strong baseline the submitted models should match or beat."}],"review_version":1}