{"id":"a003795b-d87a-42e9-8114-8d8214379a84","arxiv_id":"2608.04433","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":4.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":5,"one_line_summary":"A LoRA-tuned speech encoder with an ECAPA-TDNN head outperforms prior gender-recognition systems on most English and Southeast Asian test sets.","lead":"This paper introduces MERaLiON-GR, a speech model that guesses whether a speaker is female or male in English and several Southeast Asian languages. It reports that the model beats a leading baseline on most benchmarks and that feeding its predictions to an audio chatbot improves the chatbot's answers.","discovery_kind":"new_application","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Model-card leakage and incomplete SG-ECMT label agreement reporting weaken the claimed 'consistent' superiority, but the public-benchmark wins are credible; request two targeted additions.","rationale":"The reader's weakest_assumption correctly identified the SG-ECMT label pipeline as the most important unresolved issue, and my stress-test agrees that this is the single most load-bearing concern because the in-the-wild results are the paper's justification for segment-level robustness and practical replacement claims. I also independently flagged the abstract's 'consistently surpasses' versus the introduction's '12 of 15' mismatch and the leakage/perfect-score issue, but neither of these is fatal: the public benchmark wins are concrete evidence even if the perfect scores are suspicious. The paper has real strengths—parameter-efficient training with LoRA and ECAPA-TDNN is a sensible design, and the 12/15 public wins are credible as far as they go—but the missing annotation agreement statistics and the lack of any code release mean the reader's CONDITIONAL verdict is appropriate. My recommendation is therefore UNCHANGED relative to the reader's verdict, with the concrete test specifying exactly what to request from the authors.","tokens_in":7474,"tokens_out":1459,"duration_ms":13717,"concrete_test":"Ask the authors for (1) the SG-ECMT annotation meta-data: fraction of WavLM-ECAPA labels changed by annotators, per-language inter-annotator agreement, and a breakdown of Table 3 accuracy computed on the portion of samples where the automatic label was changed vs. unchanged (if the model is much more accurate on unchanged labels, the seed bias concern holds); (2) a check for training/test overlap with the public sets that show 100.00% accuracy, or at least a statement that FLEURS English, Tamil OpenSLR, and Thai Elderly contain no utterances from speakers seen during training. These two additions would settle whether the central claim needs to be weakened to '12/15 public benchmarks plus provisional in-the-wild results.'","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim of consistent superiority is supported by 12/15 public wins, but the paper's own reported details leave a load-bearing gap: the SG-ECMT in-the-wild evaluation (Table 3) uses labels seeded by a WavLM-ECAPA model and then corrected by native annotators, without reporting how much the automated labels changed. If the annotators were biased toward the automatic labels, Table 3's gains (e.g., +4.32 pp on Malay) could reflect alignment with the seed model's biases rather than true recognition quality. The paper also explicitly narrows its claim in the introduction ('outperforming Vox-Profile on 12 of 15 public benchmarks') while the abstract uses 'consistently surpasses,' which is an overstatement given the two public-set losses and one tie. A second concern is the 100.00% accuracies on FLEURS English, Tamil OpenSLR, and Thai Elderly: if those test sets have utterance-level gender ambiguity or label noise, perfect accuracy can only occur when such ambiguous examples are absent or the model was trained on overlapping data, and no such leakage checks are reported. Because the strongest evidence (12/15 public benchmarks) is not undermined by these gaps, the appropriate action is CONDITIONAL rather than ACCEPT or REJECT.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper presents MERaLiON-GR, a speech gender recognition model that combines a LoRA-adapted MERaLiON-SpeechEncoder-2 backbone with an ECAPA-TDNN head and attention pooling. The authors evaluate the model on fifteen public benchmark test sets spanning English and eight Southeast Asian languages, on a smaller set of public benchmarks for Audio-LLM metadata injection, and on four internal Singapore in-the-wild segment-level sets (SG-ECMT). The central claim is that MERaLiON-GR consistently outperforms the state-of-the-art Vox-Profile model and a general-purpose Audio-LLM in both full-utterance and segment-level evaluation. The training recipe, including LoRA rank, scaling, dropout, class-balanced loss, checkpoint averaging, and hyperparameters, is reported in detail.","tokens_in":7761,"tokens_out":2645,"duration_ms":26872,"significance":"If the central claim holds, the paper provides a practical, parameter-efficient recipe for multilingual gender recognition that improves on public state-of-the-art results across several SEA languages, and it demonstrates that explicit paralinguistic metadata can substantially improve Audio-LLM performance on gender-related questions. The strengths of the paper are its broad public benchmark coverage, the use of an external baseline (Vox-Profile) rather than a purely self-referential comparison, the release of the model and demo, and the clear description of training configuration. However, the abstract-level claim of 'consistently surpasses' is stronger than the reported data, and the internal SG-ECMT evaluation depends on labels that were seeded by an automatic model before human correction, with no reported agreement or correction statistics. These issues make the paper's headline claim defensible on the public benchmarks but not yet fully verified as stated.","major_comments":[{"comment":"The abstract and §7 state that MERaLiON-GR 'consistently surpasses' Vox-Profile, but Table 1 reports two losses (Malay SMALDUSC, −4.40 pp; Thai THAI-SER, −2.09 pp) and one tie (Khmer FLEURS, 99.74 both). The introduction's more careful wording ('outperforming Vox-Profile on 12 of 15 public benchmarks') should be adopted throughout, and the abstract and summary should be revised accordingly. In addition, no confidence intervals or significance tests are provided; several margins are small (e.g., IEMOCAP 98.90 vs. 97.31, Tamil Common Voice 94.00 vs. 92.30), so the reader cannot distinguish genuine improvement from evaluation noise. Please report confidence intervals, bootstrap estimates, or pairwise significance tests at least for the public benchmark comparisons.","section":"Abstract, §1, §6.1, Table 1"},{"comment":"The SG-ECMT in-the-wild labels were automatically estimated with a WavLM-ECAPA model and then 'reviewed and corrected by native-speaking annotators,' but the paper reports no fraction of label changes, no inter-annotator agreement, and no explicit correction criteria. If the annotators were anchored by the automatic labels, the segment-level improvements in Table 3 (e.g., +4.32 pp on Malay and +4.16 pp on Tamil) could reflect alignment with the seed model's biases rather than true recognition quality. Please report the proportion of samples whose labels were changed by human review, agreement statistics (e.g., Cohen's kappa or percentage agreement on a subset), and a description of the correction protocol, or provide an alternative evaluation with independently produced human labels.","section":"§4.1.1, §6.2, Table 3"},{"comment":"The paper reports 100.00% accuracy on English FLEURS, Tamil OpenSLR, and Thai Elderly. Perfect accuracy on public datasets that are likely to contain at least some label noise or utterance-level gender ambiguity is surprising and deserves explicit support. Please describe any duplicate-removal or leakage checks performed between the pretraining/fine-tuning corpora and these test sets, report the number of test utterances and any samples excluded due to label ambiguity, and state whether the reported 100.00% figures are stable under minor perturbations of the evaluation protocol.","section":"§6.1, Table 1"}],"minor_comments":[{"comment":"The sentence 'All BatchNorm layers are replaced with GroupNorm layers that are in the original ECAPA-TDNN structure' is unclear; please rephrase to indicate whether GroupNorm were already part of the original ECAPA-TDNN implementation used, or whether they were introduced as a modification in this work.","section":"§2.2"},{"comment":"The fixed prompt for MERaLiON-v2 is a reasonable choice, but a single prompt may underestimate the Audio-LLM's capability; a brief sensitivity check with one or two alternative phrasings would strengthen the claim that the Audio-LLM is a fair baseline.","section":"§5"},{"comment":"The text says training ran for 15 epochs with early stopping based on development loss, but also that final parameters were obtained by averaging the four lowest-loss checkpoints; please clarify whether the averaged checkpoints came from the same 15-epoch run and how early stopping interacted with checkpoint selection.","section":"§3.2"},{"comment":"The GR-AudioLLM results in Table 2 are evaluated on a subset of the public benchmarks (8 of 15); the text should explicitly state that this subset was chosen because gender-related metadata injection is only applicable to those datasets, or explain the selection criterion.","section":"§6.1, Table 2"},{"comment":"There are a few typographical and formatting issues, including inconsistent spacing in Equation (2) and the use of 'V ox-Profile' with a line break in several places; a careful proofread would improve readability.","section":"Throughout"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope for a speech/audio processing venue and the public benchmark evidence is genuinely valuable. The main risks are rhetorical overclaiming and the unquantified reliability of the SG-ECMT labels; both are addressable with targeted additions. I would not recommend rejection because the core public-benchmark result does not depend on the internal set. The self-citation pattern is reasonable for building-block papers, though the authors should ensure that the new contribution (the GR model and its evaluation) is clearly separable from prior MERaLiON work."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"The core result is credible. MERaLiON-GR beats Vox-Profile on 12 of 15 public test sets spanning eight languages, ties on Khmer FLEURS, and loses only on Malay SMALDUSC and Thai SER. Those are external benchmarks, so the central claim is not circular. The public model release and the SEA-language evaluation are genuinely useful, and the paper makes a fair point that a dedicated paralinguistic model outperforms a general Audio-LLM on this task. The LoRA-plus-ECAPA recipe is established, so the novelty is incremental, but that is not a flaw for a systems paper.\n\nThe soft spots are proportionate. The abstract says the model \"consistently surpasses\" Vox-Profile, but the paper's own Table 1 shows two losses and a tie. The introduction states the narrower claim (12 of 15), so this is an internal inconsistency that should be fixed. There are no confidence intervals or significance tests anywhere; given that several differences are under two points, some of the wins may be noise. The SG-ECMT in-the-wild labels were seeded by a WavLM-ECAPA model and then corrected by native annotators, but the paper does not report how many auto labels changed or inter-annotator agreement. That is a real reporting gap, though not fatal: both models are evaluated on the same labels, so the comparison is at least fair, and the training data is human-labelled VoxCeleb and NSC, not the seed model's outputs. The three 100.00% accuracies are suspicious only in the sense that perfect scores invite a check for label leakage; the datasets are clean read speech, so the numbers are plausible. Still, the paper should report the test-set sizes and check for speaker/utterance overlap with training data. No training or evaluation code is released, which limits reproducibility, but the model weights are on HuggingFace.\n\nThe citation pattern looks fine. Self-citations are to the encoder, the Audio-LLM, and prior metadata work, all of which are legitimate building blocks. The one thing I would ask for in revision is the SG-ECMT annotation statistics and a corrected abstract; the public-benchmark evidence does not need to be re-run.\n\nThis paper deserves a serious referee. It is a solid empirical systems contribution with a released model and useful SEA evaluation. The issues are fixable and do not break the central claim. I would accept it with minor-to-moderate revision, not desk-reject it.","headline":"Credible 12/15 public-benchmark wins, but the abstract overclaims and the in-the-wild label pipeline needs fuller reporting; worth a serious referee.","tokens_in":8357,"tokens_out":2070,"would_cite":true,"duration_ms":20999,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"A LoRA-tuned conformer encoder with an ECAPA-TDNN head reports higher gender-recognition accuracy than Vox-Profile on 12 of 15 public English and Southeast Asian test sets.","keywords":["speech gender recognition","paralinguistic analysis","Low-Rank Adaptation (LoRA)","ECAPA-TDNN","conformer speech encoder","Southeast Asian languages","cross-lingual generalization","audio large language models"],"falsifier":"Re-annotate the SG-ECMT segments with multiple independent native-speaking annotators who never see any automatic prediction, then compare MERaLiON-GR's two-second segment accuracy with Vox-Profile on the human-only labels; if the gap shrinks or disappears, the claimed in-the-wild advantage is an artifact of label alignment.","tokens_in":7263,"feed_emoji":"🎙️","tokens_out":8841,"duration_ms":73326,"temperature":0.7,"pith_summary":"MERaLiON-GR is a speech gender-recognition system that starts from a large pre-trained conformer speech encoder, keeps the encoder frozen, and adapts it with low-rank adaptation (LoRA) plus a compact ECAPA-TDNN classifier. The paper's central claim is that this recipe reports higher accuracy than the previous state-of-the-art dedicated gender model, Vox-Profile, on 12 of 15 public English and Southeast Asian test sets, and equal accuracy on one, while a general-purpose audio large language model (Audio-LLM) falls further behind. On two-second in-the-wild Singapore speech in four languages, the model reports gains of up to 4.32 percentage points over Vox-Profile. If these results hold, they support the argument that a task-specific, parameter-efficient model is the right tool for fine-grained paralinguistic attributes and that such attributes can generalize across languages.","feed_headline":"Speech gender model tops prior state of the art on 12 of 15 tests","feed_subtitle":"A compact fine-tuned encoder beats a general audio LLM and scales across eight Southeast Asian languages.","key_machinery":"The load-bearing mechanism is the coupling of a frozen Conformer speech encoder (a transformer variant with convolutional modules) with low-rank adaptation and a compact ECAPA-TDNN head. LoRA lets the task update only low-rank factors in the attention projections, which keeps adaptation cheap and limits catastrophic forgetting; a learned soft attention over all 25 layer outputs combines phonetic and speaker-level cues; ECAPA-TDNN, a time-delay neural network with emphasized channel attention and attention pooling, turns the frame-level representation into a fixed 256-dimensional utterance embedding; and a linear head with RMSNorm and GELU maps that embedding to the female/male logits. This combination is what lets a generic multilingual speech encoder become a specialized, cross-lingually stable gender classifier without full fine-tuning.","core_discovery":"The paper's discovery claim is that gender recognition from speech is better served by a dedicated encoder fine-tuned with low-rank adapters than by either the current standalone state of the art or a large audio LLM. The model keeps MERaLiON-SpeechEncoder-2 frozen, inserts LoRA into its attention projections, aggregates all 25 hidden states with learned layer weights, and feeds the result through an ECAPA-TDNN head with attention pooling and a two-class linear classifier. Reported accuracy reaches 100% on English FLEURS, Tamil OpenSLR, and Thai Elderly, and the largest gains over Vox-Profile are on Tamil EmoTa (+4.71 percentage points), Vietnamese Common Voice (+3.14), and Indonesian IndoWave (+3.00), while the clear losses are on Malay SMALDUSC and Thai SER. The paper further claims that feeding the predicted gender into an Audio-LLM lifts gender-question answering accuracy substantially, by up to 60 percentage points on Vietnamese Common Voice.","pith_inferences":["The paper does not test whether its SG-ECMT segment-level advantage survives fully independent human labels; because the labels were seeded by an automatic WavLM-based model, a re-annotation study with no access to those predictions would separate true gains from label alignment.","A dataset-level count such as 12 of 15 treats each benchmark equally; a pooled per-sample significance test could show whether the aggregate advantage is driven by the large Common Voice partitions.","The same frozen backbone plus LoRA and a task head is a general recipe: age, emotion, dialect, and other paralinguistic attributes could likely be added at low marginal cost, which the paper does not claim to demonstrate.","Extending the evaluation to Filipino, Burmese, and Lao would test the cross-lingual claim beyond the languages represented in the training corpus."],"forward_implications":["A dedicated gender-recognition model is likely to remain more accurate than a general-purpose audio LLM on this task, so deployment for paralinguistic metadata should use task-specific heads.","The same LoRA-plus-ECAPA recipe can be applied to other paralinguistic attributes such as age, emotion, or dialect, using the same frozen backbone.","The model can serve as a pseudo-labeling tool for two-second speech segments, which is useful for building larger annotated paralinguistic datasets.","Injecting predicted gender as explicit metadata can substantially improve audio-LLM answers to gender-related questions, suggesting a conditioning strategy for speech-language models.","Singapore-centric training data transfers to multiple Southeast Asian languages, supporting low-resource language applications without new task-specific training."],"supporting_citations":[{"why":"Defines Vox-Profile, the standalone state-of-the-art baseline that MERaLiON-GR must beat on every benchmark.","marker":"[Feng et al., 2025]"},{"why":"Supplies the pre-trained MERaLiON-SpeechEncoder-2 backbone that is frozen and adapted with LoRA.","marker":"[MERaLiON Team, 2024]"},{"why":"Introduces LoRA, the parameter-efficient adaptation method used to fine-tune the encoder.","marker":"[Hu et al., 2022]"},{"why":"Provides the ECAPA-TDNN architecture used as the downstream gender classifier.","marker":"[Desplanques et al., 2020]"},{"why":"Supplies WavLM, used inside the automatic labeler that produced the initial SG-ECMT gender labels.","marker":"[Chen et al., 2022]"},{"why":"Defines the class-balanced loss weights used to counter training-set gender imbalance.","marker":"[Cui et al., 2019]"},{"why":"Provides the VoxCeleb1 training partition with gender labels.","marker":"[Nagrani et al., 2017]"},{"why":"Describes the Singapore speech corpus from which the IMDA training partitions are drawn.","marker":"[Koh et al., 2019]"}],"fun_headline_variants":["Dedicated speech gender model beats audio LLM on 12 of 15 tests","LoRA-finetuned encoder tops Vox-Profile and audio LLM for gender","Compact dedicated model outperforms audio LLM on gender","Efficient LoRA tuning yields top gender accuracy across SEA languages"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The Singapore in-the-wild test set was labelled by an automatic model first and then corrected by native speakers, and the paper reports neither the human correction rate nor annotator agreement, so the segment-level gains could partly reflect agreement with that automatic labeler rather than true gender recognition.","fun_headline_variants_meta":{"raw":{"variants":["Dedicated speech gender model beats audio LLM on 12 of 15 tests","LoRA-finetuned encoder tops Vox-Profile and audio LLM for gender","Compact dedicated model outperforms audio LLM on gender","Efficient LoRA tuning yields top gender accuracy across SEA languages"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000671,"raw_usage":{"total_tokens":3051,"prompt_tokens":933,"completion_tokens":2118,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":549,"completion_tokens_details":{"reasoning_tokens":2039}},"tokens_in":549,"tokens_out":2118,"duration_ms":13927,"temperature":1.0,"reasoning_tokens":2039,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-07T00:09:26.636667+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-annotate the SG-ECMT segments with multiple independent native-speaking annotators who never see any automatic prediction, then compare MERaLiON-GR's two-second segment accuracy with Vox-Profile on the human-only labels; if the gap shrinks or disappears, the claimed in-the-wild advantage is an artifact of label alignment.","supporting_citations":[{"cited_title":"2024 , eprint=","cited_arxiv_id":null,"evidence_quote":"Supplies the pre-trained MERaLiON-SpeechEncoder-2 backbone that is frozen and adapted with LoRA."},{"cited_title":"2022 , publisher=","cited_arxiv_id":null,"evidence_quote":"Supplies WavLM, used inside the automatic labeler that produced the initial SG-ECMT gender labels."},{"cited_title":"Building the","cited_arxiv_id":null,"evidence_quote":"Describes the Singapore speech corpus from which the IMDA training partitions are drawn."}],"review_version":1}