pith. sign in

arxiv: 2607.01161 · v1 · pith:LG3ETKVPnew · submitted 2026-07-01 · 📡 eess.AS · cs.CL

Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages

Pol Buitrago , Javier Hernando This is my paper

Pith reviewed 2026-07-02 04:53 UTC · model grok-4.3

classification 📡 eess.AS cs.CL
keywords cross-lingual speaker verificationIberian languageslanguage mismatchspeaker variabilityevaluation setHuBERTtransfer matrix
0
0 comments X

The pith

A same-speaker bilingual test set shows language mismatch drives most cross-lingual speaker verification degradation.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The work creates an evaluation set in which the same speakers produce utterances in five Iberian languages, removing speaker identity as a confounder. This set is applied to a HuBERT-based verification system previously observed to suffer from language dependence, with results examined through a Cross-Lingual Transfer Matrix that compares every language pair. The analysis finds that speaker-related variability explains only part of the performance drop, while language mismatch accounts for the larger share. Standard protocols that use different speakers across languages therefore mix two effects and overstate the contribution of speaker differences. The new protocol supplies a cleaner separation of the two sources of error.

Core claim

By constructing a bilingual same-speaker evaluation set for five Iberian languages and analyzing results with the Cross-Lingual Transfer Matrix, the study establishes that speaker-related variability accounts for part of the observed degradation in cross-lingual speaker verification, yet language mismatch remains the main driver of performance loss.

What carries the argument

The bilingual same-speaker evaluation set that keeps speaker identity fixed across languages, paired with the Cross-Lingual Transfer Matrix for pairwise transfer analysis.

If this is right

  • Cross-lingual speaker verification performance loss cannot be attributed primarily to speaker differences once speaker identity is controlled.
  • Language mismatch must be addressed directly in system design rather than treated as a secondary effect.
  • The Cross-Lingual Transfer Matrix reveals asymmetric transfer patterns between specific language pairs.
  • Standard evaluation protocols that change speakers across languages overestimate the role of inter-speaker variability.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Training regimes that explicitly penalize language-specific embeddings may yield larger gains than additional speaker adaptation data.
  • The same-speaker isolation method could be extended to measure language effects in other verification or diarization tasks.
  • If language mismatch dominates, multilingual pre-training objectives that align phonetic spaces across languages become higher priority than speaker-discriminative losses alone.

Load-bearing premise

The new bilingual same-speaker evaluation set isolates language mismatch without collection or selection biases that would confound the comparison.

What would settle it

Repeating the same-speaker protocol on an independently collected Iberian bilingual corpus and finding that performance degradation largely disappears would falsify the claim that language mismatch is the dominant factor.

Figures

Figures reproduced from arXiv: 2607.01161 by Javier Hernando, Pol Buitrago.

Figure 2
Figure 2. Figure 2: Learning curves for the five target languages used to determine the dynamic training interval [N, 2N]. 3.3. Model and Training We use the multilingual mHuBERT-1471 encoder [16], a HuBERT-based model pretrained on 147 languages, including all Iberian languages considered in this work. Following [6], the encoder is fine-tuned for speaker identification by appending a randomly initialized linear classificatio… view at source ↗
Figure 5
Figure 5. Figure 5: Spanish-Catalan CLTM comparison. (a) Standard evaluation (b) Same-speaker evaluation [PITH_FULL_IMAGE:figures/full_fig_p003_5.png] view at source ↗
Figure 6
Figure 6. Figure 6: Spanish-Galician CLTM comparison. (a) Standard evaluation (b) Same-speaker evaluation [PITH_FULL_IMAGE:figures/full_fig_p003_6.png] view at source ↗
Figure 4
Figure 4. Figure 4: shows the 5×5 CLTM for the five Iberian languages, obtained under the standard evaluation protocol. The matrix departs markedly from the language-agnostic ideal, with struc￾tured and highly asymmetric transfer patterns. Spanish, Cata￾lan, and Galician form the main positive cluster, while Basque and Portuguese mostly yield negative transfer, indicating strong language dependence in the learned representati… view at source ↗
Figure 8
Figure 8. Figure 8: Spanish-Basque CLTM comparison [PITH_FULL_IMAGE:figures/full_fig_p003_8.png] view at source ↗
read the original abstract

Cross-lingual speaker verification (SV) systems typically exhibit performance degradation when enrollment and test utterances are spoken in different languages. However, standard evaluation protocols confound language mismatch with inter-speaker variability, as evaluation is generally performed with different speakers across languages. In this work, we introduce a bilingual same-speaker evaluation set for five Iberian languages, enabling analysis of cross-lingual SV under constant speaker identity. We apply this setup to a HuBERT-based SV system previously shown to exhibit strong language dependence, and analyze results using the Cross-Lingual Transfer Matrix (CLTM) to study pairwise cross-lingual transfer. Our results show that speaker-related variability accounts for part of the observed degradation, but language mismatch remains the main driver of cross-lingual performance loss. These findings provide a more precise characterization of language dependence in cross-lingual SV.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper introduces a bilingual same-speaker evaluation set covering five Iberian languages to isolate language mismatch from speaker variability in cross-lingual speaker verification. It applies this set to a HuBERT-based SV system previously shown to be language-dependent and analyzes pairwise transfer using the Cross-Lingual Transfer Matrix (CLTM). The central claim is that speaker-related variability explains only part of the observed degradation while language mismatch remains the dominant driver.

Significance. If the new evaluation set successfully holds speaker identity fixed without confounding factors, the work supplies a useful diagnostic tool for cross-lingual SV research and a clearer decomposition of performance loss. The CLTM analysis offers a structured way to examine language-pair effects that could guide targeted adaptation methods. The contribution is primarily methodological and empirical rather than theoretical.

major comments (2)
  1. [Description of the bilingual same-speaker evaluation set] The central claim that language mismatch is the main driver rests on the bilingual same-speaker set successfully isolating language effects. The manuscript provides no quantitative checks (e.g., utterance duration statistics, SNR distributions, or phonetic coverage comparisons) that recording conditions and content are matched across languages for the same speakers. Without such evidence or sensitivity analyses, the attribution of residual degradation to language cannot be considered load-bearing.
  2. [Results and CLTM analysis] The results section reports that speaker variability accounts for 'part' of the degradation but supplies no explicit numerical breakdown (e.g., EER deltas between same-speaker cross-lingual and different-speaker baselines) or statistical significance tests. This makes it impossible to verify the relative magnitudes asserted in the abstract and conclusion.
minor comments (2)
  1. [Abstract] The abstract states that the HuBERT system was 'previously shown to exhibit strong language dependence' but does not cite the specific prior work or dataset; a reference should be added for traceability.
  2. [Method] Notation for the CLTM is introduced without an explicit equation or matrix definition in the provided text; a formal definition would improve reproducibility.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments, which help clarify the presentation of our evaluation set and results. We address each major comment below and will revise the manuscript accordingly.

read point-by-point responses

  1. Referee: [Description of the bilingual same-speaker evaluation set] The central claim that language mismatch is the main driver rests on the bilingual same-speaker set successfully isolating language effects. The manuscript provides no quantitative checks (e.g., utterance duration statistics, SNR distributions, or phonetic coverage comparisons) that recording conditions and content are matched across languages for the same speakers. Without such evidence or sensitivity analyses, the attribution of residual degradation to language cannot be considered load-bearing.

    Authors: We agree that the current version does not include these quantitative checks, which would strengthen the claim that the set isolates language effects. In the revised manuscript we will add utterance duration statistics, SNR distributions, and phonetic coverage comparisons across languages for the same speakers, along with any feasible sensitivity analyses. revision: yes

  2. Referee: [Results and CLTM analysis] The results section reports that speaker variability accounts for 'part' of the degradation but supplies no explicit numerical breakdown (e.g., EER deltas between same-speaker cross-lingual and different-speaker baselines) or statistical significance tests. This makes it impossible to verify the relative magnitudes asserted in the abstract and conclusion.

    Authors: We accept that the manuscript would benefit from explicit numerical breakdowns and statistical tests. The revised version will report specific EER deltas between same-speaker cross-lingual and different-speaker baselines, together with statistical significance tests, to quantify the relative contributions of speaker variability and language mismatch. revision: yes

Circularity Check

0 steps flagged

No circularity; purely empirical comparison on new test set

full rationale

The paper introduces a bilingual same-speaker evaluation set and applies an existing HuBERT-based SV system to it, reporting measured EER differences via the CLTM. No derivations, first-principles predictions, parameter fitting presented as prediction, or self-citation load-bearing steps are present. The central claim rests on direct empirical comparison of same-speaker vs. different-speaker conditions, which is independent of any internal reduction to inputs. This is the most common honest non-finding for empirical ML papers.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

Empirical study introducing a new evaluation protocol; contains no free parameters, mathematical axioms, or invented entities.

pith-pipeline@v0.9.1-grok · 5673 in / 895 out tokens · 19884 ms · 2026-07-02T04:53:47.746143+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

30 extracted references · 30 canonical work pages · 1 internal anchor

  1. [1]

    Disentangling Speaker and Language Effects in Cross-Lingual Speaker Verification for Iberian Languages

    Introduction Speaker Verification (SV) systems aim to determine whether two utterances originate from the same speaker by extract- ing embeddings that capture identity-specific traits. Because speaker identity is largely independent of linguistic content, SV is expected to rely primarily on extralinguistic cues and to gen- eralize across languages [1, 2, ...

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  2. [2]

    A brief summary of the formulation is provided below; full details can be found in [6]

    Method We analyze cross-lingual transfer in SV using our previously proposed Cross-Lingual Transfer Matrix (CLTM), a framework that quantifies how incorporating training data from a donor lan- guage affects target-language performance relative to an equiv- alent amount of target-language data. A brief summary of the formulation is provided below; full det...

    work page

  3. [3]

    Experimental Setup 3.1. Data We use the Mozilla Common V oice corpus 25.0 [15], which provides broad multilingual coverage of Iberian languages un- der a uniform recording protocol, reducing extraneous variabil- ity. Its large speaker pool enables the presence of multilingual speakers, essential for our bilingual evaluation setup. For training, we constru...

    work page 2000

  4. [4]

    We then compare these results against same-speaker evaluation conditions to assess the contribution of speaker variability to cross-lingual degradation

    Results We first compute the CLTM for all considered Iberian languages under the standard evaluation protocol, using it to characterize overall cross-lingual transfer patterns. We then compare these results against same-speaker evaluation conditions to assess the contribution of speaker variability to cross-lingual degradation. CLTMs are visualized as hea...

    work page arXiv

  5. [5]

    Conclusions We study cross-lingual SV across five Iberian languages us- ing a HuBERT-based model and analyze transfer patterns with the Cross-Lingual Transfer Matrix (CLTM), observing strong language-dependent behavior aligned with linguistic similarity. A bilingual same-speaker evaluation shows that using the same speakers reduces part of the cross-lingu...

    work page

  6. [6]

    Identification and discrimination of bilingual talkers across languages,

    S. J. Winters, S. V . Levi, and D. B. Pisoni, “Identification and discrimination of bilingual talkers across languages,”Journal of the Acoustical Society of America, vol. 123, no. 6, pp. 4524–4538, 2008

    work page 2008

  7. [7]

    Laspa: Language agnostic speaker disentanglement with prefix-tuned cross-attention,

    A. S. Menon, R. P. Gohil, K. Tripathi, and P. Wasnik, “Laspa: Language agnostic speaker disentanglement with prefix-tuned cross-attention,” inProc. Interspeech 2025, 2025, pp. 3623–3627

    work page 2025

  8. [8]

    Cross-modal speaker verification and recognition: A multilingual perspective,

    S. Nawaz, M. S. Saeed, P. Morerio, A. Mahmood, I. Gallo, M. H. Yousaf, and A. Del Bue, “Cross-modal speaker verification and recognition: A multilingual perspective,” in2021 IEEE/CVF Con- ference on Computer Vision and Pattern Recognition Workshops (CVPRW), 2021, pp. 1682–1691

    work page 2021

  9. [9]

    Language dependency in text-independent speaker verification,

    R. Auckenthaler, M. Carey, and J. Mason, “Language dependency in text-independent speaker verification,” in2001 IEEE Interna- tional Conference on Acoustics, Speech, and Signal Processing. Proceedings (Cat. No.01CH37221), vol. 1, 2001, pp. 441–444 vol.1

    work page 2001

  10. [10]

    Cross-lingual speaker verification with deep feature learning,

    L. Li, D. Wang, A. Rozi, and T. F. Zheng, “Cross-lingual speaker verification with deep feature learning,” in2017 Asia-Pacific Sig- nal and Information Processing Association Annual Summit and Conference (APSIPA ASC), 2017, pp. 1040–1044

    work page 2017

  11. [11]

    Quantify- ing cross-lingual transfer in paralinguistic speech tasks,

    P. Buitrago, O. Pareras, F. Costa, and J. Hernando, “Quantify- ing cross-lingual transfer in paralinguistic speech tasks,”arXiv preprint arXiv:2603.08231, 2026, accepted at Interspeech 2026

    work page arXiv 2026

  12. [12]

    Adversarial domain adaptation for speaker verification using partially shared network,

    Z. Chen, S. Wang, and Y . Qian, “Adversarial domain adaptation for speaker verification using partially shared network,” inProc. Interspeech 2020, 2020, pp. 3017–3021

    work page 2020

  13. [13]

    On the influence of language simi- larity in non-target speaker verification trials,

    P. M. Reuter and M. Jessen, “On the influence of language simi- larity in non-target speaker verification trials,” inProceedings of Interspeech 2025, 2025, pp. 3998–4002

    work page 2025

  14. [14]

    English-chinese bilingual text-independent speaker verification,

    B. Ma and H. Meng, “English-chinese bilingual text-independent speaker verification,” in2004 IEEE International Conference on Acoustics, Speech, and Signal Processing, vol. 5, 2004, pp. V– 293

    work page 2004

  15. [15]

    Towards language-agnostic speaker verification: A cross-lingual transfer study of architectures,

    P. Buitrago and J. Hernando, “Towards language-agnostic speaker verification: A cross-lingual transfer study of architectures,” in Proc. Odyssey 2026, 2026, pp. 298–305

    work page 2026

  16. [16]

    Spoken language mismatch in speaker verification: An investigation with nist-sre and crss bi- ling corpora,

    A. Misra and J. H. L. Hansen, “Spoken language mismatch in speaker verification: An investigation with nist-sre and crss bi- ling corpora,” in2014 IEEE Spoken Language Technology Work- shop (SLT), 2014, pp. 372–377

    work page 2014

  17. [17]

    Language-aware plda for multilingual speaker recognition,

    A. Rozi, D. Wang, L. Li, and T. F. Zheng, “Language-aware plda for multilingual speaker recognition,” in2016 Conference of The Oriental Chapter of International Committee for Coordination and Standardization of Speech Databases and Assessment Tech- niques (O-COCOSDA), 2016, pp. 161–165

    work page 2016

  18. [18]

    Tackling the score shift in cross-lingual speaker verification by exploit- ing language information,

    J. Thienpondt, B. Desplanques, and K. Demuynck, “Tackling the score shift in cross-lingual speaker verification by exploit- ing language information,” inICASSP 2022 - 2022 IEEE Inter- national Conference on Acoustics, Speech and Signal Processing (ICASSP), 2022, pp. 7187–7191

    work page 2022

  19. [19]

    Disentangled representation learning for multilingual speaker recognition,

    K. Nam, Y . Kim, J. Huh, H.-S. Heo, J.-W. Jung, and J. S. Chung, “Disentangled representation learning for multilingual speaker recognition,” inProc. Interspeech 2023, 2023, pp. 5316–5320

    work page 2023

  20. [20]

    Common voice: A massively-multilingual speech corpus,

    R. Ardila, M. Branson, K. Davis, M. Kohler, J. Meyer, M. Henretty, R. Morais, L. Saunders, F. Tyers, and G. Weber, “Common voice: A massively-multilingual speech corpus,” in Proceedings of the Twelfth Language Resources and Evaluation Conference, N. Calzolari, F. B ´echet, P. Blache, K. Choukri, C. Cieri, T. Declerck, S. Goggi, H. Isahara, B. Maegaard, J....

    work page 2020

  21. [21]

    mhubert-147: A compact multilingual hubert model,

    M. Z. Boito, V . Iyer, N. Lagos, L. Besacier, and I. Calapodescu, “mhubert-147: A compact multilingual hubert model,” inInter- speech 2024, 2024, pp. 3939–3943

    work page 2024

  22. [22]

    Decoupled weight de- cay regularization,

    I. Loshchilov and F. Hutter, “Decoupled weight de- cay regularization,” inInternational Conference on Learn- ing Representations, 2019. [Online]. Available: https: //openreview.net/forum?id=Bkg6RiCqY7

    work page 2019

  23. [23]

    J. I. Hualde,The Sounds of Spanish. United Kingdom: Cam- bridge University Press, Oct. 2005

    work page 2005

  24. [24]

    Spanish (164) phoneme inventory,

    “Spanish (164) phoneme inventory,” PHOIBLE 2.0, 2019, accessed: 2026-06-29. [Online]. Available: https://phoible.org/ inventories/view/164

    work page 2019

  25. [25]

    X. L. Regueira Fern ´andez, R. A. Galega, and I. da Lingua Galega, Dicionario de pronuncia da lingua galega. A Coru ˜na: Real Academia Galega, 2010. [Online]. Available: http://galegobook. blogspot.com/2011/05/dicionario-de-pronuncia-da-lingua.html

    work page 2010

  26. [26]

    Galician (2619) phoneme inventory,

    “Galician (2619) phoneme inventory,” PHOIBLE 2.0, 2019, accessed: 2026-06-29. [Online]. Available: https://phoible.org/ inventories/view/2619

    work page 2019

  27. [27]

    Catalan (1138) phoneme inventory,

    “Catalan (1138) phoneme inventory,” PHOIBLE 2.0, 2019, accessed: 2026-06-29. [Online]. Available: https://phoible.org/ inventories/view/1138

    work page 2019

  28. [28]

    Portuguese (163) phoneme inventory,

    “Portuguese (163) phoneme inventory,” PHOIBLE 2.0, 2019, accessed: 2026-06-29. [Online]. Available: https://phoible.org/ inventories/view/163

    work page 2019

  29. [29]

    On the mutual intelligibility of spanish and portuguese,

    J. B. Jensen, “On the mutual intelligibility of spanish and portuguese,”Hispania, vol. 72, no. 4, pp. 848–852, 1989. [Online]. Available: http://www.jstor.org/stable/343562

    work page 1989

  30. [30]

    Basque (179) phoneme inventory,

    “Basque (179) phoneme inventory,” PHOIBLE 2.0, 2019, accessed: 2026-06-29. [Online]. Available: https://phoible.org/ inventories/view/179

    work page 2019