{"id":"4541cc48-d703-4817-9570-7ead9cb5db73","arxiv_id":"2507.02176","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":6.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":3,"one_line_summary":"ASV embeddings used to judge whether synthesized speech matches a target speaker mostly encode static spectral traits and miss rhythm, so the authors introduce U3D, a duration-distribution metric for speaker rhythm.","lead":"This paper examines what automatic speaker verification embeddings actually capture about a voice, and finds they mostly encode static traits like timbre and pitch, while missing dynamic markers like rhythm. It proposes a new metric, U3D, that measures rhythm by comparing the durations of phoneme-like speech units, and gives protocol advice to avoid confounds in similarity tests.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Probe analysis may understate dynamic marker encoding: per-utterance R2 conflates within-speaker content variance with representational neglect.","rationale":"The reader identified the probe assumption as the weakest point, and I agree that it is the most load-bearing. The central claim that ASV embeddings neglect dynamic rhythm rests entirely on Figure 1's per-utterance R2 values. The unaddressed confound is within-speaker variance: many of the 'dynamic' features are computed over short utterances and are strongly content-dependent, so low R2 may reflect label noise rather than missing speaker information. A speaker-level probe directly tests this. If the concern lands, the paper's motivation for U3D weakens substantially; if it does not, the claim is strengthened. The U3D validation issues (no statistical tests, no strong baselines) are real but secondary because they affect the proposed remedy rather than the core finding about ASV embeddings. Since the reader already returned CONDITIONAL, my read does not change the verdict category, but it sharpens the specific condition that should be met.","tokens_in":11659,"tokens_out":4838,"duration_ms":52999,"concrete_test":"Compute speaker-level mean embeddings and speaker-level mean features for all utterances in ARCTIC and L2-ARCTIC. Train a lasso (and random forest) to predict the per-speaker mean of speech rate, voiced segment length, and pitch std from per-speaker mean embeddings, using leave-speaker-out cross-validation. If R2 for these dynamic features is high (e.g., >0.6) while per-utterance R2 is low, the conclusion of neglect is confounded by within-speaker variance and the central claim fails; if the R2 remains near zero, the claim is supported. Report ICC for each feature to quantify the ceiling imposed by utterance-level noise.","verdict_should_be":"UNCHANGED","load_bearing_attack":"Section 3.2 concludes from low per-utterance R2 values in Figure 1 that ASV embeddings neglect dynamic markers such as rhythm. This inference assumes the selected handcrafted features are reliable, speaker-dependent targets at the utterance level. But features like speech rate, voiced/unvoiced segment lengths, and pitch std vary substantially across utterances of the same speaker due to phonetic content and expressive choices. If the target feature has high within-speaker variance, even a perfect speaker-level encoding will yield low per-utterance R2 because the regressor can only predict content-dependent noise. The paper does not report between-speaker vs. within-speaker variance for these features, nor does it aggregate to speaker-level means before probing. Consequently, the central claim that ASV embeddings 'neglect dynamic elements such as rhythm' may be an artifact of the probing granularity rather than a property of the embeddings. This is load-bearing because the entire motivation for U3D depends on ASV embeddings missing rhythm; if a speaker-level probe reveals that embeddings encode rhythm, the proposed metric loses its rationale.","agreement_with_reader":"agree"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates what automatic speaker verification (ASV) embeddings encode about voice identity, motivated by their widespread use in speech-synthesis evaluation. It reports a probe analysis in which lasso regressors predict handcrafted acoustic/prosodic features from ASV embeddings (Section 3.2, Figure 1), concluding that static spectral features such as timbre and pitch range are well represented while dynamic features such as speech rate and segment durations are not. It then measures equal error rates under duration-based sorting, additive noise, and spectral equalization (Section 3.3, Table 2), showing that these confounds can distort ASV-based similarity assessments. Finally, it proposes U3D, a Wasserstein distance between duration distributions of phoneme-like groups discovered from self-supervised speech units, and validates it on ARCTIC and L2-ARCTIC (Section 3.4, Table 3). The paper argues that U3D complements ASV embeddings by capturing rhythm-related aspects of speaker identity.","tokens_in":11851,"tokens_out":4879,"duration_ms":56509,"significance":"If the central conclusions hold, the paper provides a practically important caution for the speech-synthesis community: ASV-based speaker similarity metrics can be sensitive to duration and channel confounds and may under-represent behavioral/rhythmic identity. The confounding-factor experiments in Table 2 are carefully controlled, with per-speaker cross-validation and expected 50% EER baselines. The proposed U3D is simple, unsupervised, and open-sourced, and the confirmation using both forced-aligned and unsupervised units in Table 3 is a strength. However, the main empirical support for the 'neglect' claim requires methodological tightening: the probe evaluation protocol in Section 3.2 is ambiguous regarding held-out data, and the interpretation of per-utterance R² values as evidence of representational neglect is not fully justified without speaker-level variance analysis. The U3D validation in Table 3 also lacks variance information and significance tests.","major_comments":[{"comment":"The probe evaluation protocol is not fully specified. The text says hyperparameters are selected by 5-fold cross-validation with speaker-disjoint folds, but then 'after retraining the regressor on all data, we predict the value of each handcrafted feature' and compute R². If the final model is evaluated on the same data used to fit it, the reported R² values are in-sample and do not estimate how much information the embeddings generalize to new utterances. Please clarify whether Figure 1 reports held-out predictions, and if not, report R² on held-out utterances using the cross-validation folds.","section":"Section 3.2, Figure 1"},{"comment":"Interpreting low per-utterance R² as evidence that ASV embeddings 'neglect' dynamic markers is not justified without quantifying within-speaker versus between-speaker variance of the target features. Features such as speech rate, voiced/unvoiced segment durations, and pitch std vary across utterances of the same speaker due to phonetic content and speaking style; a speaker-level encoding can yield low per-utterance R² even if it encodes the speaker-level dynamic profile. Please add speaker-level analyses, for example by regressing on speaker-averaged embeddings and features, or by reporting variance components (e.g., ICC) for each target feature, to support the central claim.","section":"Section 3.2, Figure 1"},{"comment":"The claim that rhythm distances are 'significantly larger' between different speakers is not supported by the reported statistics. Table 3 gives only average Wasserstein distances with no per-speaker variance, confidence intervals, or significance tests. Since the Same, Nearest, and Random conditions involve different dependency structures (random splits within a speaker versus pairs of speakers), please report per-speaker standard errors and a paired or permutation test comparing Same against Nearest and Random, and for the forced-aligned versus unsupervised U3D comparison.","section":"Section 3.4, Table 3"}],"minor_comments":[{"comment":"The claim that U3D is 'language agnostic' is supported only by the absence of forced-alignment tools, not by multilingual experiments; all validation is on English (ARCTIC and L2-ARCTIC). Please qualify the claim to 'does not require language-specific phonetic annotations' or add a non-English dataset.","section":"Section 3.4 and Abstract"},{"comment":"The R² values are point estimates with no uncertainty. Adding confidence intervals across speakers or cross-validation folds would make comparisons across embeddings more interpretable.","section":"Figure 1"},{"comment":"The caption should spell out 'approx.' (approximants), 'fric.' (fricatives), and 'sil.' (silence), and should state whether the 'Nearest' condition excludes the same speaker and how many pairs are used.","section":"Table 3"},{"comment":"Please specify whether the IQR outlier removal is applied to the handcrafted features, the embeddings, or both, and how many samples are removed; this affects reproducibility.","section":"Section 3.2"},{"comment":"There is a typo in 'dynamic aspects of a identity' near the end of Section 4; it should read 'of identity'.","section":"Conclusion"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the core idea is timely. The confounding-factor experiments are a solid contribution, and the U3D metric is plausible and clearly motivated. The main risk is the Section 3.2 probe analysis: the evaluation protocol needs an explicit held-out split, and the per-utterance R² interpretation needs speaker-level variance analysis to support the 'neglect' claim. These are fixable with additional experiments, so I recommend major revision rather than rejection."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Worth a read, with caveats. The paper does two concrete things: it probes what ASV embeddings encode across seven models, and it shows how duration, noise, and equalization can corrupt speaker similarity judgments. The EER experiments in Section 3.3 are well done—controls are sensible, and the re-equalization fix genuinely restores chance-level EER. That part is a practical contribution for anyone who reports speaker similarity in TTS or voice conversion.\n\nThe probe in Section 3.2 is the weakest link. The stress-test note is fair: regressing per-utterance handcrafted features on embeddings conflates within-speaker content-driven variance with representational neglect. Speech rate and voiced/unvoiced lengths vary a lot across utterances of the same speaker, so low R2 doesn't necessarily mean the embedding carries no speaker-level rhythm information. The paper should aggregate embeddings and features to speaker means (or at least report the variance decomposition) before claiming that dynamic markers are neglected. I don't think the conclusion is false—X-vectors and ECAPA are known to emphasize timbre and pitch—but the abstract oversells the evidence.\n\nU3D is a fresh idea: duration distributions over SSL units, measured with Wasserstein distance. But Table 3 has no error bars, no significance tests, and the nearest-by-syllable-rate comparison only shows that the distances are larger, not that they are statistically separable. Also, calling it language-agnostic is premature—both ARCTIC and L2-ARCTIC are English. No comparison against established rhythm metrics (PVI, VarcoV) either, which would contextualize the added value. The reuse of [59] is transparent and fine.\n\nOverall, the paper is worth a serious referee. The confound analysis alone justifies the review, and U3D deserves a chance to be strengthened. I'd recommend acceptance only after the probe granularity is addressed and the validation table gets proper statistics. Bring it to the reading group if you care about TTS evaluation.","headline":"A useful confound analysis and a plausible but under-validated rhythm metric; the probe evidence for neglect is softer than stated.","tokens_in":12395,"tokens_out":4312,"would_cite":true,"duration_ms":49126,"reading_group":"maybe","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"Widely used speaker-verification embeddings capture timbre and pitch but ignore rhythm, and the paper's U3D metric fills that gap.","keywords":["speaker similarity","automatic speaker verification","speech synthesis evaluation","voice identity","rhythm","self-supervised speech units","duration distribution distance","confounding factors"],"falsifier":"Re-run the same lasso probe on a corpus of spontaneous or conversational speech and add dynamic features like pause durations, F0 contour shape, and articulatory rate; if the $R^2$ for these dynamic features rises well above the 0.0–0.4 range reported here, the claim that ASV embeddings neglect dynamics would be contradicted.","tokens_in":11474,"feed_emoji":"🎙️","tokens_out":7659,"duration_ms":78219,"temperature":0.7,"pith_summary":"This paper asks what automatic speaker-verification embeddings actually capture when speech-synthesis researchers use them to measure whether a generated voice matches a target speaker. The answer it argues for is that these embeddings mostly encode static, anatomical voice markers—mean pitch, timbre, and voice quality—while largely missing dynamic behavioral markers such as rhythm and speech-rate patterns. It also shows that the embeddings respond to file duration, noise, and frequency equalization, so similarity experiments can be biased by recording conditions rather than by identity. To fill the gap, the paper introduces U3D, a distance between duration distributions of speech-unit groups, which separates speakers even when their syllable rates are nearly identical. If right, this means the standard ASV-based similarity score is an incomplete identity measure and should be complemented by rhythm-aware metrics like U3D.","feed_headline":"Rhythm is missing from voice-similarity scores; U3D adds it","feed_subtitle":"Speaker-similarity scores capture timbre and pitch, not rhythm; U3D measures the missing dynamics.","key_machinery":"There are two load-bearing mechanisms. The first is a probe: lasso regressors trained per embedding model to predict 14 handcrafted phonetic features from each utterance embedding, with the coefficient of determination $R^2$ used as the measure of what the embedding encodes. The second is U3D (Unit Duration Distribution Distance), computed in three steps: cluster self-supervised speech units into broad phoneme-like groups using agglomerative hierarchical clustering; segment each utterance into contiguous same-group runs and record run durations; then compute the Wasserstein distance between genuine and synthesized duration distributions per group and average over groups. U3D is validated both with forced-alignment phoneme groups and with fully unsupervised speech-unit groups, and its key property is that it separates speakers who are nearest neighbors in syllable rate.","core_discovery":"The paper's central claim is that widely used ASV embeddings, trained to discriminate between speakers, do not characterize a voice fully: the probe experiments show high $R^2$ values (0.6–0.94) for mean pitch, HNR, shimmer, and $\\alpha$-ratio, alongside low $R^2$ values (mostly 0.0–0.4) for speech rate, voiced/unvoiced segment lengths, and pitch and loudness variability. In the same experiments, file duration is partially encoded, and follow-up EER tests confirm that the same speaker's utterances can be told apart by duration, noise level, and equalization, with EER falling from the expected 50% to as low as 1–5% under heavy noise or spectral-color changes. The proposed U3D metric instead compares the distribution of durations of contiguous same-unit segments in broad phoneme-like groups learned from self-supervised speech units, and shows that speakers who are nearest neighbors by syllable rate still have clearly larger U3D distances than same-speaker splits. The paper therefore claims that rhythm is a measurable, content-aware part of speaker identity that current ASV evaluations omit.","pith_inferences":["A natural extension not pursued in the paper: combine ASV cosine similarity with U3D into a single two-axis similarity report, since the probe results show the two capture largely separate information.","U3D could serve not only as an evaluator but as a training objective or generation-time filter for voice-conversion and text-to-speech systems, since it needs no phonetic annotations.","The duration-bias finding implies published EER comparisons between synthesis systems may need revisiting whenever compared systems produce utterances of different lengths or noise levels.","A testable extension is computing U3D at several levels of the speech-unit hierarchy to capture pause patterns and prosodic phrasing, which may separate speakers even more sharply than phoneme-group durations."],"forward_implications":["Speaker-similarity scores built on ASV embeddings will systematically understate differences in rhythm and dynamic prosody, so two voices can look equally similar to a target even when one has the wrong timing.","Duration mismatch between synthesized and genuine audio can push EER-based similarity numbers down for reasons unrelated to identity; evaluations should match utterance durations, ideally generating the same text as the reference.","Noise and spectral equalization can collapse EER from 50% to near 0% for the same speaker, so reported similarity differences across systems with different output quality should be interpreted with caution; re-equalizing to the reference spectral balance restores the expected 50%.","U3D distinguishes different speakers even when syllable rate cannot, and the unsupervised version needs no forced aligner, so it offers a language-agnostic rhythm score for synthesis evaluation.","Older X-Vector embeddings encode a broader set of markers, including dynamic ones, than newer ECAPA or WavLM-based embeddings, so better verification accuracy does not imply richer identity characterization for synthesis."],"supporting_citations":[{"why":"Defines X-Vectors, one of the evaluated embeddings, and the one found to encode more dynamic markers than newer models.","marker":"[9]"},{"why":"Defines ECAPA-TDNN, a state-of-the-art ASV model used throughout the probe and EER experiments.","marker":"[10]"},{"why":"Defines GE2E, a contrastively trained embedding whose catastrophic sensitivity to equalization is reported.","marker":"[12]"},{"why":"Established that ASV embedding distances correlate with human voice-similarity judgments, the rationale for their use in synthesis evaluation.","marker":"[14]"},{"why":"Provides the WavLM self-supervised features used by several of the evaluated embedding models.","marker":"[52]"},{"why":"Supplies the speech-unit clustering and segmentation score that U3D builds on for unsupervised rhythm modeling.","marker":"[59]"},{"why":"Provides the discrete and soft speech units from which the U3D phoneme-like groups are derived.","marker":"[23]"},{"why":"Contributes the LibriSpeech speakers used to broaden the probe experiment beyond the ARCTIC datasets.","marker":"[55]"},{"why":"Prior work finding X-Vectors encode a wider range of markers than newer embeddings, corroborated by the probe results.","marker":"[61]"}],"fun_headline_variants":["ASV embeddings ignore rhythm; U3D measures it","Voice similarity misses rhythm; U3D fills the gap","U3D: a metric for the rhythm missing from speaker scores","Speaker similarity lacks rhythm; U3D adds it"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The probe conclusion that ASV embeddings neglect dynamics assumes that low $R^2$ from a lasso regressor predicting handcrafted features truly reflects what the embeddings encode.","fun_headline_variants_meta":{"raw":{"variants":["ASV embeddings ignore rhythm; U3D measures it","Voice similarity misses rhythm; U3D fills the gap","U3D: a metric for the rhythm missing from speaker scores","Speaker similarity lacks rhythm; U3D adds it"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000151,"raw_usage":{"total_tokens":1178,"prompt_tokens":901,"completion_tokens":277,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":517,"completion_tokens_details":{"reasoning_tokens":209}},"tokens_in":517,"tokens_out":277,"duration_ms":3584,"temperature":1.0,"reasoning_tokens":209,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-06T20:35:33.358181+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Re-run the same lasso probe on a corpus of spontaneous or conversational speech and add dynamic features like pause durations, F0 contour shape, and articulatory rate; if the $R^2$ for these dynamic features rises well above the 0.0–0.4 range reported here, the claim that ASV embeddings neglect dynamics would be contradicted.","supporting_citations":[{"cited_title":"Dreambooth: Fine tuning text-to-image d iffusion models for subject-driven generation,","cited_arxiv_id":null,"evidence_quote":"Defines X-Vectors, one of the evaluated embeddings, and the one found to encode more dynamic markers than newer models."},{"cited_title":"Generat ing video game scripts with style,","cited_arxiv_id":null,"evidence_quote":"Defines GE2E, a contrastively trained embedding whose catastrophic sensitivity to equalization is reported."},{"cited_title":"Information conveyed by vow- els","cited_arxiv_id":null,"evidence_quote":"Established that ASV embedding distances correlate with human voice-similarity judgments, the rationale for their use in synthesis evaluation."},{"cited_title":"MOS vs . AB: evaluating text-to-speech systems reliably using cluster ed stan- dard errors,","cited_arxiv_id":null,"evidence_quote":"Provides the WavLM self-supervised features used by several of the evaluated embedding models."},{"cited_title":"WavLM: large-scale self-supervised pr e-training for full stack speech processing,","cited_arxiv_id":null,"evidence_quote":"Supplies the speech-unit clustering and segmentation score that U3D builds on for unsupervised rhythm modeling."},{"cited_title":"Zero-shot multi-speaker text-to-sp eech with state-of-the-art neural speaker embeddings,","cited_arxiv_id":null,"evidence_quote":"Provides the discrete and soft speech units from which the U3D phoneme-like groups are derived."},{"cited_title":"Automatic evaluation of speaker simila rity,","cited_arxiv_id":null,"evidence_quote":"Contributes the LibriSpeech speakers used to broaden the probe experiment beyond the ARCTIC datasets."},{"cited_title":"L2-ARCTIC: a non-native english speech c orpus,","cited_arxiv_id":null,"evidence_quote":"Prior work finding X-Vectors encode a wider range of markers than newer embeddings, corroborated by the probe results."}],"review_version":1}