pith. sign in

arxiv: 2606.06183 · v1 · pith:4PM7J74Vnew · submitted 2026-06-04 · 📡 eess.AS · cs.CL

Revisiting Lexicon Evaluation in Unsupervised Word Discovery

Simon Malan , Danel Slabbert , Herman Kamper This is my paper

Pith reviewed 2026-06-27 23:32 UTC · model grok-4.3

classification 📡 eess.AS cs.CL
keywords zero-resource speechunsupervised word discoverylexicon evaluationclustering metricsnormalized edit distancecluster biasground-truth distribution
0
0 comments X

The pith

Two metrics reduce bias in unsupervised speech lexicon evaluation by weighting cluster size and tracking true-class spread.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

Standard normalized edit distance averages phoneme distances within clusters but inherently favors large clusters and ignores how true words are distributed across clusters. The paper proposes a modified metric that incorporates cluster size into the consistency score and an inverse metric that measures the spread of ground-truth classes. Experiments on synthetic and real-world lexicons show the pair together tracks similarity to the true distribution more closely and resists common evaluation skews. A sympathetic reader cares because zero-resource speech systems rely on these scores to decide which discovered units form usable lexicons. Without trustworthy metrics, progress on word discovery algorithms stalls on misleading signals.

Core claim

Normalized edit distance has an inherent bias toward the quality of large clusters and ignores how true classes are distributed across clusters. Based on clustering theory, a size-weighted metric for within-cluster consistency and an inverse metric for true-word spread across clusters are introduced. On synthetic and real-world lexicons these two metrics combined correlate more closely with ground-truth distribution similarity and prove more robust to the identified biases.

What carries the argument

Size-weighted within-cluster consistency metric paired with inverse true-class spread metric, which together adjust for cluster size imbalance and measure label distribution across clusters.

If this is right

  • Lexicon evaluations will no longer systematically favor outputs with a few oversized clusters over more balanced ones.
  • Comparisons across unsupervised discovery algorithms become less skewed by the size-bias of normalized edit distance.
  • Lexicons that match the ground-truth distribution more closely will receive higher combined scores even if they contain small clusters.
  • Evaluation pipelines can now separately diagnose within-cluster consistency and cross-cluster distribution problems.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • These metrics could be adapted to evaluate discovered units in other unsupervised audio or language tasks that rely on clustering.
  • Algorithms might be retrained or selected by directly optimizing the new combined score instead of normalized edit distance.
  • Theoretical analysis could quantify exactly how much cluster-size imbalance distorts rankings under different data regimes.

Load-bearing premise

The synthetic and real-world lexicons used in the experiments are representative enough for the correlation and robustness claims to generalize to other zero-resource speech datasets and discovery algorithms.

What would settle it

A new zero-resource dataset and discovery algorithm where the combined metrics produce rankings that diverge from actual ground-truth distribution similarity in the opposite direction from the reported correlations.

Figures

Figures reproduced from arXiv: 2606.06183 by Danel Slabbert, Herman Kamper, Simon Malan.

Figure 1
Figure 1. Figure 1: Four properties that a clustering evaluation metric should [PITH_FULL_IMAGE:figures/full_fig_p002_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: The problem of cluster matching [25]. A metric should [PITH_FULL_IMAGE:figures/full_fig_p005_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Evaluation metric scores for three unsupervised word [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: Evaluation metric scores for two synthetic lexicons [PITH_FULL_IMAGE:figures/full_fig_p006_6.png] view at source ↗
read the original abstract

Building a lexicon from discovered word-like units is a central goal in zero-resource speech processing. But do our evaluations provide a trustworthy indication of lexicon quality? A common metric, normalized edit distance, averages the phoneme edit distances between discovered units in each cluster. We show that this metric has an inherent bias toward the quality of large clusters, inhibiting fair evaluation. Moreover, it ignores how well true classes are distributed across clusters. Based on established theory in clustering literature, we propose two metrics that address these shortcomings: a modified metric that weighs cluster size when assessing within-cluster consistency, and an inverse metric that assesses how true words are spread across clusters. Through experiments on synthetic and real-world lexicons, we demonstrate that combined, these metrics are: (1) more closely correlated with how similar a lexicon is to the ground-truth distribution, and (2) more robust to biases that skew lexicon evaluations.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 0 minor

Summary. The manuscript claims that normalized edit distance (NED) for evaluating discovered lexicons in zero-resource speech processing is biased toward large clusters and ignores how true classes are distributed across clusters. Drawing on clustering theory, it proposes two alternative metrics—a size-weighted within-cluster consistency metric and an inverse spread metric—and reports that experiments on synthetic and real-world lexicons show the combined metrics are more closely correlated with ground-truth distribution similarity and more robust to evaluation biases than NED.

Significance. If the experimental claims hold, the work would supply more reliable evaluation tools for unsupervised word discovery, directly addressing documented biases in a core task of zero-resource speech processing. The explicit grounding in established clustering literature is a positive feature of the metric design.

major comments (2)
  1. [Abstract] Abstract: the claim that 'experiments on synthetic and real lexicons support the claims' is load-bearing for the central argument, yet the abstract supplies no quantitative results, error bars, correlation coefficients, or details on metric computation and baseline selection, preventing assessment of the reported improvements.
  2. [Experiments] Experiments section: the generalization that the two metrics are 'more robust to biases that skew lexicon evaluations' requires that the tested synthetic and real lexicons instantiate large-cluster dominance and uneven true-class distributions at frequencies and combinations representative of other discovery algorithms and corpora; no evidence or justification for this coverage is supplied.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for the constructive comments on our manuscript. We address each major point below and indicate the revisions planned for the next version.

read point-by-point responses

  1. Referee: [Abstract] Abstract: the claim that 'experiments on synthetic and real lexicons support the claims' is load-bearing for the central argument, yet the abstract supplies no quantitative results, error bars, correlation coefficients, or details on metric computation and baseline selection, preventing assessment of the reported improvements.

    Authors: We agree that the abstract would be strengthened by including quantitative support. The revised abstract will report the key correlation coefficients (e.g., between the combined metrics and ground-truth distribution similarity) and note the baseline comparisons used in the experiments. revision: yes

  2. Referee: [Experiments] Experiments section: the generalization that the two metrics are 'more robust to biases that skew lexicon evaluations' requires that the tested synthetic and real lexicons instantiate large-cluster dominance and uneven true-class distributions at frequencies and combinations representative of other discovery algorithms and corpora; no evidence or justification for this coverage is supplied.

    Authors: The synthetic lexicons were generated with explicit control over cluster-size distributions and class spreads to cover the bias scenarios identified in the clustering literature, while the real lexicons come from standard zero-resource corpora (e.g., TIMIT-derived units). We will add a short justification subsection explaining how these choices instantiate the relevant bias conditions at representative frequencies. revision: yes

Circularity Check

0 steps flagged

No significant circularity; metrics defined from clustering quantities and validated empirically

full rationale

The paper defines its two proposed metrics directly from cluster-size weighting and inverse class-spread quantities drawn from established clustering theory. It then reports empirical correlations with ground-truth distribution similarity on synthetic and real lexicons. No equations reduce a claimed prediction to a fitted parameter by construction, no self-citation chain is load-bearing for the central result, and no ansatz or uniqueness theorem is smuggled in. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

0 free parameters · 0 axioms · 0 invented entities

No free parameters, axioms, or invented entities are introduced; the work defines two new evaluation functions based on existing clustering concepts.

pith-pipeline@v0.9.1-grok · 5681 in / 1068 out tokens · 14239 ms · 2026-06-27T23:32:42.065181+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score.

  1. Recovering the Zipfian Distribution in Unsupervised Term Discovery

    eess.AS 2026-06 unverdicted novelty 6.0

    Graph clustering with Leiden recovers Zipfian distributions in unsupervised speech term discovery more effectively than K-means, GMM or BIRCH across three languages.

Reference graph

Works this paper leans on

32 extracted references · 1 canonical work pages · cited by 1 Pith paper

  1. [1]

    Unsupervised word acquisition from speech using pattern discovery,

    A. Park and J. Glass, “Unsupervised word acquisition from speech using pattern discovery,” inICASSP, 2006

    2006

  2. [2]

    Sylber: Syllabic embedding representation of speech from raw audio,

    C. J. Cho, N. Lee, A. Gupta, D. Agarwal, E. Chen, A. W. Black, and G. K. Anumanchipalli, “Sylber: Syllabic embedding representation of speech from raw audio,” inICLR, 2025

    2025

  3. [3]

    SyllableLM: Learning coarse semantic units for speech language models,

    A. Baade, P. Peng, and D. Harwath, “SyllableLM: Learning coarse semantic units for speech language models,” inICLR, 2025

    2025

  4. [4]

    The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units,

    E. Dunbar, J. Karadayi, M. Bernard, X.-N. Cao, R. Algayres, L. Ondel, L. Besacier, S. Sakti, and E. Dupoux, “The Zero Resource Speech Challenge 2020: Discovering discrete subword and word units,” in Interspeech, 2020

    2020

  5. [5]

    On generative spoken language modeling from raw audio,

    K. Lakhotia, E. Kharitonov, W.-N. Hsu, Y . Adi, A. Polyak, B. Bolte, T.-A. Nguyen, J. Copet, A. Baevski, A. Mohamed, and E. Dupoux, “On generative spoken language modeling from raw audio,”Transactions of the Association for Computational Linguistics, vol. 9, pp. 1336–1354, 2021

    2021

  6. [6]

    AudioLM: A language modeling approach to audio generation,

    Z. Borsos, R. Marinier, D. Vincent, E. Kharitonov, O. Pietquin, M. Sharifi, D. Roblek, O. Teboul, D. Grangier, M. Tagliasacchi, and N. Zeghi- dour, “AudioLM: A language modeling approach to audio generation,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 31, pp. 2523–2533, 2023

    2023

  7. [7]

    On the landscape of spoken language models: A comprehensive survey,

    S. Arora, K.-W. Chang, C.-M. Chien, Y . Peng, H. Wu, Y . Adi, E. Dupoux, H. yi Lee, K. Livescu, and S. Watanabe, “On the landscape of spoken language models: A comprehensive survey,”Transactions on Machine Learning Research, 2025

    2025

  8. [8]

    Criterion functions for document clustering: Experiments and analysis,

    Y . Zhao and G. Karypis, “Criterion functions for document clustering: Experiments and analysis,”Technical Report TR 01–40, Department of Computer Science, University of Minnesota, 2001

    2001

  9. [9]

    V-measure: A conditional entropy- based external cluster evaluation measure,

    A. Rosenberg and J. Hirschberg, “V-measure: A conditional entropy- based external cluster evaluation measure,” inEMNLP-CoNLL, 2007

    2007

  10. [10]

    Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems,

    B. Ludusan, M. Versteegh, A. Jansen, G. Gravier, X.-N. Cao, M. Johnson, and E. Dupoux, “Bridging the gap between speech technology and natural language processing: an evaluation toolbox for term discovery systems,” inLREC, 2014

    2014

  11. [11]

    Unsupervised word discovery from speech using automatic segmentation into syllable-like units,

    O. J. R ¨as¨anen, G. Doyle, and M. Frank, “Unsupervised word discovery from speech using automatic segmentation into syllable-like units,” in Interspeech, 2015

    2015

  12. [12]

    A segmental framework for fully-unsupervised large-vocabulary speech recognition,

    H. Kamper, A. Jansen, and S. Goldwater, “A segmental framework for fully-unsupervised large-vocabulary speech recognition,”Computer Speech & Language, vol. 46, pp. 154–174, 2017

    2017

  13. [13]

    An embedded segmental K-means model for unsupervised segmentation and clustering of speech,

    H. Kamper, K. Livescu, and S. Goldwater, “An embedded segmental K-means model for unsupervised segmentation and clustering of speech,” inASRU, 2017

    2017

  14. [14]

    Unsupervised discovery of recurring speech patterns using probabilistic adaptive metrics,

    O. R¨as¨anen and M. A. C. Bland ´on, “Unsupervised discovery of recurring speech patterns using probabilistic adaptive metrics,” inInterspeech, 2020

    2020

  15. [15]

    Self-expressing autoencoders for unsupervised spoken term discovery,

    S. Bhati, J. Villalba, P. ˙Zelasko, and N. Dehak, “Self-expressing autoencoders for unsupervised spoken term discovery,” inInterspeech, 2020

    2020

  16. [16]

    Word discovery in visually grounded, self- supervised speech models,

    P. Peng and D. Harwath, “Word discovery in visually grounded, self- supervised speech models,” inInterspeech, 2022

    2022

  17. [17]

    Spoken- term discovery using discrete speech units,

    B. van Niekerk, J. Za ¨ıdi, M.-A. Carbonneau, and H. Kamper, “Spoken- term discovery using discrete speech units,” inInterspeech, 2024

    2024

  18. [18]

    Revisiting speech segmentation and lexicon learning with better features,

    H. Kamper and B. van Niekerk, “Revisiting speech segmentation and lexicon learning with better features,”arXiv preprint arXiv:2401.17902, 2024

    work page arXiv 2024

  19. [19]

    An information-theoretic external cluster-validity measure,

    B. E. Dom, “An information-theoretic external cluster-validity measure,” Technical Report RJ10219, IBM, 2001

    2001

  20. [20]

    Comparing clusterings by the variation of information,

    M. Meil ˘a, “Comparing clusterings by the variation of information,” in COLT, 2003

    2003

  21. [21]

    A comparison of extrinsic clustering evaluation metrics based on formal constraints,

    E. Amig ´o, J. Gonzalo, J. Artiles, and F. Verdejo, “A comparison of extrinsic clustering evaluation metrics based on formal constraints,” Information Retrieval, vol. 12, pp. 461–486, 2009

    2009

  22. [22]

    Unsupervised pattern discovery in speech,

    A. S. Park and J. R. Glass, “Unsupervised pattern discovery in speech,” IEEE/ACM Transactions on Audio, Speech, and Language Processing, vol. 16, pp. 186–197, 2008

    2008

  23. [23]

    What do self- supervised speech models know about words?

    A. Pasad, C.-M. Chien, S. Settle, and K. Livescu, “What do self- supervised speech models know about words?”Transactions of the Association for Computational Linguistics, vol. 12, pp. 372–391, 2024

    2024

  24. [24]

    Should top-down clustering affect boundaries in unsupervised word discovery?

    S. Malan, B. van Niekerk, and H. Kamper, “Should top-down clustering affect boundaries in unsupervised word discovery?”IEEE Transactions on Audio, Speech and Language Processing, vol. 34, pp. 1163–1172, 2026

    2026

  25. [25]

    Comparing clusterings–an information based distance,

    M. Meil ˘a, “Comparing clusterings–an information based distance,” Journal of multivariate analysis, vol. 98, pp. 873–895, 2007

    2007

  26. [26]

    LibriSpeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” inICASSP, 2015

    2015

  27. [27]

    Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,

    M. McAuliffe, M. Socolof, S. Mihuc, M. Wagner, and M. Sonderegger, “Montreal Forced Aligner: Trainable text-speech alignment using Kaldi,” inInterspeech, 2017

    2017

  28. [28]

    k-means++: the advantages of careful seeding,

    D. Arthur and S. Vassilvitskii, “k-means++: the advantages of careful seeding,” inSODA, 2007

    2007

  29. [29]

    Community detection in graphs,

    S. Fortunato, “Community detection in graphs,”Physics Reports, vol. 486, pp. 75–174, 2010

    2010

  30. [30]

    Unsupervised lexicon learning from speech is limited by representations rather than clustering,

    D. Slabbert, S. Malan, and H. Kamper, “Unsupervised lexicon learning from speech is limited by representations rather than clustering,” in ICASSP, 2026

    2026

  31. [31]

    Nielsen,Introduction to HPC with MPI for Data Science

    F. Nielsen,Introduction to HPC with MPI for Data Science. Springer, 2016, ch. Hierarchical Clustering, pp. 195–211

    2016

  32. [32]

    Unsupervised word discovery: Boundary detection with clustering vs. dynamic programming,

    S. Malan, B. van Niekerk, and H. Kamper, “Unsupervised word discovery: Boundary detection with clustering vs. dynamic programming,” in ICASSP, 2025

    2025