pith. sign in

arxiv: 2606.10781 · v1 · pith:VO7VLRDWnew · submitted 2026-06-09 · 📡 eess.AS · cs.CL

Recovering the Zipfian Distribution in Unsupervised Term Discovery

Danel Slabbert , Simon Malan , Herman Kamper This is my paper

Pith reviewed 2026-06-27 11:35 UTC · model grok-4.3

classification 📡 eess.AS cs.CL
keywords unsupervised term discoverygraph clusteringZipfian distributionspeech segmentationlexicon discoveryLeiden algorithmK-means
0
0 comments X

The pith

Graph clustering on pairwise similarities produces more Zipf-like lexicons than centre-based methods in unsupervised term discovery.

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The paper aims to show that graph-based clustering outperforms centre-based methods like K-means in unsupervised term discovery from speech. It segments audio into units, embeds them, builds a similarity graph, and uses the Leiden algorithm to cluster into a lexicon. This produces frequency distributions closer to the Zipfian shape observed in real lexicons, across word and syllable levels in three languages. Centre-based methods tend to create more uniform distributions due to their bias toward spherical clusters. The result challenges the standard use of K-means and suggests graph methods as a better alternative for lexicon discovery.

Core claim

By connecting segment embeddings through pairwise similarity and partitioning the resulting graph with the Leiden algorithm, graph clustering generates lexicons whose type frequencies more closely match Zipf's law than those produced by K-means, GMM, or BIRCH, in both word-level and syllable-level discovery tasks evaluated on three languages.

What carries the argument

Graph clustering via the Leiden algorithm applied to a pairwise similarity graph of speech segment embeddings.

If this is right

  • Graph clustering yields lexicons with frequency distributions closer to Zipfian than centre-based clustering.
  • The advantage appears at both word and syllable levels.
  • The improvement holds across three languages.
  • Agglomerative clustering with average linkage also produces good results but is less efficient.
  • The work questions the dominance of centre-based clustering for this task.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • Better Zipfian recovery could lead to improved performance in downstream speech processing applications that rely on natural lexicon statistics.
  • The method might be extended to other unsupervised discovery tasks beyond speech.
  • Controlling the distribution via graph methods could allow tuning for specific downstream needs.

Load-bearing premise

True lexicons follow a Zipfian distribution, so producing similar distributions indicates better discovery.

What would settle it

A direct comparison showing that a centre-based method achieves a higher Zipf similarity metric than graph clustering on the same datasets would falsify the superiority claim.

Figures

Figures reproduced from arXiv: 2606.10781 by Danel Slabbert, Herman Kamper, Simon Malan.

Figure 1
Figure 1. Figure 1: True word-type distribution vs K-means cluster frequencies over [PITH_FULL_IMAGE:figures/full_fig_p001_1.png] view at source ↗
Figure 3
Figure 3. Figure 3: Graph construction using similarity threshold [PITH_FULL_IMAGE:figures/full_fig_p002_3.png] view at source ↗
Figure 4
Figure 4. Figure 4: Type-frequency distributions for induced lexicons on English across the three segmentation conditions. Dashed lines correspond to [PITH_FULL_IMAGE:figures/full_fig_p005_4.png] view at source ↗
Figure 6
Figure 6. Figure 6: NES, iNES, and F1NES at different thresholds τ for true English word segments. flatter, more uniform distributions better suited to syllables; lower τ produces denser graphs whose distributions more closely follow the true word distribution (consistent with optimal τ settings from development; see Sec. III). The same trend holds for γ (Fig. 5b), but γ’s ability to approximate the true distribution is bound… view at source ↗
Figure 5
Figure 5. Figure 5: Effect of the (a) threshold τ and (b) resolution γ graph clustering hyperparameters on the type-frequency distribution for true English word segments. segments. Agglomerative clustering also has only a single setting: determining when to cut the tree, while graph clustering gives more explicit control over the resulting distributions through its two hyperparameters. Distributions induced through graph clus… view at source ↗
Figure 7
Figure 7. Figure 7: Type-frequency distributions for induced lexicons on Afrikaans [PITH_FULL_IMAGE:figures/full_fig_p006_7.png] view at source ↗
read the original abstract

Unsupervised term discovery involves segmenting unlabelled speech into word- or syllable-like units and clustering these into a lexicon of candidate types. True lexicons follow a Zipfian distribution, yet the dominant centre-based clustering approach -- K-means -- produces a more uniform distribution due to an inductive bias toward spherical clusters. In this paper we revisit graph-based clustering as a bottom-up alternative, where segment embeddings are connected by pairwise similarity and partitioned using the Leiden algorithm. We show that graph clustering substantially outperforms centre-based approaches (K-means, GMM, BIRCH) in both word- and syllable-level lexicon discovery across three languages, producing more Zipf-like distributions. Another bottom-up approach, agglomerative clustering with average linkage, also performs well, although it is computationally less efficient and allows for less control over the resulting distribution. Our work calls into question the dominance of centre-based clustering for term discovery, and promotes graph clustering as an attractive alternative.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit. Tearing a paper down is the easy half of reading it; the pith above is the substance, this is the friction.

Referee Report

2 major / 2 minor

Summary. The paper claims that centre-based clustering methods (K-means, GMM, BIRCH) in unsupervised term discovery from speech produce overly uniform distributions due to their inductive bias, while graph-based clustering via the Leiden algorithm on pairwise segment embedding similarities recovers more Zipf-like distributions. It reports that this approach substantially outperforms the centre-based baselines (and performs competitively with agglomerative clustering) at both word- and syllable-level lexicon discovery across three languages.

Significance. If the results hold, the work is significant for term discovery research because it directly addresses a known mismatch between standard clustering assumptions and the Zipfian nature of natural lexicons, providing concrete empirical comparisons across languages and granularity levels. The explicit focus on distribution shape as an evaluation axis, rather than solely on segmentation metrics, offers a useful diagnostic lens even if downstream validation is still needed.

major comments (2)
  1. [Abstract and §4] Abstract and §4 (Experiments): the central claim of superiority rests entirely on producing more Zipf-like distributions, yet no correlation is demonstrated between the chosen Zipf metric (power-law exponent or KS statistic) and standard term-discovery quality measures such as boundary F-score, type precision/recall against gold lexicons, or any downstream task performance. Without this link the practical advantage over K-means/GMM/BIRCH remains unvalidated.
  2. [§3.2] §3.2 (Graph construction): the procedure for building the similarity graph from segment embeddings (e.g., choice of k for k-NN, similarity threshold, or embedding dimensionality) is described at a high level only; the specific settings used for the Leiden runs that produce the reported Zipf improvements must be stated explicitly, as small changes in graph density can alter cluster-size distributions.
minor comments (2)
  1. [Table 1 and Figure 2] Table 1 and Figure 2: axis labels and legends should explicitly state the exact Zipf-likeness statistic being plotted (e.g., “KS distance to fitted power law”) rather than generic “Zipf score.”
  2. [§2] §2 (Related work): the discussion of prior term-discovery pipelines could usefully cite the specific clustering variants used in the most recent zero-resource speech benchmarks for direct comparison.

Simulated Author's Rebuttal

2 responses · 0 unresolved

We thank the referee for their detailed review and constructive comments on our manuscript. We address each of the major comments below.

read point-by-point responses

  1. Referee: [Abstract and §4] Abstract and §4 (Experiments): the central claim of superiority rests entirely on producing more Zipf-like distributions, yet no correlation is demonstrated between the chosen Zipf metric (power-law exponent or KS statistic) and standard term-discovery quality measures such as boundary F-score, type precision/recall against gold lexicons, or any downstream task performance. Without this link the practical advantage over K-means/GMM/BIRCH remains unvalidated.

    Authors: The paper's primary contribution is to highlight and address the mismatch between the inductive biases of centre-based clustering and the Zipfian nature of natural lexicons, using the distribution shape as a diagnostic. While a direct empirical correlation between the Zipf metrics and standard term discovery metrics is not provided in the current version, we argue that recovering the correct distribution is valuable in its own right as it better matches the statistical properties of language. We will revise §4 and the discussion to better articulate this point and include any available correlations with boundary F-score and type metrics from our experiments. revision: partial

  2. Referee: [§3.2] §3.2 (Graph construction): the procedure for building the similarity graph from segment embeddings (e.g., choice of k for k-NN, similarity threshold, or embedding dimensionality) is described at a high level only; the specific settings used for the Leiden runs that produce the reported Zipf improvements must be stated explicitly, as small changes in graph density can alter cluster-size distributions.

    Authors: We agree that explicit parameter values are necessary for reproducibility. The revised manuscript will include the specific choices for k-NN, similarity thresholds, embedding dimensionality, and other graph construction details used in our experiments. revision: yes

Circularity Check

0 steps flagged

No circularity: empirical comparison of clustering methods with no derivations or self-referential reductions

full rationale

The manuscript is an empirical evaluation of existing clustering algorithms (graph-based Leiden, K-means, GMM, BIRCH, agglomerative) on unsupervised term discovery across three languages. Performance is assessed by how closely induced lexicons match a Zipfian distribution, motivated by the external premise that true lexicons are Zipfian. No equations, fitted parameters renamed as predictions, self-citations used as load-bearing uniqueness theorems, or ansatzes smuggled via prior work appear in the provided text. The central claim rests on direct experimental measurements rather than any derivation that reduces to its own inputs by construction. This is a standard non-circular empirical study.

Axiom & Free-Parameter Ledger

0 free parameters · 1 axioms · 0 invented entities

The work rests on the domain assumption that natural lexicons are Zipfian; no free parameters or invented entities are mentioned in the abstract. Full paper would likely introduce resolution parameters for Leiden and similarity thresholds.

axioms (1)
  • domain assumption True lexicons follow a Zipfian distribution
    Explicitly invoked in the abstract as the reason centre-based clustering is inadequate.

pith-pipeline@v0.9.1-grok · 5691 in / 1193 out tokens · 28952 ms · 2026-06-27T11:35:50.237200+00:00 · methodology

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

55 extracted references · 2 canonical work pages · 1 internal anchor

  1. [1]

    Bridging the gap between speech technology and natural language processing: An evaluation toolbox for term discovery systems,

    B. Ludusanet al., “Bridging the gap between speech technology and natural language processing: An evaluation toolbox for term discovery systems,” inLREC, 2014

    2014

  2. [2]

    Statistical learning by 8-month-old infants,

    J. R. Saffran, R. N. Aslin, and E. L. Newport, “Statistical learning by 8-month-old infants,”Science, 1996

    1996

  3. [3]

    Some critical developments in acquiring native language sound organization during the first year,

    P. W. Jusczyk, “Some critical developments in acquiring native language sound organization during the first year,”The Annals of Otology, Rhinology and Laryngology, 2002

    2002

  4. [4]

    At 6-9 months, human infants know the meanings of many common nouns,

    E. Bergelson and D. Swingley, “At 6-9 months, human infants know the meanings of many common nouns,”Proceedings of the National Academy of Science, 2012

    2012

  5. [5]

    Computational modeling of phonetic and lexical learning in early language acquisition: Existing models and future directions,

    O. R ¨as¨anen, “Computational modeling of phonetic and lexical learning in early language acquisition: Existing models and future directions,” Speech Communication, 2012

    2012

  6. [6]

    Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner,

    E. Dupoux, “Cognitive science in the era of artificial intelligence: A roadmap for reverse-engineering the infant language-learner,”Cognition, 2018

    2018

  7. [7]

    Automatic speech recognition for under-resourced languages: A survey,

    L. Besacier, E. Barnard, A. Karpov, and T. Schultz, “Automatic speech recognition for under-resourced languages: A survey,”Speech Communication, 2014

    2014

  8. [8]

    Unsupervised pattern discovery in speech,

    A. S. Park and J. R. Glass, “Unsupervised pattern discovery in speech,” IEEE Transactions on Audio, Speech and Language Processing, 2008

    2008

  9. [9]

    An evaluation of graph clustering methods for unsupervised term discovery,

    V . Lyzinski, G. Sell, and A. Jansen, “An evaluation of graph clustering methods for unsupervised term discovery,” inInterspeech, 2015

    2015

  10. [10]

    HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,

    W. N. Hsuet al., “HuBERT: Self-supervised speech representation learning by masked prediction of hidden units,”IEEE/ACM Transactions on Audio Speech and Language Processing, 2021

    2021

  11. [11]

    WavLM: Large-scale self-supervised pre-training for full stack speech processing,

    S. Chenet al., “WavLM: Large-scale self-supervised pre-training for full stack speech processing,”IEEE Journal on Selected Topics in Signal Processing, 2022

    2022

  12. [12]

    Unsupervised lexicon discovery from acoustic input,

    C.-y. Lee, T. J. O’Donnell, and J. Glass, “Unsupervised lexicon discovery from acoustic input,”Transactions of the Association for Computational Linguistics, 2015

    2015

  13. [13]

    Unsupervised word discovery from speech using automatic segmentation into syllable-like units,

    O. R ¨as¨anen, G. Doyle, and M. C. Frank, “Unsupervised word discovery from speech using automatic segmentation into syllable-like units,” in Interspeech, 2015

    2015

  14. [14]

    An embedded segmental k-means model for unsupervised segmentation and clustering of speech,

    H. Kamper, K. Livescu, and S. Goldwater, “An embedded segmental k-means model for unsupervised segmentation and clustering of speech,” inASRU, 2017

    2017

  15. [15]

    Self-expressing autoencoders for unsupervised spoken term discovery,

    S. Bhati, J. Villalba, P. ˙Zelasko, and N. Dehak, “Self-expressing autoencoders for unsupervised spoken term discovery,” inInterspeech, 2020

    2020

  16. [16]

    Segmental contrastive predictive coding for unsupervised word segmentation,

    S. Bhatiet al., “Segmental contrastive predictive coding for unsupervised word segmentation,” inInterspeech, 2021

    2021

  17. [17]

    DP-Parse: Finding word boundaries from raw speech with an instance lexicon,

    R. Algayreset al., “DP-Parse: Finding word boundaries from raw speech with an instance lexicon,”Transactions of the Association for Computational Linguistics, 2022

    2022

  18. [18]

    Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and word,

    S. Cuervoet al., “Contrastive prediction strategies for unsupervised segmentation and categorization of phonemes and word,” inICASSP, 2022

    2022

  19. [19]

    Double articulation analyzer with prosody for unsupervised word and phone discovery,

    Y . Okuda, R. Ozaki, S. Komura, and T. Taniguchi, “Double articulation analyzer with prosody for unsupervised word and phone discovery,”IEEE Transactions on Cognitive and Developmental Systems, 2023

    2023

  20. [20]

    Word segmentation on discovered phone units with dynamic programming and self-supervised scoring,

    H. Kamper, “Word segmentation on discovered phone units with dynamic programming and self-supervised scoring,”IEEE/ACM Transactions on Audio Speech and Language Processing, 2023

    2023

  21. [21]

    Spoken- term discovery using discrete speech units,

    B. van Niekerk, J. Za ¨ıdi, M.-A. Carbonneau, and H. Kamper, “Spoken- term discovery using discrete speech units,” inInterspeech, 2024

    2024

  22. [22]

    What do self- supervised speech models know about words?,

    A. Pasad, C.-M. Chien, S. Settle, and K. Livescu, “What do self- supervised speech models know about words?,”Transactions of the Association for Computational Linguistics, 2024

    2024

  23. [23]

    Unsupervised word discovery: Boundary detection with clustering vs. dynamic programming,

    S. Malan, B. van Niekerk, and H. Kamper, “Unsupervised word discovery: Boundary detection with clustering vs. dynamic programming,” in ICASSP, 2025

    2025

  24. [24]

    Zipf’s word frequency law in natural language: a critical review and future directions.,

    S. T. Piantadosi, “Zipf’s word frequency law in natural language: a critical review and future directions.,”Psychonomic Bulletin & Review, 2014

    2014

  25. [25]

    BIRCH: An efficient data clustering method for very large databases,

    T. Zhang, R. Ramakrishnan, and M. Livny, “BIRCH: An efficient data clustering method for very large databases,”ACM SIGMOD Record, 1996

    1996

  26. [26]

    A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models,

    J. A. Bilmes, “A gentle tutorial of the EM algorithm and its application to parameter estimation for Gaussian mixture and hidden Markov models,” tech. rep., International Computer Science Institute, 1998

    1998

  27. [27]

    C. M. Bishop,Pattern Recognition and Machine Learning. Springer, 2006

    2006

  28. [28]

    Sylber: Syllabic embedding representation of speech from raw audio,

    C. J. Choet al., “Sylber: Syllabic embedding representation of speech from raw audio,” inICLR, 2025

    2025

  29. [29]

    SyllableLM: Learning coarse semantic units for speech language models,

    A. Baade, P. Peng, and D. Harwath, “SyllableLM: Learning coarse semantic units for speech language models,” inICLR, 2025

    2025

  30. [30]

    ZeroSyl: Simple zero-resource syllable tokenization for spoken language modeling,

    N. Visser, S. Malan, D. Slabbert, and H. Kamper, “ZeroSyl: Simple zero-resource syllable tokenization for spoken language modeling,” in Interspeech, 2026

    2026

  31. [31]

    Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings,

    K. Levin, K. Henry, A. Jansen, and K. Livescu, “Fixed-dimensional acoustic embeddings of variable-length segments in low-resource settings,” inASRU, 2013

    2013

  32. [32]

    Leveraging multilingual transfer for unsu- pervised semantic acoustic word embeddings,

    C. Jacobs and H. Kamper, “Leveraging multilingual transfer for unsu- pervised semantic acoustic word embeddings,”IEEE Signal Processing Letters, 2024

    2024

  33. [33]

    Low-resource keyword spotting using contrastively trained transformer acoustic word embeddings,

    J. Herreilers, C. Jacobs, and T. Niesler, “Low-resource keyword spotting using contrastively trained transformer acoustic word embeddings,”arXiv preprint arXiv:2506.17690, 2025

    work page arXiv 2025

  34. [34]

    From Louvain to Leiden: Guaranteeing well-connected communities,

    V . A. Traag, L. Waltman, and N. J. van Eck, “From Louvain to Leiden: Guaranteeing well-connected communities,”Scientific Reports, 2019

    2019

  35. [35]

    Narrow scope for resolution- limit-free community detection.,

    V . A. Traag, P. V . Dooren, and Y . Nesterov, “Narrow scope for resolution- limit-free community detection.,”Physical Review E, 2011

    2011

  36. [36]

    Fast algorithm for detecting community structure in networks,

    M. E. J. Newman, “Fast algorithm for detecting community structure in networks,”Physical Review E, 2004

    2004

  37. [37]

    Fast unfolding of communities in large networks,

    V . D. Blondel, J.-L. Guillaume, R. Lambiotte, and E. Lefebvre, “Fast unfolding of communities in large networks,”Journal of Statistical Mechanics: Theory and Experiment, 2008

    2008

  38. [38]

    Hierarchical clustering schemes,

    S. Johnson, “Hierarchical clustering schemes,”Psychometrika 32, 1967

    1967

  39. [39]

    Nielsen,Introduction to HPC with MPI for Data Science, ch

    F. Nielsen,Introduction to HPC with MPI for Data Science, ch. Hierar- chical Clustering, pp. 195–211. Springer, 2016

    2016

  40. [40]

    Ward’s hierarchical agglomerative clustering method: Which algorithms implement Ward’s criterion?,

    F. Murtagh and P. Legendre, “Ward’s hierarchical agglomerative clustering method: Which algorithms implement Ward’s criterion?,”Journal of Classification, 2014

    2014

  41. [41]

    The infinite Gaussian mixture model,

    C. E. Rasmussen, “The infinite Gaussian mixture model,” inNIPS, 1999

    1999

  42. [42]

    Unsupervised lexical clustering of speech segments using fixed-dimensional acoustic embeddings,

    H. Kamper, A. Jansen, and S. Goldwater, “Unsupervised lexical clustering of speech segments using fixed-dimensional acoustic embeddings,” in SLT, 2014

    2014

  43. [43]

    LibriSpeech: An ASR corpus based on public domain audio books,

    V . Panayotov, G. Chen, D. Povey, and S. Khudanpur, “LibriSpeech: An ASR corpus based on public domain audio books,” inICASSP, 2015

    2015

  44. [44]

    FLEURS: Few-shot learning evaluation of universal representations of speech,

    A. Conneauet al., “FLEURS: Few-shot learning evaluation of universal representations of speech,” inSLT, 2023

    2023

  45. [45]

    The Zero Resource Speech Challenge 2017,

    E. Dunbaret al., “The Zero Resource Speech Challenge 2017,” inASRU, 2017

    2017

  46. [46]

    Unsupervised lexicon learning from speech is limited by representations rather than clustering,

    D. Slabbert, S. Malan, and H. Kamper, “Unsupervised lexicon learning from speech is limited by representations rather than clustering,” in ICASSP, 2026

    2026

  47. [47]

    Montreal forced aligner: Trainable text-speech alignment using kaldi,

    M. McAuliffeet al., “Montreal forced aligner: Trainable text-speech alignment using kaldi,” inInterspeech, 2017

    2017

  48. [48]

    The igraph software package for complex network research,

    G. Csardi and T. Nepusz, “The igraph software package for complex network research,”InterJournal Complex Systems, 2006

    2006

  49. [49]

    The Faiss library,

    M. Douzeet al., “The Faiss library,”IEEE Transactions on Big Data, 2026

    2026

  50. [50]

    K-means++: The advantages of careful seeding,

    D. Arthur and S. Vassilvitskii, “K-means++: The advantages of careful seeding,” inSODA, 2007

    2007

  51. [51]

    Scikit-learn: Machine learning in Python,

    F. Pedregosaet al., “Scikit-learn: Machine learning in Python,”Journal of Machine Learning Research, 2011

    2011

  52. [52]

    Syllable discovery and cross-lingual generalization in a visually grounded, self- supervised speech model,

    P. Peng, S.-W. Li, O. R ¨as¨anen, A. Mohamed, and D. Harwath, “Syllable discovery and cross-lingual generalization in a visually grounded, self- supervised speech model,” inInterspeech, 2023

    2023

  53. [53]

    SD-HuBERT: Sentence-level self-distillation induces syllabic organization in HuBERT,

    C. J. Choet al., “SD-HuBERT: Sentence-level self-distillation induces syllabic organization in HuBERT,” inICASSP, 2024

    2024

  54. [54]

    Revisiting Lexicon Evaluation in Unsupervised Word Discovery

    S. Malan, D. Slabbert, and H. Kamper, “Revisiting lexicon evaluation in unsupervised word discovery,”arXiv preprint arXiv:2606.06183, 2026

    work page internal anchor Pith review Pith/arXiv arXiv 2026

  55. [55]

    The Zero Resource Speech Challenge 2020: Discover- ing discrete subword and word units,

    E. Dunbaret al., “The Zero Resource Speech Challenge 2020: Discover- ing discrete subword and word units,” inInterspeech, 2020

    2020