emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity

Anna Wegmann; Cantao Su; Dong Nguyen; Esther Ploeger; Menan Velayuthan

REVIEW 4 major objections 5 minor 18 references

emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity

T0 review · 4 major / 5 minor · reviewed 2026-08-01 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read A single open-source tool now measures embedding-based data diversity 22 ways.

desk verdict A genuinely useful, well-tested tool for embedding-based diversity measurement; the software is solid, but the demonstrations of meaningfulness need error bars and robustness checks before the claims fully land. read the letter →

arxiv 2607.19848 v1 pith:N7SFIPCT submitted 2026-07-22 cs.CL

Cantao Su , Menan Velayuthan , Esther Ploeger , Dong Nguyen , Anna Wegmann This is my paper

classification cs.CL

keywords datadiversityembedding-basedmeasuresNLPdatasetssemanticstylisticlanguagespeaker

verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

The paper argues that measuring diversity in NLP data is too important to leave to ad hoc scripts, and that the missing piece is a standardized, maintained tool for embedding-based diversity measurement. It introduces emb-diversity, a Python package and command-line tool that implements 22 diversity measures drawn from the literature, ranging from simple pairwise-distance summaries to graph-based, kernel-based, UMAP-projection, and vector-statistic measures. The authors show the tool surfaces expected differences in controlled settings: measured diversity rises as text domains, language families, or speakers are added, and human essays register clearly higher stylistic diversity than LLM essays on the same prompts. They also benchmark runtime across dataset sizes and report a user study that led to documentation improvements. If adopted, the tool would give practitioners a common, comparable yardstick for a currently fragmented measurement landscape.

What carries the argument

The load-bearing object is the measure_diversity() pipeline: one entry point that resolves a named diversity axis (currently semantics and style) to a default embedding model, or accepts any explicit embedding model, then computes pairwise distances once and dispatches to individual measures, each registered as a single function mapping embeddings to a scalar. Caching of embeddings and pairwise distances avoids redundant computation across the 22 measures, while a chunking-and-pooling strategy handles texts longer than an encoder's limit. The axis abstraction is what lets the same measure definitions transfer across modalities (text, audio, typology vectors) without code changes.

What would settle it

Take datasets with known internal structure, embed them with two models that encode different axes (e.g., a topic-focused model and a style-focused model), and resample to get error bars on the 22 scores; if score orderings between datasets flip across embeddings, or differences within one dataset vanish under resampling, the tool's demonstrations do not support a general measurement standard.

Watch

Extended reading notes

Core claim

The paper's central claim is that embedding-based diversity measurement can and should be unified in a single extensible tool, and that no such comprehensive, community-maintained tool existed before. emb-diversity operationalizes this by accepting any list of texts or vectors, embedding them (with caching and a chunking option for long documents), and computing 22 published measures that each define diversity differently over the distance matrix, kernel matrix, a UMAP projection, or raw vector statistics. Three defaults—mean pairwise distance, Vendi score, and graph entropy—are highlighted as widely used or benchmark-supported. The paper's demonstrations show monotonic increases in those de

Load-bearing premise

All demonstrations depend on the assumption that distances in the chosen embedding space faithfully capture the diversity concept being measured, and that reported score gaps are real rather than sampling noise.

Editorial extensions

If this is right

Datasets can be compared quantitatively along a chosen axis, replacing assumptions based on how data was collected.
Diversity can be measured consistently across modalities—text, audio, and any vector-representable data—with the same 22 measures.
Runtime benchmarks included in the tool let practitioners skip measures that are prohibitively expensive at their dataset size.
A registered-function interface means new diversity measures can be added and compared against existing ones without touching the rest of the pipeline.
Controlled mixtures show that the default measures track intuitive increases in diversity, supporting use as a sanity-checked measurement standard.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

Editorial inference: the paper's evaluations use single point estimates; turning the tool into a decision instrument would require bootstrap confidence intervals or significance tests around reported scores.
Editorial inference: the strongest untested payoff is in data curation—using these scores as an objective to select training subsets—something the paper motivates but does not evaluate.
Editorial inference: the strong correlations among many measures reported in the appendix suggest a smaller subset may suffice for many use cases, while uncorrelated measures may capture genuinely different properties worth separate reporting.
Editorial inference: because a diversity axis is just a choice of embedding model, new axes (topic, sentiment, demographic style) become testable as encoders improve, extending the tool beyond the four demonstrated use cases.

Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Desk Editor's Note

read the letter

emb-diversity is a real and useful piece of engineering. The package unifies 22 embedding-based diversity measures behind one API and CLI, with caching, chunking for long texts, and 511 tests on every merge. That is exactly the kind of standardized, maintainable artifact the field has been missing, and the implementations appear honest: the Vendi score calls the official package, five measures are reimplemented and cross-checked against released code, and the rest are from formal definitions. The correlation appendix and runtime comparison are a nice addition. If you measure diversity in NLP datasets, this tool will save you time.

The soft spots are in the validation, not the code. Tables 2–4 and Figure 6 are single runs with no error bars, bootstrap intervals, or checks across alternative embedding models. The 'clearly more diverse' stylistic claim in §5.1 rests on one style embedding model (ANNAWEGMANN/STYLE-EMBEDDING, an author's own model) with no robustness check against other style encoders. The default Graph Entropy is justified by an unpublished, self-cited benchmark (Su et al., 2026, under review), which leaves the default anchoring on work the referee cannot inspect. The language and speaker case studies show expected orderings, but again with no variance; the user study is small and informal, and one participant's comment about score interpretation was addressed only in documentation. None of this undercuts the claim that the tool exists and works; it undercuts the stronger claim that the tool surfaces meaningful diversity differences. The paper is honest about some of this, but the demonstrations are thinner than the 'demonstrate its potential' framing suggests.

The right fix is straightforward: add bootstrap intervals or multiple seeds, test the style conclusion with at least one alternative style embedding model, and either cite the benchmark with more detail or hedge the default choice. Also worth a direct comparison to the existing 'diversity' package, which already includes two embedding-based measures.

Bottom line: this is a solid system paper that deserves serious referee time, and the tool itself deserves adoption. I'd accept it with the expectation of a revision that strengthens the empirical section.

Referee Report

4 major / 5 minor

Summary. The paper introduces emb-diversity, an open-source Python and command-line tool for measuring the diversity of data represented as embeddings. The tool implements 22 diversity measures from the literature, grouped into distance-matrix, distance-graph, kernel-matrix, UMAP-projection, and vector-statistics families, and exposes a unified API/CLI with caching and long-text chunking. The paper evaluates the tool with a controlled experiment on mixtures of Common Pile domains (§4.1), a runtime comparison (§4.2), a small heuristic usability study (§4.3), and three case studies: stylistic vs. semantic diversity of human/LLM essays (§5.1), typological language diversity (§5.2), and speaker diversity in audio (§5.3). The central claim is that emb-diversity fills the gap of a comprehensive, community-maintained, embedding-based diversity measurement tool.

Significance. If the software artifact is as comprehensive and well-tested as described, it addresses a real need: many researchers currently rely on fragmented, ad hoc scripts for embedding-based diversity measurement. The tool's strengths are concrete: 22 measures, a clean architecture, caching, chunking, 511 tests, and released example scripts. The correlation analysis in Appendix F is also useful for understanding relationships among measures. However, the empirical demonstrations of the tool's usefulness are currently not convincing. The controlled experiments and case studies report single runs without error bars, confidence intervals, or significance tests; the style-diversity conclusion depends on one embedding model; and the default measure Graph Entropy is justified by an unpublished benchmark from overlapping authors. These issues do not undermine the existence or engineering quality of the tool, but they weaken the paper's claim that the measures surface meaningful diversity differences. With stronger validation, the tool could become a valuable community resource.

major comments (4)

[§4.1, Table 2] The claim that the default measures 'behave as expected' is not supported by the presented evidence. The table reports a single run per mixture with no error bars, confidence intervals, or significance tests. Moreover, Graph Entropy is described as 'almost monotonically' increasing, but the reported values (6898.4, 6897.5, 6900.9, 6902.2) are not monotonic: the value decreases from N to N∪A. Please provide multiple random samples (e.g., 10 or more) and report means with confidence intervals, or at least explicitly acknowledge the noise. This is load-bearing because the controlled-mixture experiment is the primary evidence that the measures surface diversity differences.
[§5.1, Figure 6] The conclusion that 'style is clearly more diverse in the human written essays' rests on a single style-embedding model (ANNAWEGMANN/STYLE-EMBEDDING), a single run, and no uncertainty quantification. Since this model is the tool's default style encoder and was presumably selected partly by the authors, the comparison is at risk of being self-fulfilling. Please add bootstrap confidence intervals or repeat with at least one or two alternative style encoders (e.g., a second content-independent style model) to show that the ordering is robust to the choice of embedding model.
[§5.2 and §5.3, Tables 3 and 4] The language and speaker demonstrations are single instances with no measure of uncertainty. For Table 4, the differences between k=4 and k=5 are small (Vendi: 24.8 vs. 26.5; MPD: 0.805 vs. 0.817) and could be within run-to-run noise; the claim that diversity 'increases with the number of speakers' needs at least repeated subsampling or a significance test. Similarly, Table 3's language selections are deliberately extreme in typological distance, so the result is not a discriminating test. Either strengthen these analyses with variance estimates and statistical tests, or explicitly label them as illustrative single runs.
[§3.2, default measure choice] Graph Entropy is marked as a default because it 'performed well on the benchmark by Su et al. (2026)'. That benchmark is under review and authored by overlapping authors. As written, the default choice is not independently validated, and the reader cannot assess whether Graph Entropy is the best default. Please either describe the benchmark result in sufficient detail (or include it in the paper), or choose a default that is well-established in the literature (e.g., Vendi Score or Mean Pairwise Distance) and place Graph Entropy on equal footing with the other measures.

minor comments (5)

[Figure 3] The code listing has spaces inserted within function names and import statements (e.g., 'me as ur e_ di ver si ty'). This appears to be a rendering artifact, but it makes the code example hard to read; ensure the code block is rendered correctly.
[References] The reference for Cox et al. (2021) misspells 'Von Der Weth' as 'V on Der Weth'. The reference for Estève and Dobrovoljc (2026) lists 'Marocco'; should be 'Morocco'.
[§5.1 and Appendix C] The case study says 'we sample ≈800 essays', but Appendix C states 'exactly 826 essays each'. Please make the numbers consistent.
[§4.2, Figure 5] The runtime figure is said to be 'averaged over 5 runs', but no variance is shown. Adding error bars or a brief note that variance was small would improve interpretability.
[Title page] The correspondence line lacks a space: 'Correspondence:c.su@uu.nl' should be 'Correspondence: c.su@uu.nl'.

Circularity Check

1 steps flagged · score 2.0 of 10

No derivation is circular; the only self-referential element is the unpublished, overlapping-author benchmark used to select a default measure.

other [§3.2 Diversity Measures; §2 Concurrent work]
"Three measures—Mean Pairwise Distance, Vendi Score, and Graph Entropy—are marked as defaults: the former two are widely used in the literature, whilst Graph Entropy performed well on the benchmark by Su et al. (2026). ... The initial code base for this tool was written as part of Su et al. (2026), who collected embedding-based diversity measures, and tested them on a newly introduced benchmark."

The only stated justification for making Graph Entropy a default is an under-review benchmark by overlapping authors that is based on the same initial code base. This is a mild self-citation: the default's quality is not independently established. It is not a circular derivation, however, because no fitted parameter is predicted from this choice, and the paper's central claim — a maintained tool implementing 22 literature measures — stands independently of which measure is marked as default.

full rationale

The paper is a systems/tool description with no fitted parameters, no first-principles derivation, and no prediction that reduces to its inputs by construction. The case studies (§5.1–5.3) demonstrate the pipeline on controlled or intuitively ordered data, but they are demonstrations rather than derived results: the claim that human essays are stylistically more diverse depends on the choice of ANNAWEGMANN/STYLE-EMBEDDING and is externally testable, not definitional. The §4.1 monotonicity check orders domains by 'internal diversity' using Appendix E scores computed with the same default measures, so it is partly a consistency check rather than an external ground-truth validation; still, the number-of-domains manipulation is externally controlled and the scores do not follow by construction. The paper itself quotes a usability participant saying the documentation gives little guidance on whether a score difference is meaningful — an acknowledged validity limitation, not circularity. The only genuine circularity-adjacent issue is the self-cited, under-review benchmark used to select Graph Entropy as a default; because this choice does not drive the paper's main claims, the appropriate score is low.

Assumptions & free parameters 2 free parameters · 4 assumptions · 0 invented entities

The tool itself introduces no fitted numeric parameters; the entries above are design/evaluation choices. The main epistemic burden is the embedding-proxy assumption and reliance on self-authored defaults and an unpublished benchmark.

free parameters (2)

Default embedding models for diversity axes = semantic: all-MPNET-base-v2; style: ANNAWEGMANN/STYLE-EMBEDDING
Chosen by the authors; all case-study numbers in §5.1 and the axis behavior in Table 2 depend on them. The style model is from the authors' own prior work (Wegmann et al., 2022).
Line length filter for Common Pile samples = 201–300 characters
Explicitly chosen in §4.1 ('To avoid long texts distorting the measures, we only sample lines containing between 201 and 300 characters'); affects the diversity scores in Table 2.

assumptions (4)

domain assumption Embedding distance in a chosen model is a valid proxy for the diversity concept being measured
Central to every measure in §3.2; if Euclidean distance in embedding space does not track semantic/style/speaker difference, the 22 scores are not meaningful.
domain assumption Ground-truth orderings used for validation are correct and detectable
§4.1 Table 2 assumes adding more Common Pile domains increases diversity; §5.2 assumes the Germanic set is less diverse than the set with Basque/Hungarian/Yoruba; §5.3 assumes more speakers = more diversity. These are asserted, not measured.
domain assumption Graph Entropy is a justified default because it performed well on the Su et al. (2026) benchmark
§3.2 marks Graph Entropy as a default citing an under-review paper by overlapping authors; the benchmark is not publicly checkable.
domain assumption STYLE-EMBEDDING represents content-independent style
Required for §5.1's conclusion that human essays are more stylistically diverse; otherwise the difference could be topic leakage.

how reviews work

0 comments

Cite this review

Pith. "Pith review of emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity." pith.science (2026). https://pith.science/paper/N7SFIPCT

@misc{pith2026260719848,
  author       = {Pith},
  title        = {Pith review of: emb-diversity: A Tool for Embedding-Based Measurement of Data Diversity},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/N7SFIPCT}},
  note         = {Machine review of arXiv:2607.19848}
}

read the original abstract

There is growing evidence that data diversity is crucial for developing fair and robust NLP models. However, current approaches to measure diversity remain inconsistent and fragmented: While there exist a number of tools for measuring the lexical diversity of texts, researchers lack standardized tools for quantifying diversity based on embeddings. Embedding-based diversity measures are highly flexible: They work with any embedding model and any data that can be embedded, and are thus applicable to many notions of diversity. With emb-diversity, we provide a comprehensive embedding-based diversity measurement tool, spanning a broad range of measures. We demonstrate its potential for several use cases: measuring the stylistic, semantic, language and speaker diversity of datasets. https://github.com/nlpsoc/emb-diversity/

Figures

Figures reproduced from arXiv: 2607.19848 by the authors.

**Figure 1.** Diagram illustrating the overall emb-diversity pipeline for an example use case of comparing datasets of texts written by humans and by LLMs. Texts are embedded, and a range of diversity scores is calculated for each dataset. The embedding model that is chosen determines what aspect is considered when calculating diversity. 2 Related Work Within diversity measurement, lexical diversity has received the greatest atte… view at source ↗

**Figure 2.** Architecture of emb-diversity. Input flows top to bottom; every measure funnels through a single embedding-and-validation step, which checks the caches before falling through to the embedding backends. The convenience function can also be skipped in case users want to manipulate parameters of diversity functions directly. directly on a list of texts or vectors and returns a structured output, making it easy to int… view at source ↗

**Figure 4.** Example usage of the CLI. our documentation. New measures can be added by registering a single function that maps a set of embeddings to a scalar, so the tool can grow as new measures are proposed. Three measures— Mean Pairwise Distance, Vendi Score, and Graph Entropy—are marked as defaults: the former two are widely used in the literature, whilst Graph Entropy performed well on the benchmark by Su et al. (2026). 3… view at source ↗

Figures from the paper (4 more)

**Figure 5.** Figure 5: Runtime (seconds) per measure on the same [PITH_FULL_IMAGE:figures/full_fig_p005_5.png]

**Figure 6.** Figure 6: Left: Contours of the density plot of generated and human-written essays on the same topics with a semantic and a style embedding model. Right: Embedding-based diversity measures over the same dataset. Note that the embedding models for semantics and style are differen…

**Figure 7.** Figure 7: Chunking is available both from the main [PITH_FULL_IMAGE:figures/full_fig_p010_7.png]

**Figure 8.** Figure 8: Pairwise correlations across measures. The correlation scores were calculated on samples from 17 [PITH_FULL_IMAGE:figures/full_fig_p011_8.png]

Discussion (0). Sign in to comment.

Reference graph

Works this paper leans on

18 extracted references · 2 linked inside Pith

[1]

The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time

Visibility of System Status ("The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time.")
[2]

Calculate the diversity over a dataset which consists of three vectors
[3]

Calculate the MST dispersion over a small text dataset
[4]

Calculate the stylistic diversity over a text dataset
[5]

In11th Inter- national Conference on Learning Representations, ICLR 2023

How much space has been explored? mea- suring the chemical space covered by databases and machine-generated molecules. In11th Inter- national Conference on Learning Representations, ICLR 2023. Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. Measuring data diversity for instruction t...

2023
[6]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242

Improving diversity of commonsense genera- tion by large language models via in-context learning. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242. Tianhui Zhang, Bei Peng, and Danushka Bollegala

2024
[11]

Calculate the diversity over a dataset from HuggingFace
[12]

We added short explanations of each heuristic, which were adapted from https://www.nngroup.com/articles/ how-to-conduct-a-heuristic-evaluation/

Compare the diversity of two datasets We asked participants to give feedback according to the following usability heuristics, defined by Nielsen and Molich (1990). We added short explanations of each heuristic, which were adapted from https://www.nngroup.com/articles/ how-to-conduct-a-heuristic-evaluation/ . We selected only heuristics applicable to our t...

1990

Show all 18 references

[14]

The correlation scores were calculated on samples from 17 Common Pile domains

Consistency and Standards ("Users should not have to wonder whether different words, situa- Figure 8: Pairwise correlations across measures. The correlation scores were calculated on samples from 17 Common Pile domains. Measures are grouped by how they were calculated. Find th...
[15]

Either eliminate error-prone conditions, or check for them and present users with a con- firmation option before they commit to the action.")

Error Prevention ("Good error messages are important, but the best designs carefully pre- vent problems from occurring in the first place. Either eliminate error-prone conditions, or check for them and present users with a con- firmation option before they commit to the action.")
[16]

Allow users to tailor frequent actions.")

Flexibility and Efficiency of Use ("Shortcuts — hidden from novice users — may speed up the interaction for the expert user such that the design can cater to both inexperienced and experienced users. Allow users to tailor frequent actions.")
[17]

Error messages should be expressed in plain language (no error codes), precisely indicate the problem, and construc- tively suggest a solution

Help Users Recognize, Diagnose, and Re- cover from Errors ("Error messages should be expressed in plain language (no error codes), precisely indicate the problem, and construc- tively suggest a solution.")
[18]

It’s best if the sys- tem doesn’t need any additional explanation. However, it may be necessary to provide doc- umentation to help users understand how to complete their tasks

Help and Documentation ("It’s best if the sys- tem doesn’t need any additional explanation. However, it may be necessary to provide doc- umentation to help users understand how to complete their tasks.") E Diversity Scores per Common Pile Domain Table 5 displays the diversity ...

2026
[2021]

Yi-An Lai, Xuan Zhu, Yi Zhang, and Mona Diab

Assessing the validity of lexical diversity in- dices using direct judgements.Language Assessment Quarterly, 18(2):154–170. Yi-An Lai, Xuan Zhu, Yi Zhang, and Mona Diab. 2020. Diversity, density, and homogeneity: Quantitative characteristic metrics for text collections. InPro-...

2020 arXiv
[2022]

InPro- ceedings of the 7th Workshop on Representation Learning for NLP, pages 249–268, Dublin, Ireland

Same author or just same topic? towards content-independent style representations. InPro- ceedings of the 7th Workshop on Representation Learning for NLP, pages 249–268, Dublin, Ireland. Association for Computational Linguistics. Xiangyu Xi, Deyang Kong, Jian Yang, Jiawei Yang...

2025
[2023]

In Advances in Neural Information Processing Systems, volume 36, pages 9931–9943

An information-theoretic evaluation of genera- tive models in learning multi-modal distributions. In Advances in Neural Information Processing Systems, volume 36, pages 9931–9943. Curran Associates, Inc. Nikhil Kandpal, Brian Lester, Colin Raffel, Sebastian Majstorovic, Stella...

2026
[2024]

InProceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 3: System Demonstra- tions), pages 346–354, Bangkok, Thailand

Variationist: Exploring multifaceted variation and bias in written language data. InProceedings of the 62nd Annual Meeting of the Association for Com- putational Linguistics (Volume 3: System Demonstra- tions), pages 346–354, Bangkok, Thailand. Associa- tion for Computational ...

2019 arXiv
[2025]

Task” and “Human

Evaluating the evaluation of diversity in com- monsense generation. InProceedings of the 63rd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 24258– 24275. Dora Zhao, Jerone T. A. Andrews, Orestis Papakyri- akopoulos, and Alice Xi...

2024

Pith tools

Reviewed August 1, 2026 · model on record in the stance chip above.

[1] [1]

The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time

Visibility of System Status ("The design should always keep users informed about what is going on, through appropriate feedback within a reasonable amount of time.")

[2] [2]

Calculate the diversity over a dataset which consists of three vectors

[3] [3]

Calculate the MST dispersion over a small text dataset

[4] [4]

Calculate the stylistic diversity over a text dataset

[5] [5]

In11th Inter- national Conference on Learning Representations, ICLR 2023

How much space has been explored? mea- suring the chemical space covered by databases and machine-generated molecules. In11th Inter- national Conference on Learning Representations, ICLR 2023. Yuming Yang, Yang Nan, Junjie Ye, Shihan Dou, Xiao Wang, Shuo Li, Huijie Lv, Tao Gui, Qi Zhang, and Xuanjing Huang. 2025. Measuring data diversity for instruction t...

2023

[6] [6]

InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242

Improving diversity of commonsense genera- tion by large language models via in-context learning. InFindings of the Association for Computational Linguistics: EMNLP 2024, pages 9226–9242. Tianhui Zhang, Bei Peng, and Danushka Bollegala

2024

[7] [11]

Calculate the diversity over a dataset from HuggingFace

[8] [12]

We added short explanations of each heuristic, which were adapted from https://www.nngroup.com/articles/ how-to-conduct-a-heuristic-evaluation/

Compare the diversity of two datasets We asked participants to give feedback according to the following usability heuristics, defined by Nielsen and Molich (1990). We added short explanations of each heuristic, which were adapted from https://www.nngroup.com/articles/ how-to-conduct-a-heuristic-evaluation/ . We selected only heuristics applicable to our t...

1990