Pith. sign in

REVIEW 2 major objections 5 minor 52 references

A Wasserstein split of text corpora into fidelity and diversity scores predicts when synthetic training data will hurt model accuracy.

Reviewed by Pith at T0; open to challenge. T0 means a machine referee read the full paper against a public rubric. the ladder, T0–T4 →

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

T0 review · grok-4.5

2026-07-11 17:12 UTC pith:Y7TSQAO3

load-bearing objection Clean OT split of fidelity vs diversity for text datasets, with solid M2D2 controls and a useful but confounded GSM8K diversity–accuracy link. the 2 major comments →

arxiv 2607.04563 v1 pith:Y7TSQAO3 submitted 2026-07-06 cs.CL

Fidelity-Diversity Metrics for Text

Amanda Wang , Tudor Manole , Florentina Bunea , John Thickstun This is my paper
classification cs.CL
keywords fidelitydiversityoptimal transportWasserstein distancesynthetic datatext evaluationGSM8Kmixing measures
verification ladder T0 review T1 audit T2 compute T3 formal T4 reserved

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

When practitioners add synthetic or auxiliary text to a training set, a single quality score cannot tell them whether the new text is merely off-topic or whether it fails to cover the modes of the original data. This paper supplies two complementary scores that answer those questions separately. Fidelity measures how close each mode of the candidate set sits to some mode of a reference set; diversity measures how evenly the candidate mass covers the reference modes. Both scores are obtained from a single optimal-transport calculation between discrete summaries of the two corpora (mixture atoms or empirical embeddings). Controlled topic experiments on M2D2 show that the scores move independently in the expected directions. On synthetic GSM8K-style math problems whose diversity is deliberately throttled by the number of seed problems, the diversity score tracks downstream exact-match accuracy of finetuned language models almost linearly.

Core claim

Fidelity and diversity, defined by the normalized nearest-neighbor transport cost and the residual optimal-transport cost between discrete summaries of two text corpora, successfully disentangle geometric support mismatch from mass-coverage failure, and the diversity score on synthetic math data correlates strongly with the accuracy of models trained on that data.

What carries the argument

The multiplicative Wasserstein decomposition exp(−W1(G,H)/κ(H)) = F(G,H)·D(G,H), where F is the exponentiated cost of greedy nearest-neighbor reassignment of evaluation atoms onto reference atoms and D is the exponentiated residual cost of full optimal transport.

Load-bearing premise

That distances in a fixed pretrained embedding space, together with the chosen discrete summary of each corpus, actually track the semantic notions of fidelity and diversity that practitioners care about.

What would settle it

Construct synthetic GSM8K-style training sets whose seed-controlled diversity ranking is known, recompute the diversity scores under a different embedding or ground metric, and check whether the reported correlation with finetuned test accuracy (r ≈ +0.97) disappears.

Watch this falsifier — get emailed when new claim-graph text bears on it.

If this is right

  • Dataset-curation pipelines can rank candidate augmentation sets by diversity before any training run, discarding low-diversity synthetic text that would degrade accuracy.
  • A single Wasserstein computation yields both diagnostics, so practitioners obtain a support-mismatch flag and a coverage flag without separate models.
  • Controlled seed-based generation of synthetic math data becomes a calibrated testbed for diversity metrics because the expected diversity ranking is known a priori.
  • When embeddings lack clear clusters, the same formulas apply directly to empirical measures, so the method remains usable for unstructured text.

Where Pith is reading between the lines

These are editorial extensions of the paper, not claims the author makes directly.

  • The same decomposition could serve as a cheap pre-filter for preference or instruction-tuning corpora whose mode coverage is hard to inspect by hand.
  • If embedding geometry is the main load-bearing premise, multi-embedding ensembles or domain-adapted encoders would be the natural robustness check.
  • The residual OT term may also diagnose mode collapse inside generative models themselves, not only inside static training sets.

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, simulated authors' rebuttal, and a circularity audit.

Referee Report

2 major / 5 minor

Summary. The paper proposes fidelity and diversity metrics for comparing an evaluation text corpus to a reference corpus via optimal transport on discrete summaries of pretrained embeddings. Fidelity is the (normalized, exponentiated) nearest-neighbor reassignment cost of evaluation atoms onto reference atoms; diversity is the residual gap between that cost and the full Wasserstein cost, yielding the decomposition exp(-W1(G,H)/κ(H)) = F(G,H)·D(G,H). On M2D2, nested topic subsets and controlled distribution shifts show diversity rising with topic coverage while fidelity stays largely flat (or falls under shift). On synthetic GSM8K-style data generated by rewriting a seed fraction of the train split, diversity (computed on unquantized empirical measures with debiased Sinkhorn) correlates strongly with exact-match accuracy of Llama-3.2 models finetuned on those sets (reported r=+0.973).

Significance. If the metrics reliably disentangle support mismatch from mass-coverage failure and if D predicts useful training-data quality, the work supplies a practical, reference-based diagnostic for synthetic and curated LM datasets that goes beyond scalar divergences such as MAUVE or FID. Strengths include a clean OT decomposition with a short closed-form proof for the fidelity identity (App. D), open-sourced code and synthetic datasets, controlled M2D2 topic experiments with OLS and bootstrap CIs, and an external downstream check (finetune accuracy) that is not plugged into the definition of F or D. The contribution is incremental relative to precision-recall work for generative models but is well motivated for dataset curation rather than pure generation evaluation.

major comments (2)
  1. [§4.2 / App. C.1] §4.2.1 / App. C.1: diversity is controlled by taking the first k% of the ordered GSM8K train split as seeds and generating (n-k)/k rewrites per seed. Contiguous prefixes systematically omit later problem types, so both D and finetune accuracy can rise simply because more of the original support is present in the seed, not because residual OT cost measures structural mode coverage in embedding space. The paper already notes that embeddings lack cluster structure and therefore uses unquantized empirical measures with Sinkhorn (ε=0.001), which heightens sensitivity to which concrete points appear. The central applied claim (r=+0.973, Fig. 4) is therefore under-supported until the ladder is re-run with random or stratified seeds of equal size (or an ablation that holds seed support fixed while varying rewrite multiplicity).
  2. [§3.1–3.2, §4.2, App. C.2] §3.1–3.2 and §4.2: the GSM8K pipeline relies on E5-small mean-pooled embeddings plus Euclidean cost on empirical measures, while M2D2 uses BIC-selected spherical GMMs with a location-scale ground metric. The paper reports that conventional clustering fails on the synthetic embeddings (App. C.2), so the discrete summary is essentially the raw point cloud. Without sensitivity checks across embeddings (beyond the brief GPT-2 comparison in App. A), pooling choices, or Sinkhorn ε, it remains unclear whether D tracks human-interpretable structural diversity rather than embedding geometry artifacts. A short multi-embedding or multi-ε ablation on the same GSM8K ladder would substantially strengthen the claim.
minor comments (5)
  1. [App. A] App. A comparison to Le Bronnec et al. precision/recall is useful but limited to M2D2 Wikipedia; a parallel comparison on the GSM8K ladder would clarify when residual OT and k-NN support estimation diverge.
  2. [Fig. 4a / §4.2.1] Fig. 4a reports debiased Sinkhorn with ε=0.001; state the solver, number of iterations, and whether the same ε is used for all seed fractions so that residual costs remain comparable.
  3. [Table 1 / Fig. 3] Table 1 and Fig. 3: a few fidelity OLS coefficients are statistically significant (e.g., S2ORC coll. 2, Wikipedia coll. 3). Briefly discuss whether this is residual topic imbalance or noise, so the 'fidelity flat' narrative is not overstated.
  4. [§3.3] Notation: mixing measures G,H and embedded sets U,V are clear, but κ(H) is introduced mid-definition; a short sentence that it is the expected pairwise distance under H would help readers.
  5. [Throughout] Typos / polish: 'fidelity-diveristy' (p. 3), 'thereferencedistribution' (p. 5), and occasional missing spaces after periods in the arXiv text.

Circularity Check

0 steps flagged

No circularity: F and D are defined from OT on embeddings before any finetune; the GSM8K accuracy correlation is an external empirical check, not forced by construction.

full rationale

The paper defines fidelity and diversity via an explicit Wasserstein decomposition (F = exp(−W1(G, HG)/κ(H)), D = exp(−(W1(G, H) − W1(G, HG))/κ(H))) on discrete summaries of pretrained embeddings (§3.3, Eq. 3 and surrounding text; closed form proved in App. D). These quantities are computed solely from the evaluation and reference corpora (GMM mixing measures on M2D2; empirical measures + debiased Sinkhorn on GSM8K). Downstream exact-match accuracy is obtained by a completely separate SFT + test-set evaluation pipeline (§4.2.1–4.2.2, Fig. 4b) and is never an input to F or D. The reported Pearson r = +0.973 is therefore an observed correlation, not a quantity that is definitionally equal to D or to any fitted parameter of D. No uniqueness theorem, ansatz, or load-bearing self-citation is invoked to force the definitions or the correlation; citations to prior precision-recall and OT work are comparative, not foundational. Design choices (BIC k, Sinkhorn ε, E5-small) are ordinary methodological degrees of freedom and do not create a self-definitional loop. The derivation chain is therefore self-contained against external benchmarks.

Axiom & Free-Parameter Ledger

5 free parameters · 4 axioms · 2 invented entities

The central empirical claims rest on standard OT and mixture/empirical discrete summaries, plus domain assumptions that embedding space and chosen quantizations capture semantic modes. Free choices (BIC component counts, Sinkhorn ε, embedding model, GMM ground metric, seed-prefix construction) affect scores but are not fitted to the downstream accuracy target. The metrics themselves are definitional constructs, not new physical entities.

free parameters (5)
  • GMM component count k (BIC-selected per dataset)
    Chosen from a discrete grid via BIC for each M2D2 reference/subset; changes the discrete support of G and H and thus F and D.
  • Sinkhorn entropic regularization ε
    Set to 0.001 for debiased Sinkhorn approximation of diversity on empirical GSM8K embeddings; affects approximate OT costs.
  • Embedding model and pooling (E5-small, mean-pool)
    Defines the metric space for all distances; not learned end-to-end for the metrics but is a fixed modeling choice the scores depend on.
  • GMM atom ground metric (location-scale form with z|σ1²−σ2²|)
    Hand-chosen theoretical metric for spherical GMM parameters; scales fidelity/diversity numerics on M2D2.
  • Synthetic seed fractions and rewrite multiplicities
    Design knobs {0.1%…10%}×n and rewrites-per-seed that define the diversity ladder; not fitted to accuracy but construct the independent variable.
axioms (4)
  • standard math 1-Wasserstein distance and couplings on finite mixing measures are well-defined for the chosen ground metric on embedding/parameter space.
    Definition 1 and App. D; standard OT.
  • domain assumption Pretrained sentence embeddings place semantically similar documents nearer than dissimilar ones in a way that matches the intended notions of fidelity and diversity.
    §3.1 embedding step; all experiments use E5-small (and App. A GPT-2 comparison).
  • domain assumption GMM mixing measures (M2D2) or empirical measures (GSM8K) are adequate discrete summaries of the data distributions for mode coverage.
    §3.2; App. C.2 notes clustering fails on synthetic math embeddings, motivating empirical measures.
  • ad hoc to paper Varying the number of GSM8K seed problems primarily manipulates structural diversity rather than unrelated quality factors once a fully synthetic reference is used.
    §4.2.1 experimental design; load-bearing for interpreting diversity–accuracy correlation as diversity-driven.
invented entities (2)
  • Fidelity functional F(G,H) via nearest-neighbor reassignment HG no independent evidence
    purpose: Scalar measure of support alignment of evaluation modes to reference modes.
    Defined in §3.3; definitional metric, not an external physical object.
  • Diversity functional D(G,H) as residual OT cost after fidelity no independent evidence
    purpose: Scalar measure of mass-coverage / mode-spreading relative to the reference.
    Defined in §3.3; complementary to F by construction of the decomposition.

pith-pipeline@v1.1.0-grok45 · 26961 in / 3376 out tokens · 32917 ms · 2026-07-11T17:12:55.959919+00:00 · methodology

0 comments
read the original abstract

As language modeling technology matures, there is an increasing research focus on the composition and curation of datasets used to train these models. For instance, practitioners commonly seek to augment high-quality datasets with additional text to enhance the performance of models trained on that data. However, informed decisions about data augmentation require more nuanced assessments about data quality. We build on work measuring the precision and recall of generative models to develop a pair of metrics that quantify (1) fidelity, capturing how closely candidate text resembles reference data, and (2) diversity, capturing how well it covers the modes of the reference dataset. Our metrics are based on optimal transport divergence functionals between discrete text summaries. In experiments on M2D2 text datasets, we show that these metrics are able to disentangle a lack of fidelity from a lack of diversity in deficient candidate text. In further experiments, our metrics detect diversity deficits in synthetic GSM8K-style math datasets, which correlate with degradations in downstream accuracy of language models finetuned on this synthetic data.

Figures

Figures reproduced from arXiv: 2607.04563 by Amanda Wang, Florentina Bunea, John Thickstun, Tudor Manole.

Figure 1
Figure 1. Figure 1: Illustration of four evaluation mixing measures [PITH_FULL_IMAGE:figures/full_fig_p006_1.png] view at source ↗
Figure 2
Figure 2. Figure 2: Fidelity & diversity scores for controlled distribution-shift experiments. Fidelity scores for both collections exhibit significant negative association with the number of categories disjoint from the reference; computing the OLS regression coefficient β and t-test p-value for the null hypothesis β = 0 gives β = −0.0096 (p < 0.001) for M2D2 S2ORC and β = −0.0093 (p < 0.001) for M2D2 Wikipedia (a) M2D2 S2OR… view at source ↗
Figure 3
Figure 3. Figure 3: Fidelity and diversity scores for collections 0, 1, 2, 3, and 4 of text from M2D2 S2ORC and M2D2 Wikipedia. The x-axes indicate the subset of the collection being measured against the reference. Dashed lines indicate baseline values of fidelity and diversity for each collection. See App. B.1 for a detailed list of the ten categories in each collection, as well as bootstrapped confidence bands for all colle… view at source ↗
Figure 4
Figure 4. Figure 4: The x-coordinates correspond to the synthetic dataset being evaluated or finetuned on, identified by the percentage of GSM8K training samples present in the seed data. (a) Fidelity￾diversity scores with OT approximated by debiased Sinkhorn divergences, with entropic regularization hyperparameter = 0.001. The shaded regions in both plots correspond to 95% confidence intervals computed from bootstrapped esti… view at source ↗
Figure 5
Figure 5. Figure 5: Comparison of our metric (top) with precision and recall (bottom) using [PITH_FULL_IMAGE:figures/full_fig_p021_5.png] view at source ↗
Figure 7
Figure 7. Figure 7: Fidelity + diversity of swapped GSM8K-style datasets. [PITH_FULL_IMAGE:figures/full_fig_p028_7.png] view at source ↗
Figure 9
Figure 9. Figure 9: Exact-match test accuracies of Llama-3.2 base models finetuned on synthetic [PITH_FULL_IMAGE:figures/full_fig_p029_9.png] view at source ↗
Figure 6
Figure 6. Figure 6: Bootstrapped estimates of 95% confidence intervals of fidelity and diversity scores, [PITH_FULL_IMAGE:figures/full_fig_p031_6.png] view at source ↗
Figure 8
Figure 8. Figure 8: Training and validation curves from SFT on GSM8K. [PITH_FULL_IMAGE:figures/full_fig_p032_8.png] view at source ↗

discussion (0)

Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.

Reference graph

Works this paper leans on

52 extracted references · 8 linked inside Pith

  1. [1]

    A. Alaa, B. Van Breugel, E. S. Saveliev, and M. van der Schaar. How faithful is your synthetic data? sample-level metrics for evaluating and auditing generative models. In International Conference on Machine Learning. PMLR, 2022. 3

    2022

  2. [2]

    Alvarez-Melis and N

    D. Alvarez-Melis and N. Fusi. Geometric dataset distances via optimal transport. Advances in Neural Information Processing Systems, 33:21428–21439, 2020. 3

    2020

  3. [3]

    X. Bing, F. Bunea, and J. Niles-Weed. Estimation and inference for the wasserstein distance between mixing measures in topic models.Bernoulli, forthcoming, 2025. 3

    2025

  4. [4]

    X. Bing, F. Bunea, S. Strimas-Mackey, and M. Wegkamp. Likelihood estimation of sparse topic distributions in topic models and its applications to wasserstein document distance calculations.The Annals of Statistics, 50(6):3307–3333, 12 2022. 3

    2022

  5. [5]

    Celikyilmaz, E

    A. Celikyilmaz, E. Clark, and J. Gao. Evaluation of text generation: A survey.arXiv preprint arXiv:2006.14799, 2020. 3

    Pith/arXiv arXiv 2006

  6. [6]

    Chewi, J

    S. Chewi, J. Niles-Weed, and P. Rigollet.Statistical optimal transport. Springer, 2025. 5

    2025

  7. [7]

    Cobbe, V

    K. Cobbe, V. Kosaraju, M. Bavarian, M. Chen, H. Jun, L. Kaiser, M. Plappert, J. Tworek, J. Hilton, R. Nakano, et al. Training verifiers to solve math word problems.arXiv preprint arXiv:2110.14168, 2021. 2, 23

    Pith/arXiv arXiv 2021

  8. [8]

    Djolonga, M

    J. Djolonga, M. Lucic, M. Cuturi, O. Bachem, O. Bousquet, and S. Gelly. Precision-recall curves using information divergence frontiers. InInternational Conference on Artificial Intelligence and Statistics, pages 2550–2559. PMLR, 2020. 3

    2020

  9. [9]

    Feydy, T

    J. Feydy, T. Séjourné, F.-X. Vialard, S.-i. Amari, A. Trouve, and G. Peyré. Interpo- lating between optimal transport and mmd using sinkhorn divergences. InThe 22nd International Conference on Artificial Intelligence and Statistics, pages 2681–2690, 2019. 12

    2019

  10. [10]

    Friedman and A

    D. Friedman and A. B. Dieng. The vendi score: A diversity evaluation metric for machine learning.Transactions on Machine Learning Research, 2023. 3

    2023

  11. [11]

    Kauffmann, G

    S.Gunasekar, Y.Zhang, J.Aneja, C.C.T.Mendes, A.DelGiorno, S.Gopi, M.Javaheripi, P. Kauffmann, G. de Rosa, O. Saarikivi, et al. Textbooks are all you need.arXiv preprint arXiv:2306.11644, 2023. 1

    Pith/arXiv arXiv 2023

  12. [12]

    Hardt and E

    M. Hardt and E. Price. Tight bounds for learning a mixture of two gaussians. In Proceedings of the forty-seventh annual ACM symposium on Theory of computing, pages 753–760, 2015. 23

    2015

  13. [13]

    Heusel, H

    M. Heusel, H. Ramsauer, T. Unterthiner, B. Nessler, and S. Hochreiter. GANs Trained by a Two Time-Scale Update Rule Converge to a Local Nash Equilibrium. InAdvances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017. 2 14

    2017

  14. [14]

    Ho and X

    N. Ho and X. Nguyen. Convergence rates of parameter estimation for some weakly identifiable finite mixtures.The Annals of Statistics, 44(6):2726, 2016. 23

    2016

  15. [15]

    T. Ito, K. van Deemter, and J. Suzuki. Reference-free evaluation metrics for text generation: A survey.arXiv preprint arXiv:2501.12011, 2025. 3

    Pith/arXiv arXiv 2025

  16. [16]

    Jiralerspong, J

    M. Jiralerspong, J. Bose, I. Gemp, C. Qin, Y. Bachrach, and G. Gidel. Feature likelihood divergence: evaluating the generalization of generative models using samples. InAdvances in Neural Information Processing Systems, 2023. 3

    2023

  17. [17]

    Kusner, Y

    M. Kusner, Y. Sun, N. Kolkin, and K. Weinberger. From word embeddings to document distances. InInternational conference on machine learning, pages 957–966. PMLR, 2015. 3

    2015

  18. [18]

    Kynkäänniemi, T

    T. Kynkäänniemi, T. Karras, S. Laine, J. Lehtinen, and T. Aila. Improved precision and recall metric for assessing generative models.Advances in neural information processing systems, 32, 2019. 3

    2019

  19. [19]

    Le Bronnec, A

    F. Le Bronnec, A. Verine, B. Negrevergne, Y. Chevaleyre, and A. Allauzen. Exploring precision and recall to assess the quality and diversity of llms. InProceedings of the 62nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 11418–11441, 2024. 3, 20

    2024

  20. [20]

    B. Liu, S. Bubeck, R. Eldan, J. Kulkarni, Y. Li, A. Nguyen, R. Ward, and Y. Zhang. Tinygsm: achieving >8011

  21. [21]

    K. Lo, L. L. Wang, M. Neumann, R. Kinney, and D. S. Weld. S2orc: The semantic scholar open research corpus, 2020. 20

    2020

  22. [22]

    Loshchilov and F

    I. Loshchilov and F. Hutter. Decoupled weight decay regularization, 2019. 28

    2019

  23. [23]

    Magnusson, A

    I. Magnusson, A. Bhagia, V. Hofmann, L. Soldaini, A. H. Jha, O. Tafjord, D. Schwenk, E. P. Walsh, Y. Elazar, K. Lo, D. Groeneveld, I. Beltagy, H. Hajishirzi, N. A. Smith, K. Richardson, and J. Dodge. Paloma: A benchmark for evaluating language model fit,

  24. [24]

    Manole and N

    T. Manole and N. Ho. Uniform convergence rates for maximum likelihood estimation under two-component gaussian mixture models.arXiv preprint arXiv:2006.00704, 2020. 23

    Pith/arXiv arXiv 2006

  25. [25]

    Manole and N

    T. Manole and N. Ho. Refined convergence rates for maximum likelihood estimation under finite mixture models. InInternational Conference on Machine Learning, pages 14979–15006. PMLR, 2022. 3

    2022

  26. [26]

    M. F. Naeem, S. J. Oh, Y. Uh, Y. Choi, and J. Yoo. Reliable fidelity and diversity metrics for generative models. InInternational Conference on Machine Learning. PMLR,

  27. [27]

    Nguyen, N

    H. Nguyen, N. Ho, and A. Rinaldo. Sigmoid gating is more sample efficient than softmax gating in mixture of experts.Advances in Neural Information Processing Systems, 37:118357–118388, 2024. 3

    2024

  28. [28]

    X. Nguyen. Convergence of latent mixing measures in finite and infinite mixture models. The Annals of Statistics, pages 370–400, 2013. 3

    2013

  29. [29]

    X. Nguyen. Posterior contraction of the population polytope in finite admixture models. Bernoulli, pages 618–646, 2015. 3

    2015

  30. [30]

    Achiam, S

    OpenAI, J. Achiam, S. Adler, S. Agarwal, L. Ahmad, I. Akkaya, F. L. Aleman, D. Almeida, J. Altenschmidt, S. Altman, S. Anadkat, R. Avila, I. Babuschkin, S. Bal- aji, V. Balcom, P. Baltescu, H. Bao, M. Bavarian, J. Belgum, I. Bello, J. Berdine, G. Bernadett-Shapiro, C. Berner, L. Bogdonoff, O. Boiko, M. Boyd, A.-L. Brakman, G. Brockman, T. Brooks, M. Brund...

  31. [31]

    Peyré and M

    G. Peyré and M. Cuturi.Computational optimal transport: With applications to data science. Now Foundations and Trends, 2019. 5

    2019

  32. [32]

    Pillutla, L

    K. Pillutla, L. Liu, J. Thickstun, S. Welleck, S. Swayamdipta, R. Zellers, S. Oh, Y. Choi, and Z. Harchaoui. Mauve scores for generative models: Theory and practice. arXiv:2212.14578, 2023. 3

    Pith/arXiv arXiv 2023

  33. [33]

    Pillutla, S

    K. Pillutla, S. Swayamdipta, R. Zellers, J. Thickstun, S. Welleck, Y. Choi, and Z. Har- chaoui. Mauve: Measuring the gap between neural text and human text using divergence frontiers.Advances in Neural Information Processing Systems, 34:4816–4828, 2021. 1, 3

    2021

  34. [34]

    Räisä, B

    O. Räisä, B. Van Breugel, and M. Van Der Schaar. Position: All current generative fidelity and diversity metrics are flawed.arXiv preprint arXiv:2505.22450, 2025. 13

    Pith/arXiv arXiv 2025

  35. [35]

    M. Reid, V. Zhong, S. Gururangan, and L. Zettlemoyer. M2d2: A massively multi- domain language modeling dataset. InProceedings of the 2022 Conference on Empirical Methods in Natural Language Processing, pages 964–975, 2022. 2, 8

    2022

  36. [36]

    Reimers and I

    N. Reimers and I. Gurevych. Sentence-bert: Sentence embeddings using siamese bert- networks. InProceedings of the 2019 conference on empirical methods in natural language processing and the 9th international joint conference on natural language processing (EMNLP-IJCNLP), pages 3982–3992, 2019. 4

    2019

  37. [37]

    Rioux, A

    G. Rioux, A. Nitsure, M. Rigotti, K. Greenewald, and Y. Mroueh. Multivariate stochastic dominance via optimal transport and applications to models benchmarking.Advances in Neural Information Processing Systems, 37:39190–39223, 2024. 3

    2024

  38. [38]

    M. S. Sajjadi, O. Bachem, M. Lucic, O. Bousquet, and S. Gelly. Assessing generative models via precision and recall.Advances in neural information processing systems, 31,

  39. [39]

    Salimans, I

    T. Salimans, I. Goodfellow, W. Zaremba, V. Cheung, A. Radford, and X. Chen. Improved techniques for training gans.Advances in neural information processing systems, 29,

  40. [40]

    Singh, A

    A. Singh, A. Fry, A. Perelman, A. Tart, A. Ganesh, A. El-Kishky, A. McLaughlin, A. Low, A. Ostrow, A. Ananthram, A. Nathan, A. Luo, A. Helyar, A. Madry, A. Efre- mov, A. Spyra, A. Baker-Whitcomb, A. Beutel, A. Karpenko, A. Makelov, A. Neitz, A. Wei, A. Barr, A. Kirchmeyer, A. Ivanov, A. Christakis, A. Gillespie, A. Tam, A. Bennett, A. Wan, A. Huang, A. M....

    2026

  41. [41]

    Stein, J

    G. Stein, J. Cresswell, R. Hosseinzadeh, Y. Sui, B. Ross, V. Villecroze, Z. Liu, A. L. Caterini, E. Taylor, and G. Loaiza-Ganem. Exposing flaws of generative model evaluation metrics and their unfair treatment of diffusion models.Advances in Neural Information Processing Systems, 2023. 13

    2023

  42. [42]

    D. Su, K. Kong, Y. Lin, J. Jennings, B. Norick, M. Kliegl, M. Patwary, M. Shoeybi, and B. Catanzaro. Nemotron-cc: Transforming common crawl into a refined long-horizon pretraining dataset, 2025. 11

    2025

  43. [43]

    L. Team. (aaron grattafiori et al) the llama 3 herd of models.arXiv:2407.21783, 2024. 12, 28

    Pith/arXiv arXiv 2024

  44. [44]

    Toshniwal, W

    S. Toshniwal, W. Du, I. Moshkov, B. Kisacanin, A. Ayrapetyan, and I. Gitman. Openmathinstruct-2: Accelerating ai for math with massive open-source instruction data, 2024. 11

    2024

  45. [45]

    Villani.Topics in Optimal Transportation, volume 58

    C. Villani.Topics in Optimal Transportation, volume 58. American Mathematical Soc.,

  46. [46]

    L. Wang, N. Yang, X. Huang, B. Jiao, L. Yang, D. Jiang, R. Majumder, and F. Wei. Text embeddings by weakly-supervised contrastive pre-training, 2024. 9

    2024

  47. [47]

    Y. Wang, Y. Kordi, S. Mishra, A. Liu, N. A. Smith, D. Khashabi, and H. Hajishirzi. Self-instruct: Aligning language models with self-generated instructions. InProceedings of the 61st annual meeting of the association for computational linguistics (volume 1: long papers), pages 13484–13508, 2023. 1

    2023

  48. [48]

    Wasserman.All of statistics: a concise course in statistical inference, volume 26

    L. Wasserman.All of statistics: a concise course in statistical inference, volume 26. Springer, 2004. 23

    2004

  49. [49]

    Z. Yang, N. Band, S. Li, E. Candès, and T. Hashimoto. Synthetic continued pretraining,

  50. [50]

    Yurochkin, S

    M. Yurochkin, S. Claici, E. Chien, F. Mirzazadeh, and J. M. Solomon. Hierarchical op- timal transport for document representation.Advances in neural information processing systems, 32, 2019. 3

    2019

  51. [51]

    Zhang, V

    T. Zhang, V. Kishore, F. Wu, K. Q. Weinberger, and Y. Artzi. Bertscore: Evaluating text generation with bert. InInternational Conference on Learning Representations,

  52. [52]

    high” and verbosity=“medium

    1 A Comparisons to Precision & Recall While our metrics bear a strong resemblance to the precision and recall metrics put forth in [19], we point out conceptual differences in our approaches. At a high level, their precision and recall scores quantify overlap in the supports of the evaluation and reference distributions P and Q. Precision captures the pro...

    1961