Pith. sign in

REVIEW 4 major objections 7 minor 1 cited by

Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models

T0 review · 4 major / 7 minor · reviewed 2026-08-06 · deepseek-v4-flash

Challenge this review Re-run · record.json Download PDF Read on arXiv ↗

Pith's one-line read This paper claims that instruction-tuning creates a measurable diversity gap in narrative generation, that DPO is the main culprit, and that mixing the instruct model's log-probabilities with its base model's restores much of the gap…

desk verdict A careful empirical demonstration that DPO is the main diversity-reducing stage for OLMo, plus a simple decoding fix that mostly works — main gap is lack of human validation. read the letter →

arxiv 2507.20956 v1 pith:DHKIQ34W submitted 2025-07-28 cs.CL cs.AI

Max Peeperkorn , Tom Kouwenhoven , Dan Brown , Anna Jordanous This is my paper
classification cs.CLcs.AI
keywords outputdiversityinstruction-tuningdirectpreferenceoptimizationconformativedecodingnarrativegenerationVendiScorestrategieslargelanguagemodels
verification ladder T0 review T1 audit T2 compute T3 formal

The pith

A machine-rendered reading of the paper's core claim, the machinery that carries it, and where it could break.

The reading

Instruction-tuning narrows what a language model will write: across five open-weight LLMs, the instruction-tuned versions produce significantly less diverse stories than their base models from the same writing prompts, with the drop visible in every measured diversity metric. The paper traces this 'diversity gap' through the OLMo fine-tuning chain and finds that direct preference optimization (DPO) is the stage that removes the most diversity, while the chat template used at inference time narrows output further. To counter the loss, the paper introduces conformative decoding, which mixes the next-token log-probabilities of an instruct model with those of its base model over the truncated candidate vocabulary, and shows that this typically restores diversity while maintaining or slightly improving automatic quality scores. The practical reason to care is that creative writing, brainstorming, and agent-based simulations all need models that can produce many different plausible outputs from one prompt, and this method offers a model-agnostic way to get that diversity back without retraining.

What carries the argument

Conformative decoding is a sampling strategy defined by the mixture $$\log p_{\$\theta$}(x_t \mid x_{<t}) \text{ mixed with } (1-\gamma) \log p_{\phi}(x_t \mid x_{<t})$$ for tokens in the valid set $V_{\text{valid}}$ produced by a truncation strategy such as nucleus sampling, where $\theta$ is the instruct model, $\phi$ is its base model, and $\gamma$ controls the blend (set to $0.5$ in the experiments). It works by asking the instruction-tuned model to conform to the wider next-token distribution of its base model while still sampling only from tokens the instruct model considers plausible, which the paper reports as a way to reintroduce diversity without the degenerate output that comes from untruncated mixing. The other load-bearing machinery is the measurement suite: Vendi Score with n-gram counts and jina-embedding features, Truncated Entropy, MAUVE, and Improved Precision and Recall.

What would settle it

A reader study in which people rank sets of stories from base, instruct, and conformative-decoding models for perceived variety would settle the claim: if reader rankings do not follow the Vendi Score and Truncated Entropy ordering, the measured diversity gap and recovery would be artifacts of the metrics rather than properties of the texts.

Watch

Extended reading notes

Core claim

On a narrative generation task built from a curated subset of the r/WritingPrompts dataset, the paper shows that instruction-tuned versions of Gemma 2 9B, Llama 3.1 8B, Mistral 7B v0.3, OLMo 7B, and OLMo 2 7B generate significantly less diverse stories than their base models: per-prompt Vendi Score (with n-gram and jina-embedding features) and Truncated Entropy all drop with $p < .001$, and recall-based coverage of human reference stories falls while precision rises. Stage-wise inspection of OLMo and OLMo 2 attributes most of the loss to DPO, with SFT contributing less and RLVR little. The paper further claims that conformative decoding — mixing the instruct model's next-token log-probabilities with the base model's over the nucleus-sampled vocabulary — significantly increases per-prompt diversity for all tested models except Llama 3.1 8B on the lexical n-gram measures, and improves or maintains cross-prompt diversity, MAUVE, and improved precision and recall.

Load-bearing premise

The load-bearing premise is that the automatic scoring measures used in the study reflect the kind of story diversity a human reader would notice, since no human evaluation was performed to confirm it.

Editorial extensions

If this is right

  • Users of open-weight instruct models can sample more varied stories from the same prompt by keeping the base checkpoint and applying conformative decoding, without retraining or prompt engineering.
  • Preference-tuning pipelines that care about output diversity can target DPO as the main source of collapse, monitoring diversity at that stage and possibly adjusting preference data or regularization.
  • Because the chat template itself further narrows outputs, applications that value diversity may prefer completion-style prompting or should account for the template's effect when designing evaluations.
  • The method's failure on Llama 3.1 8B per-prompt lexical diversity suggests conformative decoding is most useful for models with a large diversity gap, and the paper explicitly invites follow-up on the relationship between gap size and improvement.

Reading between the lines

Editorial extensions of the paper, not claims the author makes directly.

  • If the automatic metrics track human perception, conformative decoding could be used in creative co-writing tools to offer more distinct story options per prompt; a human study would be needed to verify that, and the paper does not report one.
  • The same logit-mixture idea could be extended to other truncation schemes such as top-k or locally typical sampling, or reformulated as a beam-search objective analogous to contrastive decoding, which the paper mentions as future work but does not test.
  • A direct test of the DPO-diversity link would be to measure the entropy of preference datasets or reward-model scores: if preference pairs systematically reward a narrow style, that would explain why DPO compresses output diversity and suggest dataset-level remedies.
Share X Bluesky LinkedIn Reddit HN

Editorial analysis

A structured set of objections, weighed in public.

Desk editor's note, referee report, and a circularity audit.

Referee Report

4 major / 7 minor

Summary. The paper investigates the reduction in output diversity caused by instruction-tuning of large language models. On a narrative generation task based on 53 writing prompts from the Writing Prompts dataset, the authors compare five open-weight base/instruct model pairs and report significant per-prompt decreases in Vendi Score (n-gram and jina-embedding features) and Truncated Entropy for instruction-tuned models, along with decreased Recall and increased Precision in across-prompt evaluations. Using intermediate OLMo checkpoints, they attribute the largest diversity loss to the DPO fine-tuning stage. They then propose conformative decoding, which mixes the next-token log-probabilities of the instruct model and its base model over the nucleus-sampled vocabulary of the instruct model, with a fixed mixing weight gamma=0.5. Experiments show that conformative decoding significantly increases per-prompt diversity for most models while maintaining or slightly improving automatic quality metrics.

Significance. If valid, the paper makes two useful contributions: it provides further evidence, with open models and intermediate checkpoints, that DPO is a major driver of the diversity drop, and it offers a simple, training-free decoding intervention that can partially restore diversity in instruction-tuned models. The experimental design is thoughtful: incipits are used to align base and instruct behavior, multiple diversity metrics are reported, and paired significance tests are performed. The main caveats are that the per-prompt diversity metrics are computed on only 50 samples per prompt without a small-sample sensitivity analysis, and that the diversity and quality claims rest entirely on automatic metrics in the absence of human evaluation, a limitation the authors acknowledge. The method is not a fitting procedure and does not bake in the evaluation metrics, so the reported improvements are empirical.

major comments (4)
  1. [§3.1.4] The similarity kernel k used in Eq. (1) is never specified for either the n-gram features or the jina embeddings. For example, the manuscript does not state whether the n-gram count vectors are L2-normalized before cosine similarity is computed, or whether the jina embeddings are used in raw or normalized form. Without this information, the numerical diversity scores cannot be reproduced, and differences among models could partly stem from kernel choices. Please specify the kernel for each feature type and the exact jina-embeddings-v3 configuration (task LoRA, pooling, normalization).
  2. [§3.1.4 and §4.3] All per-prompt diversity metrics (VSjina, VSn-gram, TEjina) are computed with N=50 samples per prompt. Vendi Score is bounded by N, and Truncated Entropy uses eigenvalue estimates of a covariance matrix that can be singular when N is not much larger than the embedding dimension. The manuscript does not report how stable the measured gaps and conformative-decoding improvements are with respect to N. I request an N-sensitivity analysis: recompute the per-prompt metrics on random subsamples of the 50 generations (e.g., N=10, 20, 30, 40) and show that the base-versus-instruct gap and the baseline-versus-conformative difference do not change direction or order of magnitude. This is important because the headline claim that conformative decoding 'typically increases diversity' is based on these small-sample metrics.
  3. [§3.2] The claim that DPO has the largest impact on diversity is based on visual inspection of Figure 2 rather than on paired statistical tests. Since the OLMo checkpoints allow a stage-by-stage comparison on the same 53 prompts, the authors should report paired one-tailed tests (or confidence intervals) comparing the SFT checkpoint with the DPO checkpoint for each per-prompt metric, and likewise for the RLVR checkpoint. Without such tests, contribution point 2 ('DPO is primarily responsible for the diversity gap') is not statistically supported.
  4. [§4.3] The per-prompt improvements from conformative decoding are statistically significant (p < .001) but are described as modest, and no effect sizes are reported. With N=50 generations per prompt and 53 prompts, a one-tailed t-test can detect a small systematic metric shift that may not correspond to a perceptible diversity difference in narrative text. Please report effect sizes (e.g., paired Cohen's d) for the significant comparisons and, ideally, include example narratives from configuration A and B to illustrate the type and magnitude of diversity change. This would help the reader judge the practical relevance of the improvements.
minor comments (7)
  1. [§3.1.2] The incipit length is fixed at 20 tokens without any sensitivity analysis; since the incipit introduces shared content across all 50 generations, a different length could change the diversity measurements. The authors should at least discuss this choice or provide a robustness check.
  2. [§3.1.4] The significance tests do not account for multiple comparisons (6 per-prompt metrics × 5 models = 30 tests, plus the stage-wise tests). The authors should state whether any correction was applied; given the very low p-values this is unlikely to alter conclusions, but it should be explicit.
  3. [§4.1, Eq. (3)] The mixture of log-probabilities should be followed by renormalization over the truncated vocabulary Vvalid; this is implied by the softmax step mentioned in the text, but it would be clearer to include the explicit normalization in the equation or its surrounding text.
  4. [§2.2 and §3.1.4] The notation in Eq. (2) uses λᵢ^(l) without defining l; the text later refers to the latent space, but the equation should define it.
  5. [Figure 1] The caption of Figure 1 states p < .001 for all comparisons, while the text reports p < .01 for some chat-template comparisons; please make the captions consistent with the exact p-value thresholds reported in the text.
  6. [Appendix A and References] Appendix A contains a typo, 'confirmation decoding' instead of 'conformative decoding'; the bibliography entry for Sturua et al. appears to be duplicated.
  7. [Limitations] The limitations section acknowledges the absence of human evaluation; I encourage the authors to add a small human perception study on a subset of prompts to validate that the metric improvements translate to perceived diversity differences.

Circularity Check

0 steps flagged · score 1.0 of 10

No significant circularity: the diversity gap is measured on public checkpoints with fixed metrics, the DPO attribution is an empirical checkpoint comparison, and conformative decoding uses a fixed gamma=0.5 mixture evaluated on the same task rather than a fitted parameter.

full rationale

The paper's derivation chain is not circular. Section 3 measures the diversity gap by comparing base and instruct versions of public open-weight models under fixed decoding settings ('we use nucleus sampling (p = .95) with temperature t = 1') and fixed, pre-defined metrics (Vendi Score, Truncated Entropy, MAUVE, Improved Precision/Recall). No parameter is fitted to the diversity outcome; the per-prompt comparisons are tested with one-tailed paired t-tests. The DPO attribution is an empirical measurement across OLMo intermediate checkpoints, not an optimization that bakes in the conclusion. Conformative decoding (Eq. 3) mixes the instruct and base log-probability distributions over the instruct model's nucleus-sampled vocabulary, with gamma chosen explicitly as a neutral value: 'We choose gamma = .5 as this is a neutral value; higher gamma would increase diversity and vice versa, but we leave further hyperparameter tuning for future work.' Because the truncation set Vvalid is determined by the instruct model alone and gamma is not tuned, the observed diversity increase is an empirical result rather than a mathematical identity. The paper's self-citations (e.g., Peeperkorn et al. 2024 on temperature, Jordanous and Keller 2016 on creativity components, Robinson et al. 2020 on music diversity, Ackerman and Brown 2024 on representation) are background or supporting references and are not load-bearing for the central quantitative claims. The Limitations section openly flags the difficulty of assessing narrative diversity and the absence of human evaluation; this is a correctness and validity concern, not circularity. Overall, the central claims are supported by measurements on held-out tasks with fixed evaluation protocols, so no step reduces by construction to its own inputs.

Assumptions & free parameters 2 free parameters · 3 assumptions · 1 invented entities

The paper introduces one hyperparameter (gamma = 0.5), one hand-set incipit length, and several domain assumptions about metric validity. It does not fit parameters to data to produce its central claim, but it does assume that the chosen automatic metrics are faithful proxies for narrative diversity, which the authors themselves flag as an open limitation.

free parameters (2)
  • gamma = 0.5
    Mixing weight in conformative decoding (Equation 3). Chosen as a neutral value; no tuning or sensitivity analysis is provided.
  • incipit length = 20 tokens
    First 20 tokens of each human response are used to prompt all models. The choice is made to align base and instruct behavior but is an arbitrary hand-set value.
assumptions (3)
  • domain assumption The Writing Prompts dataset, as a human reference, represents a suitably diverse ground truth for narrative diversity.
    Section 3.1.1 treats the human stories as a diverse reference for Recall and MAUVE, following Fan et al. 2018, and the paper relies on this for the claim that instruct-tuned models cover less of the reference manifold.
  • domain assumption Automatic embedding-based diversity metrics (Vendi Score with jina embeddings, Truncated Entropy) meaningfully capture narrative diversity.
    The paper's entire measurement of the diversity gap and its fix depends on this. The Limitations section explicitly notes that narrative diversity is hard to assess and no human evaluation was performed.
  • domain assumption Nucleus sampling with p = .95 is a fair and sufficient baseline for diversity comparison.
    All generations use p = .95, t = 1, and the paper notes truncation restricts diversity. The choice of truncation directly affects the measured diversity gap and the effect of conformative decoding.
invented entities (1)
  • Conformative decoding
    purpose: A decoding strategy that mixes instruct and base model log-probabilities over the nucleus-sampled vocabulary to reintroduce diversity.
    This is a new algorithmic procedure, not an empirical entity. Its effectiveness is only demonstrated on the paper's own automatic metrics, and no outside replication or downstream validation is provided.

how reviews work

0 comments
Cite this review

Pith. "Pith review of Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models." pith.science (2026). https://pith.science/paper/DHKIQ34W

@misc{pith2026250720956,
  author       = {Pith},
  title        = {Pith review of: Mind the Gap: Conformative Decoding to Improve Output Diversity of Instruction-Tuned Large Language Models},
  year         = {2026},
  howpublished = {\url{https://pith.science/paper/DHKIQ34W}},
  note         = {Machine review of arXiv:2507.20956}
}
read the original abstract

Instruction-tuning large language models (LLMs) reduces the diversity of their outputs, which has implications for many tasks, particularly for creative tasks. This paper investigates the ``diversity gap'' for a writing prompt narrative generation task. This gap emerges as measured by current diversity metrics for various open-weight and open-source LLMs. The results show significant decreases in diversity due to instruction-tuning. We explore the diversity loss at each fine-tuning stage for the OLMo and OLMo 2 models to further understand how output diversity is affected. The results indicate that DPO has the most substantial impact on diversity. Motivated by these findings, we present a new decoding strategy, conformative decoding, which guides an instruct model using its more diverse base model to reintroduce output diversity. We show that conformative decoding typically increases diversity and even maintains or improves quality.

Figures

Figures reproduced from arXiv: 2507.20956 by the authors.

Figure 1
Figure 1. We compare the base and instruct models for [PITH_FULL_IMAGE:figures/full_fig_p005_1.png] view at source ↗
Figure 2
Figure 2. This figure presents experiments that investi [PITH_FULL_IMAGE:figures/full_fig_p006_2.png] view at source ↗
Figure 3
Figure 3. This figure shows automatic evaluations for [PITH_FULL_IMAGE:figures/full_fig_p007_3.png] view at source ↗

Discussion (0). Sign in to comment.

Forward citations

Cited by 1 Pith paper

Reviewed papers in the Pith corpus that reference this work. Sorted by Pith novelty score. Full citation record

  1. The Homogenization Problem in LLMs: Towards Meaningful Diversity in AI Safety

    cs.AI 2026-01 unverdicted novelty 6.0 of 10

    The paper formalizes homogenization in LLMs as a loss of deviance and core entropy, and proposes xeno-reproduction—a structure-aware diversity-pursuit objective—with a proof that diversity and fairness trade off.

Reference graph

Works this paper leans on

56 extracted references · 32 canonical work pages · cited by 1 Pith paper

  1. [1]

    Margareta Ackerman and Daniel Brown. 2024. Depictions of jews in large generative models. In 15th International Conference on Computational Creativity, pages 342--347

    work page 2024

  2. [2]

    Tom Brown, Benjamin Mann, Nick Ryder, Melanie Subbiah, Jared D Kaplan, Prafulla Dhariwal, Arvind Neelakantan, , and Dario Amodei. 2020. https://proceedings.neurips.cc/paper_files/paper/2020/file/1457c0d6bfcb4967418bfb8ac142f64a-Paper.pdf Language models are few-shot learners . In Advances in Neural Information Processing Systems, volume 33, pages 1877--1901

    work page 2020

  3. [3]

    Jiaju Chen, Chongming Gao, Shuai Yuan, Shuchang Liu, Qingpeng Cai, and Peng Jiang. 2025. https://arxiv.org/abs/2408.12470 DLCRec : A novel approach for managing diversity in LLM -based recommender systems . Preprint, arXiv:2408.12470. ArXiv:2408.12470

    work page Pith review arXiv 2025

  4. [4]

    KuanChao Chu, Yi-Pei Chen, and Hideki Nakayama. 2024. https://arxiv.org/abs/2412.21102 Exploring and controlling diversity in LLM -agent conversation . Preprint, arXiv:2412.21102. ArXiv:2412.21102

    arXiv 2024

  5. [5]

    Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A

    Shibhansh Dohare, J. Fernando Hernandez-Garcia, Qingfeng Lan, Parash Rahman, A. Rupam Mahmood, and Richard S. Sutton. 2024. https://doi.org/10.1038/s41586-024-07711-7 Loss of plasticity in deep continual learning . Nature, 632(8026):768--774

    doi:10.1038/s41586-024-07711-7 2024

  6. [6]

    Angela Fan, Mike Lewis, and Yann Dauphin. 2018. https://doi.org/10.18653/v1/P18-1082 Hierarchical neural story generation . In 56th Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 889--898. Association for Computational Linguistics

    doi:10.18653/v1/p18-1082 2018

  7. [7]

    Matthew Finlayson, John Hewitt, Alexander Koller, Swabha Swayamdipta, and Ashish Sabharwal. 2024. https://openreview.net/forum?id=dONpC9GL1o Closing the curious case of neural text degeneration . In 12th International Conference on Learning Representations

    work page 2024

  8. [8]

    Kathleen Fraser, Svetlana Kiritchenko, and Isar Nejadgholi. 2023. Diversity is not a one-way street: pilot study on ethical interventions for racial bias in text-to-image systems. In 14th International Conference on Computational Creativity, pages 288--292

    work page 2023

Show all 56 references
  1. [9]

    Dan Friedman and Adji Bousso Dieng. 2023. https://openreview.net/forum?id=g97OHbQyk1 The Vendi Score : A diversity evaluation metric for machine learning . Transactions on Machine Learning Research

    2023

  2. [10]

    Fabricio Goes, Zisen Zhou, Piotr Sawicki, Marek Grze \'s , and Daniel G Brown. 2022. https://arxiv.org/abs/2212.11214 Crowd score: A method for the evaluation of jokes using large language model AI voters as judges . Preprint, arXiv:2212.11214

    2022 arXiv

  3. [11]

    Kazjon Grace and Mary Lou Maher. 2016. https://doi.org/10.1609/aaai.v30i1.9890 Surprise-triggered reformulation of design goals . Proceedings of the AAAI Conference on Artificial Intelligence, 30(1)

    2016 doi

  4. [12]

    Aaron Grattafiori, Abhimanyu Dubey, Abhinav Jauhri, Abhinav Pandey, Abhishek Kadian, Ahmad Al-Dahle, Aiesha Letman, Akhil Mathur, Alan Schelten, Alex Vaughan, Amy Yang, Angela Fan, Anirudh Goyal, Anthony Hartshorn, Aobo Yang, Archi Mitra, Archie Sravankumar, Artem Korenev, Art...

    2024 arXiv

  5. [13]

    Dirk Groeneveld, Iz Beltagy, Pete Walsh, Akshita Bhagia, Rodney Kinney, Oyvind Tafjord, Ananya Harsh Jha, Hamish Ivison, Ian Magnusson, Yizhong Wang, Shane Arora, David Atkinson, Russell Authur, Khyathi Raghavi Chandu, Arman Cohan, Jennifer Dumas, Yanai Elazar, Yuling Gu, Jack...

    2024 arXiv

  6. [14]

    a m\" a l\

    Perttu H\" a m\" a l\" a inen, Mikke Tavast, and Anton Kunnari. 2023. https://doi.org/10.1145/3544548.3580688 Evaluating large language models in generating synthetic hci research data: a case study . In 2023 CHI Conference on Human Factors in Computing Systems, CHI '23. Assoc...

    2023

  7. [15]

    Martin Heusel, Hubert Ramsauer, Thomas Unterthiner, Bernhard Nessler, and Sepp Hochreiter. 2017. https://proceedings.neurips.cc/paper_files/paper/2017/file/8a1d694707eb0fefe65871369074926d-Paper.pdf GANs trained by a two time-scale update rule converge to a local nash equilibr...

    2017

  8. [16]

    Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. https://openreview.net/forum?id=rygGQyrFvH The curious case of neural text degeneration . In 8th International Conference on Learning Representations, ICLR 2020

    2020

  9. [17]

    Francisco Ibarrola and Kazjon Grace. 2024. Measuring diversity in co-creative image generation. In 15th International Conference on Computational Creativity

    2024

  10. [18]

    Francisco Ibarrola, Tomas Lawton, and Kazjon Grace. 2024. https://doi.org/10.1109/TVCG.2023.3293853 A collaborative, interactive and context-aware drawing agent for co-creative design . IEEE Transactions on Visualization and Computer Graphics, 30(8):5525--5537

    2024

  11. [19]

    Albert Q. Jiang, Alexandre Sablayrolles, Arthur Mensch, Chris Bamford, Devendra Singh Chaplot, Diego de las Casas, Florian Bressand, Gianna Lengyel, Guillaume Lample, Lucile Saulnier, Lélio Renard Lavaud, Marie-Anne Lachaux, Pierre Stock, Teven Le Scao, Thibaut Lavril, Thomas ...

    2023 arXiv

  12. [20]

    Guangyuan Jiang, Manjie Xu, Song-Chun Zhu, Wenjuan Han, Chi Zhang, and Yixin Zhu. 2023 b . https://proceedings.neurips.cc/paper_files/paper/2023/file/21f7b745f73ce0d1f9bcea7f40b1388e-Paper-Conference.pdf Evaluating and inducing personality in pre-trained language models . In A...

    2023

  13. [21]

    Anna Jordanous and Bill Keller. 2016. https://doi.org/10.1371/journal.pone.0162959 Modelling creativity: Identifying key components through a corpus-based approach . PLoS ONE, 11(10)

    2016 doi

  14. [22]

    Robert Kirk, Ishita Mediratta, Christoforos Nalmpantis, Jelena Luketina, Eric Hambro, Edward Grefenstette, and Roberta Raileanu. 2024. https://openreview.net/forum?id=PXD3FAVHJT Understanding the effects of RLHF on LLM generalisation and diversity . In 12th International Confe...

    2024

  15. [23]

    Tuomas Kynk\" a \" a nniemi, Tero Karras, Samuli Laine, Jaakko Lehtinen, and Timo Aila. 2019. https://proceedings.neurips.cc/paper_files/paper/2019/file/0234c510bc6d908b28c70ff313743079-Paper.pdf Improved precision and recall metric for assessing generative models . In Advance...

    2019

  16. [24]

    Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D

    Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luc...

    2025 arXiv

  17. [25]

    Florian Le Bronnec, Alexandre Verine, Benjamin Negrevergne, Yann Chevaleyre, and Alexandre Allauzen. 2024. https://doi.org/10.18653/v1/2024.acl-long.616 Exploring precision and recall to assess the quality and diversity of LLM s . In 62nd Annual Meeting of the Association for ...

    2024 doi

  18. [26]

    Tom Leinster. 2021. Entropy and Diversity: The Axiomatic Approach. Cambridge University Press

    2021

  19. [27]

    Jiwei Li, Michel Galley, Chris Brockett, Jianfeng Gao, and Bill Dolan. 2016. https://doi.org/10.18653/v1/N16-1014 A diversity-promoting objective function for neural conversation models . In 2016 Conference of the N orth A merican Chapter of the Association for Computational L...

    2016 doi

  20. [28]

    Xiang Lisa Li, Ari Holtzman, Daniel Fried, Percy Liang, Jason Eisner, Tatsunori Hashimoto, Luke Zettlemoyer, and Mike Lewis. 2023. https://doi.org/10.18653/v1/2023.acl-long.687 Contrastive decoding: Open-ended text generation as optimization . In 61st Annual Meeting of the Ass...

    2023 doi

  21. [29]

    Clara Meister, Tiago Pimentel, Gian Wiher, and Ryan Cotterell. 2023. https://doi.org/10.1162/tacl_a_00536 Locally typical sampling . Transactions of the Association for Computational Linguistics, 11:102--121

    2023 doi

  22. [30]

    Niklas Muennighoff, Nouamane Tazi, Loïc Magne, and Nils Reimers. 2023. https://arxiv.org/abs/2210.07316 MTEB : Massive text embedding benchmark . Preprint, arXiv:2210.07316. ArXiv:2210.07316

    2023 arXiv

  23. [31]

    Moin Nadeem, Tianxing He, Kyunghyun Cho, and James Glass. 2020. https://aclanthology.org/2020.aacl-main.36 A systematic characterization of sampling algorithms for open-ended language generation . In 1st Conference of the Asia-Pacific Chapter of the Association for Computation...

    2020

  24. [32]

    Vishakh Padmakumar and He He. 2024. https://openreview.net/forum?id=Feiz5HtCD0 Does writing with language models reduce content diversity? In 12th International Conference on Learning Representations

    2024

  25. [33]

    Bernstein

    Joon Sung Park, Joseph O'Brien, Carrie Jun Cai, Meredith Ringel Morris, Percy Liang, and Michael S. Bernstein. 2023. https://doi.org/10.1145/3586183.3606763 Generative agents: Interactive simulacra of human behavior . In 36th Annual ACM Symposium on User Interface Software and...

    2023

  26. [34]

    Max Peeperkorn, Tom Kouwenhoven, Dan Brown, and Anna Jordanous. 2024. Is temperature the creativity parameter of large language models? In 15th International Conference on Computational Creativity, pages 226--235

    2024

  27. [35]

    Krishna Pillutla, Lang Liu, John Thickstun, Sean Welleck, Swabha Swayamdipta, Rowan Zellers, Sewoong Oh, Yejin Choi, and Zaid Harchaoui. 2023. http://jmlr.org/papers/v24/23-0023.html Mauve scores for generative models: Theory and practice . Journal of Machine Learning Research...

    2023

  28. [36]

    Krishna Pillutla, Swabha Swayamdipta, Rowan Zellers, John Thickstun, Sean Welleck, Yejin Choi, and Zaid Harchaoui. 2021. https://proceedings.neurips.cc/paper_files/paper/2021/file/260c2432a0eecc28ce03c10dadc078a4-Paper.pdf Mauve: Measuring the gap between neural text and human...

    2021

  29. [37]

    Tiago Pimentel, Clara Isabel Meister, and Ryan Cotterell. 2023. https://openreview.net/forum?id=bvpkw7UIRdU On the usefulness of embeddings, clusters and strings for text generation evaluation . In The 11th International Conference on Learning Representations

    2023

  30. [38]

    Rafael Rafailov, Archit Sharma, Eric Mitchell, Christopher D Manning, Stefano Ermon, and Chelsea Finn. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/a85b405ed65c6477a4fe8302b5e06ce7-Paper-Conference.pdf Direct preference optimization: Your language model is ...

    2023

  31. [39]

    Kyle Robinson, Dan Brown, and Markus Schedl. 2020. https://doi.org/10.5281/zenodo.4245464 User insights on diversity in music recommendation lists . In 21st International Society for Music Information Retrieval Conference, pages 446--453

    2020 doi

  32. [40]

    Mehdi S. M. Sajjadi, Olivier Bachem, Mario Lucic, Olivier Bousquet, and Sylvain Gelly. 2018. Assessing generative models via precision and recall. In Advances in Neural Information Processing Systems, volume 31

    2018

  33. [41]

    Siu, Byron C

    Chantal Shaib, Joe Barrow, Jiuding Sun, Alexa F. Siu, Byron C. Wallace, and Ani Nenkova. 2025. https://arxiv.org/abs/2403.00553 Standardizing the measurement of text diversity: A tool and a comparative analysis of scores . Preprint, arXiv:2403.00553

    2025

  34. [42]

    Mike Sharples and Rafael Pérez y Pérez. 2022. https://doi.org/10.4324/9781003161431 Story machines: How computers have become creative writers , 1st edition. Routledge

    2022 doi

  35. [43]

    Juliana Shihadeh and Margareta Ackerman. 2023 a . Shattering bias: A path to bridging the gender divide with creative machines. In 14th International Conference on Computational Creativity, pages 278--282

    2023

  36. [44]

    Juliana Shihadeh and Margareta Ackerman. 2023 b . What does genius look like? an analysis of brilliance bias in text-to-image models. In 14th International Conference on Computational Creativity, pages 235--244

    2023

  37. [45]

    Saba Sturua, Isabelle Mohr, Mohammad Kalim Akram, Michael Günther, Bo Wang, Markus Krimmel, Feng Wang, Georgios Mastrapas, Andreas Koukounas, Andreas Koukounas, Nan Wang, and Han Xiao. 2024. https://arxiv.org/abs/2409.10173 jina-embeddings-v3: Multilingual embeddings with task...

    2024 arXiv

  38. [46]

    Gemma Team, Morgane Riviere, Shreya Pathak, Pier Giuseppe Sessa, Cassidy Hardin, Surya Bhupatiraju, Léonard Hussenot, Thomas Mesnard, Bobak Shahriari, Alexandre Ramé, Johan Ferret, Peter Liu, Pouya Tafti, Abe Friesen, Michelle Casbon, Sabela Ramos, Ravin Kumar, Charline Le Lan...

    2024 arXiv

  39. [47]

    Team OLMo , Pete Walsh, Luca Soldaini, Dirk Groeneveld, Kyle Lo, Shane Arora, , and Hannaneh Hajishirzi. 2025. https://arxiv.org/abs/2501.00656 2 OLMo 2 furious . Preprint, arXiv:2501.00656. ArXiv:2501.00656

    2025 arXiv

  40. [48]

    Selim Furkan Tekin, Fatih Ilhan, Tiansheng Huang, Sihao Hu, and Ling Liu. 2024. https://doi.org/10.18653/v1/2024.findings-emnlp.698 LLM - TOPLA : Efficient LLM ensemble by maximising diversity . In Findings of the Association for Computational Linguistics: EMNLP 2024, pages 11...

    2024 doi

  41. [49]

    Guy Tevet and Jonathan Berant. 2021. https://doi.org/10.18653/v1/2021.eacl-main.25 Evaluating the evaluation of diversity in natural language generation . In 16th Conference of the European Chapter of the Association for Computational Linguistics: Main Volume, pages 326--346. ...

    2021 doi

  42. [50]

    Hugo Touvron, Thibaut Lavril, Gautier Izacard, Xavier Martinet, Marie-Anne Lachaux, Timothée Lacroix, Baptiste Rozière, Naman Goyal, Eric Hambro, Faisal Azhar, Aurelien Rodriguez, Armand Joulin, Edouard Grave, and Guillaume Lample. 2023. https://arxiv.org/abs/2302.13971 Llama:...

    2023 arXiv

  43. [51]

    Ashwin Vijayakumar, Michael Cogswell, Ramprasaath Selvaraju, Qing Sun, Stefan Lee, David Crandall, and Dhruv Batra. 2018. https://doi.org/10.1609/aaai.v32i1.12340 Diverse beam search for improved description of complex scenes . AAAI Conference on Artificial Intelligence , 32(1)

    2018 doi

  44. [52]

    Zhilin Yang, Zihang Dai, Ruslan Salakhutdinov, and William W. Cohen. 2018. https://openreview.net/forum?id=HkwZSG-CZ Breaking the softmax bottleneck: A high-rank RNN language model . In International Conference on Learning Representations

    2018

  45. [53]

    Chunting Zhou, Pengfei Liu, Puxin Xu, Srinivasan Iyer, Jiao Sun, Yuning Mao, Xuezhe Ma, Avia Efrat, Ping Yu, LILI YU, Susan Zhang, Gargi Ghosh, Mike Lewis, Luke Zettlemoyer, and Omer Levy. 2023. https://proceedings.neurips.cc/paper_files/paper/2023/file/ac662d74829e4407ce1d126...

    2023

  46. [54]

    George Kingsley Zipf. 1949. Human behavior and the principle of least effort: An introduction to human ecology. Cambridge: Addison-Wesley

    1949

  47. [55]

    online" 'onlinestring :=

    ENTRY address archivePrefix author booktitle chapter edition editor eid eprint eprinttype howpublished institution journal key month note number organization pages publisher school series title type volume year doi pubmed url lastchecked label extra.label sort.label short.list...

  48. [56]

    write newline

    " write newline "" before.all 'output.state := FUNCTION n.dashify 't := "" t empty not t #1 #1 substring "-" = t #1 #2 substring "--" = not "--" * t #2 global.max substring 't := t #1 #1 substring "-" = "-" * t #2 global.max substring 't := while if t #1 #1 substring * t #2 gl...

Pith tools

Reviewed August 6, 2026 · model on record in the stance chip above.