{"id":"23bc99ec-bf7e-401c-9b42-a2b8a1d63bc8","arxiv_id":"2507.20956","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":5.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"Instruction-tuning reduces LLM output diversity, DPO causes the biggest drop, and conformative decoding, a log-probability mixture of instruct and base models, partly restores diversity while keeping quality.","lead":"The paper shows that instruction-tuning makes several open-source LLMs produce less diverse stories, and that the DPO preference-tuning step causes the largest drop. It then introduces conformative decoding, which mixes an instruct model's token probabilities with its more diverse base model, and reports that this typically restores diversity without hurting quality.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"The central quantitative claims appear supported; the strongest unresolved concern is that the per-prompt diversity gap and conformative-decoding gains are only measured on 50 samples per prompt, so the reported effect sizes might be inflated by small-sample sensitivity of Vendi Score and…","rationale":"I read the full text in good faith. The paper is a careful empirical study with a clear protocol: 53 prompts, 50 human responses per prompt, matched incipits, nucleus sampling p=0.95, and both per-prompt and across-prompt metrics. The strongest claims are: (1) instruction-tuning reduces diversity on five open-weight models with p<0.001 on per-prompt metrics; (2) DPO is the main contributor for OLMo/OLMo 2; (3) conformative decoding increases diversity for most models while maintaining quality. Each claim has a demonstrated exception or caveat: Llama 3.1 shows declines on per-prompt lexical diversity, and RLVR has little effect, so the paper itself correctly qualifies 'typically increases diversity.' The DPO attribution is supported by staged OLMo checkpoints and by MAUVE/Precision/Recall across prompts. I do not find a hidden fatal flaw; the method is simple, the write-up is honest about limitations, and the negative result for Llama is reported rather than hidden. My concern differs from the reader's: I worry less about metric construct validity in the abstract than about the specific sample-size/N-sensitivity of the two metrics that carry the per-prompt significance tests. Truncated Entropy is explicitly designed to cope with N < dimension but the paper does not validate its behavior for N=50 with jina-embeddings-v3; Vendi Score's eigenvalue estimate is known to depend on N. Since every per-prompt comparison uses the same N, such bias can still shift both arms differently if the true distributions differ in rank/spread, which is exactly the setting here. The proposed concrete test (N=50 vs. N=100) would settle whether the gap is robust. A human-elicited pairwise diversity judgment on a small sample would further strengthen the conclusion, but adding it as a hard requirement would be overly punitive given compute constraints and the paper's stated limitations. The paper's internal evidence plus prior work is sufficient for a CONDITIONAL accept that asks for the N-sensitivity check and artifact release. So I keep the reader's CONDITIONAL verdict and partially agree with the weakest-assumption statement: the reader targets construct validity of the metrics, while I target sample-size sensitivity of the very metrics used for the p<0.001 claims. These are complementary, so 'partial' is the honest agreement rating.","tokens_in":13704,"tokens_out":2249,"duration_ms":23321,"concrete_test":"Re-run the conformative-decoding evaluation with N=50 vs. N=100 (or subsample to N=25) for one model pair (e.g., OLMo 0724 instruct + base) while keeping the same prompts and incipits. If the per-prompt Vendi Score and Truncated Entropy gap between baseline and conformative decoding shrinks by more than ~30% when N is doubled, the reported improvement is partly a small-sample artifact; if the gap is stable, the claim survives this check. As a second check, report the Vendi Score of the human 50-response sets under the same kernel and the human-vs-human spread, since that gives a calibration anchor for what a 'significant' metric gap means at N=50.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The paper's per-prompt diversity claims (VSjina, VSn-gram, TEjina) are computed on only 50 generated stories per prompt. The paper itself notes that Truncated Entropy was introduced because empirical covariance estimates are singular when N < latent dimension, and that Vendi Score is bounded by N, so its behavior with small N depends on the spectrum estimator. The central claim is that instruction-tuning 'significantly reduces' diversity and conformative decoding 'typically increases' it. If these metrics have strong small-sample bias or N-dependent baseline shifts, part of the measured gap could be a metric artifact rather than a property of the text distribution. Within-model comparisons use the same N, so a monotone metric bias would cancel; however, the paper does not report calibration or N-sensitivity checks for Vendi Score or Truncated Entropy on this exact setup, and the metric literature (e.g., Shaib et al. 2025, cited in Limitations) warns that diversity scores are sensitive to length, tokenization, and sample size. The DPO attribution and the diversity-gap direction are corroborated by MAUVE, Precision/Recall, and by prior work cited in the paper, so the gap itself is not in doubt. The load-bearing weak point is the magnitude interpretation of per-prompt improvements from conformative decoding: with N=50 and a one-tailed t-test across only 53 prompts, a small systematic shift in the metric could be statistically significant without corresponding to a perceptible diversity increase. No human evaluation exists (stated in Limitations), so the practical claim that conformative decoding 'reintroduces diversity' rests entirely on these automatic metrics.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper investigates the reduction in output diversity caused by instruction-tuning of large language models. On a narrative generation task based on 53 writing prompts from the Writing Prompts dataset, the authors compare five open-weight base/instruct model pairs and report significant per-prompt decreases in Vendi Score (n-gram and jina-embedding features) and Truncated Entropy for instruction-tuned models, along with decreased Recall and increased Precision in across-prompt evaluations. Using intermediate OLMo checkpoints, they attribute the largest diversity loss to the DPO fine-tuning stage. They then propose conformative decoding, which mixes the next-token log-probabilities of the instruct model and its base model over the nucleus-sampled vocabulary of the instruct model, with a fixed mixing weight gamma=0.5. Experiments show that conformative decoding significantly increases per-prompt diversity for most models while maintaining or slightly improving automatic quality metrics.","tokens_in":13936,"tokens_out":9026,"duration_ms":98824,"significance":"If valid, the paper makes two useful contributions: it provides further evidence, with open models and intermediate checkpoints, that DPO is a major driver of the diversity drop, and it offers a simple, training-free decoding intervention that can partially restore diversity in instruction-tuned models. The experimental design is thoughtful: incipits are used to align base and instruct behavior, multiple diversity metrics are reported, and paired significance tests are performed. The main caveats are that the per-prompt diversity metrics are computed on only 50 samples per prompt without a small-sample sensitivity analysis, and that the diversity and quality claims rest entirely on automatic metrics in the absence of human evaluation, a limitation the authors acknowledge. The method is not a fitting procedure and does not bake in the evaluation metrics, so the reported improvements are empirical.","major_comments":[{"comment":"The similarity kernel k used in Eq. (1) is never specified for either the n-gram features or the jina embeddings. For example, the manuscript does not state whether the n-gram count vectors are L2-normalized before cosine similarity is computed, or whether the jina embeddings are used in raw or normalized form. Without this information, the numerical diversity scores cannot be reproduced, and differences among models could partly stem from kernel choices. Please specify the kernel for each feature type and the exact jina-embeddings-v3 configuration (task LoRA, pooling, normalization).","section":"§3.1.4"},{"comment":"All per-prompt diversity metrics (VSjina, VSn-gram, TEjina) are computed with N=50 samples per prompt. Vendi Score is bounded by N, and Truncated Entropy uses eigenvalue estimates of a covariance matrix that can be singular when N is not much larger than the embedding dimension. The manuscript does not report how stable the measured gaps and conformative-decoding improvements are with respect to N. I request an N-sensitivity analysis: recompute the per-prompt metrics on random subsamples of the 50 generations (e.g., N=10, 20, 30, 40) and show that the base-versus-instruct gap and the baseline-versus-conformative difference do not change direction or order of magnitude. This is important because the headline claim that conformative decoding 'typically increases diversity' is based on these small-sample metrics.","section":"§3.1.4 and §4.3"},{"comment":"The claim that DPO has the largest impact on diversity is based on visual inspection of Figure 2 rather than on paired statistical tests. Since the OLMo checkpoints allow a stage-by-stage comparison on the same 53 prompts, the authors should report paired one-tailed tests (or confidence intervals) comparing the SFT checkpoint with the DPO checkpoint for each per-prompt metric, and likewise for the RLVR checkpoint. Without such tests, contribution point 2 ('DPO is primarily responsible for the diversity gap') is not statistically supported.","section":"§3.2"},{"comment":"The per-prompt improvements from conformative decoding are statistically significant (p < .001) but are described as modest, and no effect sizes are reported. With N=50 generations per prompt and 53 prompts, a one-tailed t-test can detect a small systematic metric shift that may not correspond to a perceptible diversity difference in narrative text. Please report effect sizes (e.g., paired Cohen's d) for the significant comparisons and, ideally, include example narratives from configuration A and B to illustrate the type and magnitude of diversity change. This would help the reader judge the practical relevance of the improvements.","section":"§4.3"}],"minor_comments":[{"comment":"The incipit length is fixed at 20 tokens without any sensitivity analysis; since the incipit introduces shared content across all 50 generations, a different length could change the diversity measurements. The authors should at least discuss this choice or provide a robustness check.","section":"§3.1.2"},{"comment":"The significance tests do not account for multiple comparisons (6 per-prompt metrics × 5 models = 30 tests, plus the stage-wise tests). The authors should state whether any correction was applied; given the very low p-values this is unlikely to alter conclusions, but it should be explicit.","section":"§3.1.4"},{"comment":"The mixture of log-probabilities should be followed by renormalization over the truncated vocabulary Vvalid; this is implied by the softmax step mentioned in the text, but it would be clearer to include the explicit normalization in the equation or its surrounding text.","section":"§4.1, Eq. (3)"},{"comment":"The notation in Eq. (2) uses λᵢ^(l) without defining l; the text later refers to the latent space, but the equation should define it.","section":"§2.2 and §3.1.4"},{"comment":"The caption of Figure 1 states p < .001 for all comparisons, while the text reports p < .01 for some chat-template comparisons; please make the captions consistent with the exact p-value thresholds reported in the text.","section":"Figure 1"},{"comment":"Appendix A contains a typo, 'confirmation decoding' instead of 'conformative decoding'; the bibliography entry for Sturua et al. appears to be duplicated.","section":"Appendix A and References"},{"comment":"The limitations section acknowledges the absence of human evaluation; I encourage the authors to add a small human perception study on a subset of prompts to validate that the metric improvements translate to perceived diversity differences.","section":"Limitations"}],"recommendation":"major_revision","confidential_remarks":"The paper is within scope and the central idea is promising. The main concern is the evidentiary weight placed on per-prompt metrics with N=50; the requested sensitivity analysis should be feasible with the existing data. The DPO attribution claim also needs statistical backing. I do not see a circularity problem: conformative decoding is not fit to the evaluation metrics, and the DPO attribution is measured on public checkpoints. The paper's framing in the abstract is somewhat stronger than the results (e.g., 'even maintains or improves quality' is based on automatic metrics only), but this can be corrected by hedging."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"This paper is worth reading. The new thing is the per-stage attribution: for OLMo and OLMo 2, the diversity loss from instruction-tuning happens mostly at the DPO step, not SFT or RLVR. That is a clean measurement on public checkpoints, and it is actionable for the open-model community. The second contribution, conformative decoding, is simple: mix the instruct model's logits with the base model's over the nucleus-sampled vocabulary, gamma=0.5. It is not a fitting procedure, no free parameters beyond gamma, and it does typically increase diversity on the automatic metrics while keeping precision and MAUVE roughly flat.\n\nThe experimental design is mostly careful. Using incipits to align base and instruct behavior is a sensible compromise, and they explain why. They use several diversity metrics (Vendi with lexical and semantic features, Truncated Entropy) plus improved Precision/Recall and MAUVE, and they do paired significance tests across 53 prompts. The known exception—Llama 3.1 8B failing to improve on per-prompt lexical diversity—is reported honestly.\n\nWhere are the soft spots? The evaluation is entirely automatic. The paper says in Limitations that narrative diversity is hard for humans to assess and that no human evaluation was done. That is not fatal, but it means the claim that conformative decoding reintroduces diversity as a user-perceptible property is unverified. The per-prompt metrics are computed on 50 samples per prompt. Within-model comparisons use the same N, so a monotone N-dependent bias would cancel, but the magnitude of the improvements could still be inflated, and statistical significance across 53 prompts does not guarantee a perceptible difference. I would like to see N-sensitivity checks for Vendi Score and Truncated Entropy on this setup, plus code and generated outputs released. A contrastive decoding baseline would also help position the method.\n\nNone of this undermines the central claim. The diversity gap direction is corroborated by prior work and by MAUVE, and the DPO attribution is a real contribution. This deserves a serious referee, and I would bring it to reading group. Recommend: major revision with requests for artifact release and a robustness check on sample size, not a reject.","headline":"A careful empirical demonstration that DPO is the main diversity-reducing stage for OLMo, plus a simple decoding fix that mostly works — main gap is lack of human validation.","tokens_in":723,"tokens_out":952,"would_cite":true,"duration_ms":34524,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"This paper claims that instruction-tuning creates a measurable diversity gap in narrative generation, that DPO is the main culprit, and that mixing the instruct model's log-probabilities with its base model's restores much of the gap…","keywords":["output diversity","instruction-tuning","direct preference optimization","conformative decoding","narrative generation","Vendi Score","decoding strategies","large language models"],"falsifier":"A reader study in which people rank sets of stories from base, instruct, and conformative-decoding models for perceived variety would settle the claim: if reader rankings do not follow the Vendi Score and Truncated Entropy ordering, the measured diversity gap and recovery would be artifacts of the metrics rather than properties of the texts.","tokens_in":13445,"feed_emoji":"🎲","tokens_out":9338,"duration_ms":88002,"temperature":0.7,"pith_summary":"Instruction-tuning narrows what a language model will write: across five open-weight LLMs, the instruction-tuned versions produce significantly less diverse stories than their base models from the same writing prompts, with the drop visible in every measured diversity metric. The paper traces this 'diversity gap' through the OLMo fine-tuning chain and finds that direct preference optimization (DPO) is the stage that removes the most diversity, while the chat template used at inference time narrows output further. To counter the loss, the paper introduces conformative decoding, which mixes the next-token log-probabilities of an instruct model with those of its base model over the truncated candidate vocabulary, and shows that this typically restores diversity while maintaining or slightly improving automatic quality scores. The practical reason to care is that creative writing, brainstorming, and agent-based simulations all need models that can produce many different plausible outputs from one prompt, and this method offers a model-agnostic way to get that diversity back without retraining.","feed_headline":"Two-model logit blend brings back story diversity lost in fine-tuning","feed_subtitle":"Conformative decoding mixes instruct and base model log-probabilities, raising diversity scores without dropping quality.","key_machinery":"Conformative decoding is a sampling strategy defined by the mixture $$\\log p_{\\$\\theta$}(x_t \\mid x_{