REVIEW 6 cited by
Modifying Large Language Model Post-Training for Diverse Creative Writing
Not yet reviewed by Pith; the record is open.
This paper has not been read by Pith yet. Machine review is queued; the pith claim, tier, and objections will appear here once it completes.
SPECIMEN: schema-true, not a live event
T0 review · schema-true
One-sentence machine reading of the paper's core claim.
pith:XXXXXXXX · record.json · timestamp
Modifying Large Language Model Post-Training for Diverse Creative Writing
read the original abstract
As creative writing tasks do not have singular correct answers, large language models (LLMs) trained to perform these tasks should be able to generate diverse valid outputs. However, LLM post-training often focuses on improving generation quality but neglects to facilitate output diversity. Hence, in creative writing generation, we investigate post-training approaches to promote both output diversity and quality. Our core idea is to include deviation -- the degree of difference between a training sample and all other samples with the same prompt -- in the training objective to facilitate learning from rare high-quality instances. By adopting our approach to direct preference optimization (DPO) and odds ratio preference optimization (ORPO), we demonstrate that we can promote the output diversity of trained models while minimally decreasing quality. Our best model with 8B parameters could achieve on-par diversity as a human-created dataset while having output quality similar to the best instruction-tuned models we examined, GPT-4o and DeepSeek-R1. We further validate our approaches with a human evaluation, an ablation, and a comparison to an existing diversification approach, DivPO.
Forward citations
Cited by 6 Pith papers
-
Task-Dependent Evaluation of LLM Output Homogenization: A Taxonomy-Guided Framework
Proposes a task taxonomy for functional diversity in LLM outputs, validates it via user study, introduces targeted sampling to boost diversity only where needed, and presents evidence that the diversity-quality tradeo...
-
Anchorless Diversification for Parallel LLM Ideation
Semantic direction stratification yields the best diversity-quality-compute frontier for parallel LLM ideation across creative tasks.
-
LLM Jaggedness Unlocks Scientific Creativity
Jagged capabilities in LLMs for scientific idea generation can be leveraged through inference-time ensembles to outperform individual models.
-
Too Correct to Learn: Reinforcement Learning on Saturated Reasoning Data
A parameter-free sampling strategy called CUTS combined with Mixed-CUTS training prevents mode collapse in RL for saturated LLM reasoning tasks and raises AIME25 Pass@1 accuracy by up to 15.1% over standard GRPO.
-
Verbalized Sampling: How to Mitigate Mode Collapse and Unlock LLM Diversity
Typicality bias (human preference for familiar text) is estimated as α>0 in preference data and mathematically sharpens aligned LLMs toward a single mode; prompting for a verbalized response distribution (Verbalized S...
-
LLM Jaggedness Unlocks Scientific Creativity
LLMs exhibit jagged scientific creativity across models, prompts, and domains, and this unevenness can be leveraged via model ensembles to outperform any single model on idea generation.
discussion (0)
Sign in with ORCID, Apple, or X to comment. Anyone can read and Pith papers without signing in.