{"id":"ccf20382-f8bb-47c7-a7ef-58b033ee3e43","arxiv_id":"2608.07460","paper_version":1,"verdict":"CONDITIONAL","confidence":"MODERATE","novelty_score":7.0,"correctness_risk":"medium","formal_verification":"none","parameter_count":2,"one_line_summary":"A single instruction-tuned LLM can recover much of the diversity lost during alignment by learning to emit special creativity tokens, without using a second model at inference.","lead":"CreativeInstruct trains a single language model to insert special creativity markers into its own writing, using data from a two-model routing system. It reports diversity gains in story generation and better math results when that creative model is later fine-tuned with reinforcement learning.","discovery_kind":"new_method","skeptic_critique":{"model":"deepseek-v4-flash","headline":"Concern: LLM-GED validation conflates paraphrase insensitivity with narrative-diversity sensitivity; the claimed 63% structural-diversity gain for LLaMA-3.1 8B may be inflated by judge artifacts.","rationale":"The reader's weakest assumption is that automatic LLM-based metrics are valid proxies for both quality and structural diversity. I partially agree, and I sharpen the concern specifically to LLM-GED's validation gap. The reader correctly notes that the human quality annotations were dropped after negative agreement and that LLM-GED was validated only on synthetic transformations and a 50-story pipeline. However, I would push further: the equivalence check in Appendix B.2 does not independently validate the metric, because the deterministic pipeline embeds the same narrative-structure assumptions as the LLM-GED prompt (free substitution for same coarse narrative role; temporal-order edits as distinct). Agreement between two implementations of the same assumption is not evidence of validity. Also, the synthetic validation in Table 9 measures monotonicity against the authors' own pre-registered ordering, which is a weak check for a metric that then supports the paper's headline claim. For the quality half of the central claim, Table 2 shows no consistent improvement and several losses (e.g., Qwen2.5 7B coherence 3.82 vs. 4.18; Qwen3 8B coherence 3.47 vs. 3.50; WQRM mixed across families), so the abstract's 'without sacrificing quality' is really 'not sacrificing on automatic quality proxies,' which is a weaker and more defensible reading. The RL claim is more robust: Table 6 reports only single numbers averaged over three seeds, but the comparison is apples-to-apples and a +4% AMC gain is plausible if the creative checkpoint does provide diverse rollouts; however, the paper does not report rollout diversity during RL, so the causal mechanism is inferred rather than shown. Overall, the diversity half of the central claim is plausible but not yet established because LLM-GED is the only metric that specifically claims narrative-level diversity, and it is the least validated. This does not change the verdict: CONDITIONAL remains appropriate, with the requested human validation as the condition for upgrading to ACCEPT.","tokens_in":18175,"tokens_out":2128,"duration_ms":18441,"concrete_test":"Run a human validation study for LLM-GED on the actual evaluation setting: take 50 prompt groups from Narrative Discourse (5-10 generations per group) whose model identities are masked, have 3-5 trained annotators rate pairwise structural diversity (not overall creativity), and compute rank correlation between mean human structural-diversity judgments and LLM-GED rankings per prompt group. If the correlation is below ~0.6 (comparable to the synthetic correlation of 0.889), recompute Table 1 with LLM-GED removed or reweighted; the headline 63% structural-diversity gain would not be established. Also report LLM-GED with different judge models (e.g., GPT5-mini vs. an open judge) to quantify judge sensitivity.","verdict_should_be":"CONDITIONAL","load_bearing_attack":"The central claim that CreativeInstruct 'matches or exceeds the diversity of both multi-model baselines and distilled variants... without sacrificing quality' rests on Table 1, where the headline metric LLM-GED shows CreativeInstruct at 0.545 versus Instruct at 0.366 for LLaMA-3.1 8B (a 63% relative gain). The metric's validation in Appendix B.1/B.2 is not sufficient to support this. First, Table 9 is a correlation with a hand-authored ranking of four synthetic transformation types (same, lexical, temporal, different), not with human judgments of real narrative diversity; high correlation on this toy ordering cannot establish that LLM-GED is unbiased across real model outputs. Second, Appendix B.2's equivalence check compares LLM-GED against a deterministic pipeline whose node-substitution cost already encodes a strong invariance assumption: substitution is free for event nodes sharing a 'coarse narrative role' (observation, quote, reflection). This pipeline was validated on only 50 stories with 'less than 1000 tokens' and the reported mean-difference test (p=0.56) has low power at that size; equivalence is asserted, not demonstrated with a proper TOST or bounds. Third, the LLM-GED prompt tells the judge to canonicalize synonyms/paraphrases as zero-cost but to treat temporal reorderings as edits, so the metric is explicitly constructed to agree with the authors' narrative-structure hypothesis; agreement with the deterministic pipeline therefore only shows the same assumption was consistently applied, not that the assumption is what humans perceive. Because 'without sacrificing quality' also depends on automatic judges (GPT5-mini and WQRM) after the human quality annotations were dropped (kappa = -0.167), both halves of the central claim are vulnerable to the same unvalidated-judge problem.","agreement_with_reader":"partial"},"referee_report":{"model":"deepseek-v4-flash","summary":"The paper proposes CreativeInstruct, an instruction-tuning method that builds creativity-tagged training data by routing between a base model and its aligned counterpart (using BACo), then fine-tunes a single aligned model with special [StartCreativity]/[EndCreativity] tokens so that the model learns to inject creative spans at inference time. The paper also introduces an LLM-based graph-edit-distance metric (LLM-GED) intended to capture structural narrative diversity. Experiments on the Narrative Discourse dataset across five model families (LLaMA-3.1 8B, Qwen2.5 7B/32B, Qwen3 8B/32B) report diversity improvements over Instruct, BACo, a tag-ablated Distill baseline, and (for LLaMA-3.1 8B) CrPO, along with automatic quality scores and a writing-quality reward model. A human study finds a 70.3% preference for CreativeInstruct on creativity, but quality judgments were dropped due to negative inter-annotator agreement. Finally, GRPO applied to a CreativeInstruct Qwen3 8B checkpoint improves MATH and AMC accuracy relative to the same RL recipe applied to the base instruct checkpoint.","tokens_in":18500,"tokens_out":5254,"duration_ms":48722,"significance":"If the claims hold, CreativeInstruct would be a practically valuable single-model alternative to multi-model inference-time routing, and the RL result would demonstrate a useful connection between generation diversity and downstream reasoning. The paper has several concrete strengths: the Distill ablation isolates the contribution of the creativity tokens; the RL comparison uses an identical training recipe for both starting points; the evaluation spans multiple model families and sizes; and the code is released. The main open question is whether the two headline conclusions survive stronger measurement: the \"without sacrificing quality\" claim is contradicted by parts of Table 2 and rests on automatic metrics after the human quality annotation was dropped, and the structural-diversity metric LLM-GED is validated largely against synthetic transformations and a deterministic pipeline that encodes a similar structural assumption.","major_comments":[{"comment":"The abstract's claim that CreativeInstruct matches or exceeds diversity 'without sacrificing quality' is not supported by Table 2. For Qwen3 32B, CreativeInstruct is lower than the Instruct baseline on all four reported quality measures (coherence 4.36 vs. 4.45, fluency 4.14 vs. 4.85, relevance 4.01 vs. 4.76, WQRM 6.32 vs. 7.39). For Qwen2.5 7B, coherence, fluency, and relevance all decrease relative to Instruct (3.82 vs. 4.18, 4.02 vs. 4.62, 3.84 vs. 4.26). The Section 4.1 wording 'generally maintains competitive performance' is more accurate; the abstract should be revised to match the evidence, or the authors should add a human quality evaluation that supports the stronger claim.","section":"Abstract and Section 4.1, Table 2"},{"comment":"The abstract states that the results 'are mirrored in our human evaluation,' but the human evaluation in Section 4.2 explicitly omits quality judgments because inter-annotator agreement on quality was negative (kappa = -0.167). Table 5 reports preferences only for diversity and creativity. Consequently, the human evaluation supports the creativity-preference claim but cannot be cited as evidence for the 'without sacrificing quality' part of the central claim. Please either reword the abstract and Section 4.2 to separate the supported claims from the unsupported one, or collect human quality judgments with a revised annotation protocol.","section":"Section 4.2 and Abstract"},{"comment":"LLM-GED is the headline metric for the structural-diversity gains, but its validation is not yet strong enough to carry that weight. Table 9 reports correlation against a hand-authored ordering of four synthetic transformation types (same, lexical, temporal, different), which is not a substitute for correlation with human judgments of structural diversity on real model outputs. Appendix B.2's equivalence check uses a paired t-test on 50 stories (mean difference -0.012, p = 0.56); failure to reject the null is not evidence of equivalence, and 50 stories is too small for high power. Moreover, the deterministic pipeline already treats substitution as free for event nodes sharing the same coarse narrative role (observation, quote, reflection), and the LLM-GED prompt instructs the judge to canonicalize synonyms and paraphrases while penalizing temporal reorderings; agreement between the two therefore shows internal consistency of assumptions, not unbiased measurement. I would ask for an independent validation of LLM-GED against human structural-diversity ratings on real generated stories, and for a confidence interval or TOST-style analysis for the pipeline comparison, before relying on the 63% relative LLM-GED gain for LLaMA-3.1 8B.","section":"Section 3.2 and Appendices B.1/B.2"}],"minor_comments":[{"comment":"The sentence 'the AM performance actually drops slightly' should read 'the AMC performance'.","section":"Section 4.3"},{"comment":"The caption contains the typo 'Anotation interface'; it should be 'Annotation interface'.","section":"Figure 6 caption"},{"comment":"The text says CreativeInstruct 'consistently' improves diversity, but in Table 1 the Qwen3 8B MiniLM cosine dissimilarity is lower for CreativeInstruct (0.369) than for Instruct (0.399), and for Qwen3 32B the Distill baseline outperforms CreativeInstruct on several semantic metrics. Please qualify 'consistently' as 'in most settings and on most metrics'.","section":"Section 4.1 and Table 1"},{"comment":"The claim that general-purpose Tülu data outperforms in-domain data 'even at the same data set size' is not quite supported: the in-domain variant uses 2,020 samples while the nearest Tülu comparison uses 2,000 samples, and the prompt sets differ. Clarify the comparison or match the datasets more carefully.","section":"Section 4.1, 'Importance of Data Diversity'"},{"comment":"No significance tests or confidence intervals are reported for the diversity or automatic quality metrics. At least for the headline LLM-GED and the quality metrics that support the 'no sacrifice' claim, error bars or significance tests would help the reader assess stability across prompts and sampling.","section":"Tables 1 and 2"},{"comment":"The significance asterisk is shown only for Creativity (70.3%). Please state explicitly whether the Diversity win rate (57.4%) is not statistically significant, and report the inter-annotator agreement for the diversity and creativity judgments on the 14-prompt subset.","section":"Section 4.2, Table 5"}],"recommendation":"major_revision","confidential_remarks":"This is a well-executed empirical paper with a transparent report of the failed human quality annotation, which I take as a sign of good-faith reporting. The main blocker is the gap between the abstract's strong claims and the evidence: the quality claim is contradicted by parts of Table 2, and the LLM-GED validation is not yet strong enough to rule out judge artifacts in the headline structural-diversity numbers. Both issues seem fixable within the manuscript's scope by softening claims or adding validation, so I recommend major revision rather than rejection. I would also gently flag that the paper relies heavily on a single LLM judge (GPT5-mini) for both the new metric and the quality evaluation; an independent human-quality or human-structural-diversity component would substantially strengthen the revision."},"author_rebuttal":null,"desk_editor":{"model":"deepseek-v4-flash","letter":"Hi [colleague],\n\nThe thing to know: CreativeInstruct is a practical, single-model alternative to two-model routing like BACo, and the core diversity gains look real across several model families. The paper that arrives with it, however, overstates the quality guarantee, and the new LLM-GED metric is not validated well enough to carry the structural-diversity headline.\n\nWhat is actually new: instead of optimizing whole outputs for diversity or routing at test time, they train an aligned model on spans from the base model wrapped in special tokens, and at inference the model self-injects those tokens. The Distill ablation (same data without the tags) is the right control and shows the tags are doing real work. The RL result, GRPO on a CreativeInstruct checkpoint beating GRPO on the instruct checkpoint by about 4% on AMC and 5 points on MATH, is a useful, independently interesting finding even if the mechanism (more diverse rollouts) is not directly measured.\n\nCredit where due: the paper is honest about the Qwen3 32B case where Distill wins on most diversity metrics, and it reports the dropped human quality judgments with negative kappa rather than hiding them. Across four of five model families the diversity improvements are consistent, which is the strongest evidence that the method, not a particular model, works.\n\nSoft spots, in proportion. First, the abstract says \"without sacrificing quality,\" but Table 2 shows Qwen3 32B CreativeInstruct is below Instruct on all four quality scores and Qwen2.5 7B is below on coherence and relevance. There are no error bars on any automatic metric, so we cannot tell whether these are real degradations or noise. The claim should be softened to \"comparable quality on most models.\"\n\nSecond, LLM-GED is the load-bearing metric for structural diversity, and its validation is thinner than the headline needs. The correlation in Table 9 is against a hand-authored ranking of four synthetic transformation types, not human judgments of real story pairs. The deterministic pipeline comparison in Appendix B.2 uses n=50, a p=0.56 from a paired t-test, which is not equivalence testing, and the pipeline's node-substitution rule already encodes the same \"coarse narrative role\" invariance that LLM-GED is supposed to prove. So the 63% structural gain for LLaMA-3.1 8B could be partly judge bias. This is addressable, with a small human study on pairwise structural diversity or a proper TOST with reasonable bounds, but as written it is a real vulnerability.\n\nThird, the human evaluation is 50 prompts on one model, with quality dropped due to negative kappa. The 70.3% creativity preference is nice, but it does not rescue the automatic-quality claims.\n\nBottom line: the central diversity result is probably right, and the method is worth serious referee time. The authors need to fix the quality overstatement and either validate LLM-GED against human ratings or present it as a proposed metric with known limitations. This paper is for anyone working on creative generation, diversity-aware post-training, or RL exploration, and it deserves a serious referee rather than a desk reject.","headline":"A practical single-model diversity recipe with real gains, but the quality and structural-diversity headlines lean harder on unvalidated LLM judges than the data supports.","tokens_in":19087,"tokens_out":2348,"would_cite":true,"duration_ms":21867,"reading_group":"yes","serious_thinker":"yes","would_accept_peer_review":true},"rs_alignment":null,"lean_confirmation":null,"pith_extraction":{"msc":[],"pacs":[],"model":"deepseek-v4-flash","headline":"CreativeInstruct claims a single trained LLM can recover the diversity that post-training removes, by learning to emit its own [StartCreativity] spans, and that the result is a better substrate for reinforcement learning.","keywords":["creativity","output diversity","instruction tuning","reinforcement learning","graph edit distance","narrative generation","LLM evaluation","post-training"],"falsifier":"Run a human study with enough annotators to get trustworthy quality ratings (for instance, at least five independent ratings per story with pre-agreed rubrics) comparing CreativeInstruct against the Instruct baseline on matched outputs. If human-rated quality is significantly lower for CreativeInstruct, or if human-rated structural diversity does not favor it in line with LLM-GED, the paper's central 'better and more diverse, no quality loss' claim would be falsified.","tokens_in":17984,"feed_emoji":"✨","tokens_out":7699,"duration_ms":66529,"temperature":0.7,"pith_summary":"Alignment makes LLMs better at following instructions but also makes their outputs more repetitive and homogeneous. CreativeInstruct's claim is that a single model can recover much of that lost diversity by learning where to be creative: the model is trained on outputs whose creative, base-model-written spans are wrapped in special [StartCreativity] tokens, and at test time it injects those tokens itself. Across five model families and sizes, the paper reports that CreativeInstruct matches or beats two-model decoding (BACo) and distilled variants on semantic, lexical, and structural diversity, while keeping quality scores competitive. It also reports that applying GRPO to a CreativeInstruct checkpoint improves math reasoning by about 4 points on AMC and about 5 points on MATH over the same RL applied to the plain post-trained checkpoint. The reason to care is that diversity is usually treated as a separate objective or a test-time cost; this paper argues it can be taught once, inside a single checkpoint, and that doing so pays off downstream.","feed_headline":"Creativity tags let one LLM beat two-model routing","feed_subtitle":"Training on tagged base-model spans restores diversity without quality loss and boosts RL math scores.","key_machinery":"The mechanism is a pair of learned control tokens, [StartCreativity] and [EndCreativity], placed around contiguous spans that a base model contributed to a routed generation during data creation, plus the instruction-tuning corpus built that way. Training data comes from routing each token of an output between base and aligned models by entropy and punctuation, so high-entropy tokens go to the base model; after fine-tuning, the aligned model learns to emit those tokens itself, effectively switching its own style at the right moments. The second machine is LLM-GED, an evaluation method that has an LLM turn each story into a canonical abstract event graph of entities, events, and temporal edges, and then reports the average normalized graph edit distance between pairs; this is the metric that claims to see narrative-level differences that lexical and semantic metrics miss.","core_discovery":"The paper's central claim is that post-training's loss of diversity is not irreversible and does not require a second model at inference. CreativeInstruct says a single aligned LLM can be taught to act as its own router: it learns to insert [StartCreativity]...[/EndCreativity] spans that mark where the base model's more varied, less constrained style should take over, and it does this at generation time on its own. The paper further claims this is not just distillation of the routing teacher: the trained model outperforms the two-model BACo baseline that generated its training data on most diversity metrics. The same mechanism, the paper argues, makes a better substrate for RL, because more diverse rollouts give GRPO more to explore; the evidence is a roughly 4-point gain on AMC and 5-point gain on MATH when GRPO is run on CreativeInstruct rather than on the Instruct checkpoint.","pith_inferences":["A testable extension suggested by Qwen3 32B's cross-family transfer: if the creativity-tagging recipe works when no base model exists for the target family, the data-generation stage could in principle be decoupled from the model being trained, letting creativity be distilled across generations of models.","The [StartCreativity] tokens can be read as a discrete, interpretable switch; one could verify this by ablating the tokens at generation time and checking whether diversity collapses to the aligned model's level, which would show the model is genuinely routing through its own tags.","If LLM-GED is accepted as a structural-diversity measure, it could be reused as a cheap mode-collapse audit for any open-ended generation system, not just stories; the paper's validation, however, only covers 50 synthetic story variations, so that reuse would need independent testing."],"forward_implications":["At inference, diversity no longer costs twice the compute: a single CreativeInstruct checkpoint replaces the base-plus-aligned pair that created its training data, and the paper reports it beats that pair on most diversity metrics.","The training recipe is scalable from general-purpose instruction data: gains on semantic and structural diversity keep rising as the training set grows from 1,000 to 12,000 samples, with no plateau at the largest size.","The [StartCreativity] and [EndCreativity] tags matter for the method: the Distill baseline trained on the same data without the tags has lower quality and structural diversity, so the tags themselves carry the balance.","Creative instruction-tuned models give RL better exploration: GRPO on the CreativeInstruct checkpoint beats GRPO on the Instruct checkpoint by roughly 4 points on AMC and 5 points on MATH, with the AMC gain concentrated at higher difficulty.","Proper-noun uniqueness roughly doubles over the Instruct model (37.1% versus 18.1% per prompt group), a concrete sign that repetitive characters and locations are reduced."],"supporting_citations":[{"why":"Supplies BACo, the base-aligned token routing method that generates the creativity-tagged training data and serves as the two-model test-time baseline.","marker":"Wang et al. (2026)"},{"why":"Provides the Tülu V3 SFT writing prompts that CreativeInstruct fine-tunes on.","marker":"Lambert et al. (2024)"},{"why":"Provides the Narrative Discourse dataset on which narrative generation diversity and quality are evaluated.","marker":"Tian et al. (2024)"},{"why":"Supplies GRPO, the reinforcement-learning algorithm used to test CreativeInstruct as an RL substrate.","marker":"Shao et al. (2024)"},{"why":"Supplies the MATH split used for RL training and in-domain math evaluation.","marker":"Hendrycks et al. (2021)"},{"why":"Supplies the AMC benchmark used for out-of-domain math generalization after RL.","marker":"Yang et al. (2024a)"},{"why":"Supplies the writing-quality reward model used to score generation quality.","marker":"Chakrabarty et al. (2025)"},{"why":"Supplies CrPO, the creative-preference-optimization baseline compared on LLaMA-3.1 8B.","marker":"Ismayilzada et al. (2025)"}],"fun_headline_variants":["One LLM routes itself with creativity tags","Self-routing language model keeps diversity and quality","CreativeInstruct: single model, tag-injected diversity","Tagged creativity spans let one LLM match two-model diversity","RL gains from diversity-preserving instruction tuning"],"cache_read_input_tokens":3200,"weakest_assumption_plain":"The load-bearing premise is that the automated LLM-based judge used for quality and structural diversity (plus the writing-quality reward model) measures what the paper claims it measures; the paper's own human quality ratings were too unreliable to use (kappa = -0.167), so 'without sacrificing quality' rests entirely on automatic metrics.","fun_headline_variants_meta":{"raw":{"variants":["One LLM routes itself with creativity tags","Self-routing language model keeps diversity and quality","CreativeInstruct: single model, tag-injected diversity","Tagged creativity spans let one LLM match two-model diversity","RL gains from diversity-preserving instruction tuning"]},"model":"deepseek-v4-flash","effort":"low","cost_usd":0.000649,"raw_usage":{"total_tokens":2982,"prompt_tokens":951,"completion_tokens":2031,"prompt_tokens_details":{"cached_tokens":384},"prompt_cache_hit_tokens":384,"prompt_cache_miss_tokens":567,"completion_tokens_details":{"reasoning_tokens":1957}},"tokens_in":567,"tokens_out":2031,"duration_ms":14201,"temperature":1.0,"reasoning_tokens":1957,"cache_read_input_tokens":384,"cache_creation_input_tokens":0},"cache_creation_input_tokens":0},"created_at":"2026-08-10T04:17:26.556936+00:00","model_set":{"reader":"deepseek-v4-flash"},"falsifier":"Run a human study with enough annotators to get trustworthy quality ratings (for instance, at least five independent ratings per story with pre-agreed rubrics) comparing CreativeInstruct against the Instruct baseline on matched outputs. If human-rated quality is significantly lower for CreativeInstruct, or if human-rated structural diversity does not favor it in line with LLM-GED, the paper's central 'better and more diverse, no quality loss' claim would be falsified.","supporting_citations":[],"review_version":1}